⚡ One Interface, Any Model

Microsoft.Extensions.AI in .NET: Build a Switchable Multi-Provider AI Gateway with IChatClient

Build one provider-neutral gateway for OpenAI-compatible services, Azure OpenAI, and local Ollama models—then switch models through configuration without rewriting your application layer.

Published Updated 48 min readIntermediate

What You Will Build

Build one ASP.NET Core gateway that can talk to OpenAI-compatible services, Azure OpenAI, or a local Ollama model without leaking provider SDK types into endpoints or business logic.

The finished API exposes a stable contract:

  • POST /api/chat — complete response
  • POST /api/chat/stream — Server-Sent Events stream
  • GET /api/providers — enabled providers and declared capabilities
  • GET /health — gateway health
HTTP endpoint
     │
     ▼
IAiGateway
     │
     ▼
Provider registry ──► descriptor (identity + model + capabilities + IChatClient)
     │
     ├── OpenAI-compatible client
     ├── Azure OpenAI client
     └── OllamaSharp client
✅
Version target: .NET 10 LTS and Microsoft.Extensions.AI 10.x. Provider-adapter versions are pinned in Directory.Packages.props, not scattered through the prose. Last reviewed: July 22, 2026.

Why AI Provider Coupling Becomes Expensive

A prototype often begins with a provider SDK directly inside an endpoint. That works until the provider, deployment, model, authentication mechanism, or test environment changes.

Provider-coupled anti-pattern
app.MapPost("/api/chat", async (ChatRequest request) =>
{
    var client = new OpenAI.Chat.ChatClient(
        model: builder.Configuration["OpenAI:Model"]!,
        apiKey: builder.Configuration["OpenAI:ApiKey"]!);

    var completion = await client.CompleteChatAsync(request.Message);
    return Results.Ok(completion.Value.Content[0].Text);
});

The endpoint now owns provider construction, credentials, request conversion, and response mapping. Testing it requires a real network. A provider change becomes an endpoint rewrite.

The honest promise: provider-specific registration still exists in the composition root. The abstraction removes provider-specific code from endpoints, application services, tests, and conversation logic.

Prerequisites

  • .NET 10 SDK and an IDE
  • Ollama installed locally with llama3.2 or another chat model pulled
  • Optional OpenAI-compatible API key
  • Optional Azure OpenAI resource and chat deployment
Verify prerequisites
dotnet --version
ollama --version
ollama pull llama3.2
ollama list
Tool calling varies by model. Only enable the tool chapter with a model/deployment you have explicitly declared as tool-capable.

Create the Solution

Create the projects
mkdir SwitchableAiGateway
cd SwitchableAiGateway

dotnet new sln -n SwitchableAiGateway
dotnet new web -n SwitchableAiGateway.Api -o src/SwitchableAiGateway.Api
dotnet new classlib -n SwitchableAiGateway.Core -o src/SwitchableAiGateway.Core
dotnet new xunit -n SwitchableAiGateway.Tests -o tests/SwitchableAiGateway.Tests

dotnet sln add src/SwitchableAiGateway.Api
dotnet sln add src/SwitchableAiGateway.Core
dotnet sln add tests/SwitchableAiGateway.Tests

dotnet add src/SwitchableAiGateway.Api reference src/SwitchableAiGateway.Core
dotnet add tests/SwitchableAiGateway.Tests reference src/SwitchableAiGateway.Core

Enable Central Package Management so fast-moving package versions live in one reviewable file.

Directory.Packages.props
Central package management
<Project>
  <PropertyGroup>
    <ManagePackageVersionsCentrally>true</ManagePackageVersionsCentrally>
  </PropertyGroup>

  <!-- Pin versions verified on your publication date. -->
  <ItemGroup>
    <PackageVersion Include="Microsoft.Extensions.AI" Version="10.x.y" />
    <PackageVersion Include="Microsoft.Extensions.AI.OpenAI" Version="10.x.y" />
    <PackageVersion Include="OpenAI" Version="2.x.y" />
    <PackageVersion Include="Azure.AI.OpenAI" Version="2.x.y" />
    <PackageVersion Include="Azure.Identity" Version="1.x.y" />
    <PackageVersion Include="OllamaSharp" Version="5.x.y" />
    <PackageVersion Include="OpenTelemetry.Extensions.Hosting" Version="1.x.y" />
    <PackageVersion Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.x.y" />
    <PackageVersion Include="OpenTelemetry.Instrumentation.Http" Version="1.x.y" />
    <PackageVersion Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.x.y" />
    <PackageVersion Include="Microsoft.AspNetCore.OpenApi" Version="10.x.y" />
  </ItemGroup>
</Project>
Publication workflow: replace each x.y placeholder with the stable version restored and tested on the publication date. Keeping placeholders here is deliberate: provider adapters change faster than the tutorial’s architectural lessons.
SwitchableAiGateway.Api.csproj
Project references
<Project Sdk="Microsoft.NET.Sdk.Web">
  <PropertyGroup>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.Extensions.AI" />
    <PackageReference Include="Microsoft.Extensions.AI.OpenAI" />
    <PackageReference Include="OpenAI" />
    <PackageReference Include="Azure.AI.OpenAI" />
    <PackageReference Include="Azure.Identity" />
    <PackageReference Include="OllamaSharp" />
    <PackageReference Include="OpenTelemetry.Extensions.Hosting" />
    <PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" />
    <PackageReference Include="OpenTelemetry.Instrumentation.Http" />
    <PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" />
    <PackageReference Include="Microsoft.AspNetCore.OpenApi" />
  </ItemGroup>
</Project>

Understand IChatClient

IChatClient is the provider-neutral application contract. Its current methods are GetResponseAsync and GetStreamingResponseAsync.

IAiGateway.cs
Core gateway contract
using Microsoft.Extensions.AI;

public interface IAiGateway
{
    Task<GatewayResult> SendAsync(
        GatewayChatRequest request,
        CancellationToken cancellationToken);

    IAsyncEnumerable<GatewayStreamEvent> StreamAsync(
        GatewayChatRequest request,
        CancellationToken cancellationToken);
}
Outdated samples: code that uses CompleteAsync, CompleteStreamingAsync, or .AsChatClient() predates the current API. Use GetResponseAsync, GetStreamingResponseAsync, and .AsIChatClient().

Do not name the public DTO ChatResponse; that name already belongs to Microsoft.Extensions.AI.ChatResponse.

GatewayChatContracts.cs
Public request and response contracts
public sealed record GatewayChatRequest(
    string Message,
    string? ConversationId = null,
    string? Provider = null,
    bool EnableTools = false);

public sealed record GatewayChatResponse(
    string ConversationId,
    string Provider,
    string Model,
    string Content,
    string? FinishReason,
    TokenUsageDto? Usage,
    long DurationMs);

public sealed record TokenUsageDto(long? InputTokens, long? OutputTokens);

Configure Providers and Validate at Startup

appsettings.json
Application configuration
{
  "AiGateway": {
    "DefaultProvider": "ollama",
    "AllowRequestProviderSelection": true,
    "MaximumConversationMessages": 20,
    "MaximumConversationCharacters": 40000,
    "ConversationLifetimeMinutes": 30,
    "RequestTimeoutSeconds": 60
  },
  "AiProviders": {
    "OpenAI": { "Enabled": false, "Model": "configured-model-name" },
    "AzureOpenAI": {
      "Enabled": false,
      "Endpoint": "https://example.openai.azure.com/",
      "Deployment": "configured-deployment-name"
    },
    "Ollama": {
      "Enabled": true,
      "Endpoint": "http://localhost:11434",
      "Model": "llama3.2"
    }
  }
}
AiGatewayOptions.cs
Options model
using System.ComponentModel.DataAnnotations;

public sealed class AiGatewayOptions
{
    public const string SectionName = "AiGateway";

    [Required] public string DefaultProvider { get; init; } = "ollama";
    public bool AllowRequestProviderSelection { get; init; }
    [Range(2, 100)] public int MaximumConversationMessages { get; init; } = 20;
    [Range(1000, 200000)] public int MaximumConversationCharacters { get; init; } = 40000;
    [Range(1, 1440)] public int ConversationLifetimeMinutes { get; init; } = 30;
    [Range(5, 600)] public int RequestTimeoutSeconds { get; init; } = 60;
}
Program.cs
Validate configuration on startup
builder.Services
    .AddOptions<AiGatewayOptions>()
    .Bind(builder.Configuration.GetSection(AiGatewayOptions.SectionName))
    .ValidateDataAnnotations()
    .ValidateOnStart();

Store API keys with User Secrets locally. Never put credentials in appsettings.json.

Local secrets
cd src/SwitchableAiGateway.Api
dotnet user-secrets init
dotnet user-secrets set "AiProviders:OpenAI:ApiKey" "YOUR_KEY"

Model Providers as Capability-Aware Descriptors

AiProviderDescriptor.cs
Provider descriptor
using Microsoft.Extensions.AI;

public sealed record AiProviderCapabilities(
    bool SupportsStreaming,
    bool SupportsTools,
    bool ReportsUsage,
    int? MaximumContextTokens,
    bool SupportsProviderManagedConversation);

public sealed record AiProviderDescriptor(
    string Name,
    string Model,
    AiProviderCapabilities Capabilities,
    IChatClient Client);
CapabilityWhy declare it
StreamingReject streaming requests before invoking an unsupported adapter.
ToolsPrevent tools from being offered to deployments that cannot call them reliably.
Usage reportingDo not promise token counts when the provider does not return them.
Context limitUse it as an operator-supplied guardrail, not a runtime guess.
Provider conversationControls whether a returned conversation identifier can be reused.
Do not probe capabilities automatically. Models and deployments change, probes cost money, and “supported” does not mean “reliable.” Keep capabilities operator-declared and reviewed.

Register OpenAI, Azure OpenAI, and Ollama

OpenAI-compatible

API-key authentication, model name, optional compatible endpoint.

Azure OpenAI

Endpoint plus deployment name, preferably Microsoft Entra authentication.

OllamaSharp

Local inference through an IChatClient-compatible client.

AiProviderNames.cs
Provider keys
public static class AiProviderNames
{
    public const string OpenAi = "openai";
    public const string AzureOpenAi = "azure-openai";
    public const string Ollama = "ollama";
}
Program.cs
Register keyed chat clients
using Azure.AI.OpenAI;
using Azure.Identity;
using Microsoft.Extensions.AI;
using OllamaSharp;
using OpenAI;
using System.ClientModel;

// OpenAI-compatible cloud client
builder.Services
    .AddKeyedChatClient(AiProviderNames.OpenAi, sp =>
    {
        IConfiguration config = sp.GetRequiredService<IConfiguration>();
        string model = config["AiProviders:OpenAI:Model"]
            ?? throw new InvalidOperationException("OpenAI model is missing.");
        string apiKey = config["AiProviders:OpenAI:ApiKey"]
            ?? throw new InvalidOperationException("OpenAI API key is missing.");

        return new OpenAI.Chat.ChatClient(model, apiKey).AsIChatClient();
    })
    .UseFunctionInvocation()
    .UseOpenTelemetry(configure: telemetry =>
        telemetry.EnableSensitiveData = builder.Environment.IsDevelopment());

// Azure OpenAI with Microsoft Entra ID / managed identity
builder.Services
    .AddKeyedChatClient(AiProviderNames.AzureOpenAi, sp =>
    {
        IConfiguration config = sp.GetRequiredService<IConfiguration>();
        var endpoint = new Uri(config["AiProviders:AzureOpenAI:Endpoint"]!);
        string deployment = config["AiProviders:AzureOpenAI:Deployment"]!;

        return new AzureOpenAIClient(endpoint, new DefaultAzureCredential())
            .GetChatClient(deployment)
            .AsIChatClient();
    })
    .UseFunctionInvocation()
    .UseOpenTelemetry(configure: telemetry =>
        telemetry.EnableSensitiveData = builder.Environment.IsDevelopment());

// Local Ollama. Microsoft.Extensions.AI.Ollama is deprecated; use OllamaSharp.
builder.Services
    .AddKeyedChatClient(AiProviderNames.Ollama, sp =>
    {
        IConfiguration config = sp.GetRequiredService<IConfiguration>();
        var endpoint = new Uri(config["AiProviders:Ollama:Endpoint"]!);
        string model = config["AiProviders:Ollama:Model"]!;

        return new OllamaApiClient(endpoint, model);
    })
    .UseFunctionInvocation()
    .UseOpenTelemetry(configure: telemetry =>
        telemetry.EnableSensitiveData = builder.Environment.IsDevelopment());
Azure local fallback: where managed identity is unavailable, an ApiKeyCredential can be used locally. Prefer DefaultAzureCredential for deployed workloads.

Build the Provider Registry

IAiProviderRegistry.cs
Registry contract
public interface IAiProviderRegistry
{
    AiProviderDescriptor GetRequired(string? requestedProvider);
    IReadOnlyCollection<AiProviderDescriptor> GetEnabledProviders();
}
AiProviderRegistry.cs
Capability-aware registry
using Microsoft.Extensions.AI;
using Microsoft.Extensions.Options;

public sealed class AiProviderRegistry : IAiProviderRegistry
{
    private readonly IReadOnlyDictionary<string, AiProviderDescriptor> _providers;
    private readonly AiGatewayOptions _options;

    public AiProviderRegistry(IServiceProvider services, IOptions<AiGatewayOptions> options)
    {
        _options = options.Value;

        _providers = new Dictionary<string, AiProviderDescriptor>(StringComparer.OrdinalIgnoreCase)
        {
            [AiProviderNames.Ollama] = new(
                AiProviderNames.Ollama,
                "llama3.2",
                new(true, false, false, null, false),
                services.GetRequiredKeyedService<IChatClient>(AiProviderNames.Ollama)),

            [AiProviderNames.OpenAi] = new(
                AiProviderNames.OpenAi,
                "configured-model-name",
                new(true, true, true, null, false),
                services.GetRequiredKeyedService<IChatClient>(AiProviderNames.OpenAi)),

            [AiProviderNames.AzureOpenAi] = new(
                AiProviderNames.AzureOpenAi,
                "configured-deployment-name",
                new(true, true, true, null, false),
                services.GetRequiredKeyedService<IChatClient>(AiProviderNames.AzureOpenAi))
        };
    }

    public AiProviderDescriptor GetRequired(string? requestedProvider)
    {
        string name = requestedProvider ?? _options.DefaultProvider;

        if (!_options.AllowRequestProviderSelection && requestedProvider is not null &&
            !name.Equals(_options.DefaultProvider, StringComparison.OrdinalIgnoreCase))
        {
            throw new ProviderSelectionDisabledException();
        }

        return _providers.TryGetValue(name, out AiProviderDescriptor? descriptor)
            ? descriptor
            : throw new UnsupportedAiProviderException(name);
    }

    public IReadOnlyCollection<AiProviderDescriptor> GetEnabledProviders() =>
        _providers.Values.ToArray();
}
Production refinement: build descriptors from validated provider options and only include enabled providers. Hard-coded model names above keep the registry example readable; the complete application should obtain them from configuration.

Manage Bounded Conversation History

The portable mode stores framework ChatMessage instances in the gateway and sends a bounded history on every request.

IConversationStore.cs
Conversation store contract
using Microsoft.Extensions.AI;

public interface IConversationStore
{
    ValueTask<ConversationSnapshot> GetOrCreateAsync(
        string? conversationId,
        string provider,
        CancellationToken cancellationToken);

    ValueTask SaveAsync(
        string conversationId,
        string provider,
        IReadOnlyList<ChatMessage> messages,
        string? providerConversationId,
        CancellationToken cancellationToken);
}

public sealed record ConversationSnapshot(
    string ConversationId,
    string Provider,
    IReadOnlyList<ChatMessage> Messages,
    string? ProviderConversationId);
InMemoryConversationStore.cs
Bounded in-memory store
using System.Collections.Concurrent;
using Microsoft.Extensions.AI;
using Microsoft.Extensions.Options;

public sealed class InMemoryConversationStore : IConversationStore
{
    private sealed record Entry(
        string Provider,
        IReadOnlyList<ChatMessage> Messages,
        string? ProviderConversationId,
        DateTimeOffset ExpiresAt);

    private readonly ConcurrentDictionary<string, Entry> _entries = new();
    private readonly AiGatewayOptions _options;
    private readonly TimeProvider _timeProvider;

    public InMemoryConversationStore(
        IOptions<AiGatewayOptions> options,
        TimeProvider timeProvider)
    {
        _options = options.Value;
        _timeProvider = timeProvider;
    }

    public ValueTask<ConversationSnapshot> GetOrCreateAsync(
        string? conversationId,
        string provider,
        CancellationToken cancellationToken)
    {
        string id = string.IsNullOrWhiteSpace(conversationId)
            ? Guid.NewGuid().ToString("N")
            : conversationId;

        if (_entries.TryGetValue(id, out Entry? entry) &&
            entry.ExpiresAt > _timeProvider.GetUtcNow())
        {
            if (!entry.Provider.Equals(provider, StringComparison.OrdinalIgnoreCase))
                throw new ConversationProviderMismatchException(id, entry.Provider, provider);

            return ValueTask.FromResult(new ConversationSnapshot(
                id, entry.Provider, entry.Messages, entry.ProviderConversationId));
        }

        return ValueTask.FromResult(new ConversationSnapshot(
            id, provider, Array.Empty<ChatMessage>(), null));
    }

    public ValueTask SaveAsync(
        string conversationId,
        string provider,
        IReadOnlyList<ChatMessage> messages,
        string? providerConversationId,
        CancellationToken cancellationToken)
    {
        IReadOnlyList<ChatMessage> bounded = messages
            .TakeLast(_options.MaximumConversationMessages)
            .ToArray();

        _entries[conversationId] = new Entry(
            provider,
            bounded,
            providerConversationId,
            _timeProvider.GetUtcNow().AddMinutes(_options.ConversationLifetimeMinutes));

        return ValueTask.CompletedTask;
    }
}

Two valid state models

Gateway-managedProvider-managed
Store and resend bounded ChatMessage history.Store the returned provider conversation identifier.
Portable across providers.Conversation remains bound to the issuing provider.
Gateway controls retention.Provider controls remote state and retention semantics.
When ChatResponse.ConversationId is returned, pass it in the next ChatOptions.ConversationId instead of resending the same history. Never send one provider’s conversation identifier to another provider.

Implement the Gateway Service

AiGateway.cs
Gateway implementation
using System.Diagnostics;
using Microsoft.Extensions.AI;

public sealed class AiGateway(
    IAiProviderRegistry providers,
    IConversationStore conversations,
    TimeProvider timeProvider) : IAiGateway
{
    public async Task<GatewayResult> SendAsync(
        GatewayChatRequest request,
        CancellationToken cancellationToken)
    {
        AiProviderDescriptor descriptor = providers.GetRequired(request.Provider);
        ValidateCapabilities(request, descriptor);

        ConversationSnapshot conversation = await conversations.GetOrCreateAsync(
            request.ConversationId,
            descriptor.Name,
            cancellationToken);

        var messages = conversation.Messages.ToList();
        messages.Add(new ChatMessage(ChatRole.User, request.Message));

        var options = BuildOptions(request, conversation);
        long started = Stopwatch.GetTimestamp();

        ChatResponse response = await descriptor.Client.GetResponseAsync(
            messages,
            options,
            cancellationToken);

        messages.AddRange(response.Messages);
        await conversations.SaveAsync(
            conversation.ConversationId,
            descriptor.Name,
            messages,
            response.ConversationId,
            cancellationToken);

        return new GatewayResult(
            conversation.ConversationId,
            descriptor,
            response,
            Stopwatch.GetElapsedTime(started));
    }

    private static void ValidateCapabilities(
        GatewayChatRequest request,
        AiProviderDescriptor descriptor)
    {
        if (request.EnableTools && !descriptor.Capabilities.SupportsTools)
            throw new UnsupportedProviderCapabilityException(descriptor.Name, "tools");
    }

    private static ChatOptions BuildOptions(
        GatewayChatRequest request,
        ConversationSnapshot conversation)
    {
        var options = new ChatOptions
        {
            ConversationId = conversation.ProviderConversationId
        };

        if (request.EnableTools)
            options.Tools = [TimeTools.GetCurrentUtcTimeFunction];

        return options;
    }
}

The service resolves a descriptor once. Provider identity, capability checks, model metadata, telemetry labels, and the actual client therefore cannot drift apart.

Add One Safe Tool

TimeTools.cs
Deterministic time tool
using System.ComponentModel;
using Microsoft.Extensions.AI;

public static class TimeTools
{
    public static readonly AIFunction GetCurrentUtcTimeFunction =
        AIFunctionFactory.Create(GetCurrentUtcTime);

    [Description("Returns the current UTC time as an ISO 8601 string.")]
    public static string GetCurrentUtcTime() =>
        DateTimeOffset.UtcNow.ToString("O");
}

.UseFunctionInvocation() handles the function-call loop: the model requests the function, the middleware invokes it, returns the result, and lets the model continue.

Tool safety: models can invent or corrupt function arguments. Validate them exactly as you would validate public HTTP input. Avoid shell execution, arbitrary URLs, SQL, file deletion, email, purchases, or any irreversible operation in an introductory tutorial.

Map the HTTP Endpoints

ChatEndpoints.cs
Non-streaming endpoint
app.MapPost("/api/chat", async (
    GatewayChatRequest request,
    IAiGateway gateway,
    HttpContext httpContext) =>
{
    GatewayResult result = await gateway.SendAsync(
        request,
        httpContext.RequestAborted);

    ChatResponse response = result.Response;

    var dto = new GatewayChatResponse(
        result.ConversationId,
        result.Provider.Name,
        response.ModelId ?? result.Provider.Model,
        response.Text,
        response.FinishReason?.ToString(),
        response.Usage is null
            ? null
            : new TokenUsageDto(
                response.Usage.InputTokenCount,
                response.Usage.OutputTokenCount),
        (long)result.Duration.TotalMilliseconds);

    return Results.Ok(dto);
})
.RequireRateLimiting("ai-gateway");
ProviderEndpoints.cs
Providers endpoint
app.MapGet("/api/providers", (IAiProviderRegistry registry) =>
    Results.Ok(registry.GetEnabledProviders().Select(provider => new
    {
        provider.Name,
        provider.Model,
        provider.Capabilities
    })));

Stream Responses with Server-Sent Events

ChatEndpoints.cs
Streaming endpoint
app.MapPost("/api/chat/stream", async (
    GatewayChatRequest request,
    IAiGateway gateway,
    HttpContext context) =>
{
    context.Response.ContentType = "text/event-stream";
    context.Response.Headers.CacheControl = "no-cache";
    context.Response.Headers.Connection = "keep-alive";

    await foreach (GatewayStreamEvent item in gateway.StreamAsync(
        request,
        context.RequestAborted))
    {
        string json = JsonSerializer.Serialize(item);
        await context.Response.WriteAsync($"event: {item.Type}\n", context.RequestAborted);
        await context.Response.WriteAsync($"data: {json}\n\n", context.RequestAborted);
        await context.Response.Body.FlushAsync(context.RequestAborted);
    }
})
.RequireRateLimiting("ai-gateway");
AiGateway.Streaming.cs
Stream updates from IChatClient
public async IAsyncEnumerable<GatewayStreamEvent> StreamAsync(
    GatewayChatRequest request,
    [EnumeratorCancellation] CancellationToken cancellationToken)
{
    AiProviderDescriptor descriptor = providers.GetRequired(request.Provider);

    if (!descriptor.Capabilities.SupportsStreaming)
        throw new UnsupportedProviderCapabilityException(descriptor.Name, "streaming");

    ConversationSnapshot conversation = await conversations.GetOrCreateAsync(
        request.ConversationId, descriptor.Name, cancellationToken);

    var messages = conversation.Messages.ToList();
    messages.Add(new ChatMessage(ChatRole.User, request.Message));

    var assistantText = new StringBuilder();
    string? providerConversationId = conversation.ProviderConversationId;

    await foreach (ChatResponseUpdate update in descriptor.Client.GetStreamingResponseAsync(
        messages,
        new ChatOptions { ConversationId = providerConversationId },
        cancellationToken))
    {
        if (!string.IsNullOrEmpty(update.Text))
        {
            assistantText.Append(update.Text);
            yield return new GatewayStreamEvent("delta", update.Text);
        }

        providerConversationId ??= update.ConversationId;
    }

    messages.Add(new ChatMessage(ChatRole.Assistant, assistantText.ToString()));
    await conversations.SaveAsync(
        conversation.ConversationId,
        descriptor.Name,
        messages,
        providerConversationId,
        cancellationToken);

    yield return new GatewayStreamEvent("completed", conversation.ConversationId);
}
Always propagate HttpContext.RequestAborted. When the browser disconnects, the gateway should stop generating instead of continuing an abandoned, potentially billable request.

Build the Pipeline and Add Observability

Each keyed registration receives its own ChatClientBuilder. Cross-cutting behavior can be added once per client without changing the gateway service.

Program.cs
OpenTelemetry setup
using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource.AddService("SwitchableAiGateway"))
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddSource("Microsoft.Extensions.AI")
        .AddOtlpExporter())
    .WithMetrics(metrics => metrics
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddMeter("Microsoft.Extensions.AI")
        .AddOtlpExporter());
CaptureGuidance
Provider and modelLow-cardinality dimensions suitable for dashboards.
Duration and errorsMeasure complete and streaming operations separately.
UsageOnly report token counts when the adapter returns them.
Prompt/response contentDisabled by default; development-only opt-in.
Tool callsRecord function name and outcome, never secret arguments.
Generative-AI OpenTelemetry semantic conventions are evolving. Treat field names as version-sensitive and avoid building brittle business logic around telemetry attributes.

Handle Failures Honestly

FailureGateway response
Unsupported provider400 Bad Request
Invalid request400 Bad Request
Unsupported capability422 Unprocessable Entity
Context too large422 Unprocessable Entity
Provider authentication failure502 Bad Gateway
Provider rate limit429 or controlled 503
Provider unavailable503 Service Unavailable
Gateway timeout504 Gateway Timeout
Client disconnectedCancel without logging an application fault
Program.cs
Problem Details exception handler
app.UseExceptionHandler(errorApp => errorApp.Run(async context =>
{
    IExceptionHandlerFeature? feature =
        context.Features.Get<IExceptionHandlerFeature>();

    (int status, string title) = feature?.Error switch
    {
        UnsupportedAiProviderException => (400, "Unsupported AI provider"),
        ProviderSelectionDisabledException => (400, "Provider selection is disabled"),
        UnsupportedProviderCapabilityException => (422, "Capability not supported"),
        ConversationProviderMismatchException => (409, "Conversation belongs to another provider"),
        OperationCanceledException when context.RequestAborted.IsCancellationRequested
            => (499, "Client closed request"),
        TimeoutException => (504, "AI gateway timeout"),
        _ => (503, "AI provider unavailable")
    };

    context.Response.StatusCode = status;
    await Results.Problem(statusCode: status, title: title)
        .ExecuteAsync(context);
}));
Retries can cost money and duplicate side effects. A completed response lost in transit can be billed again if retried. Tool-enabled requests can repeat an action. Retry only failures known to be safe, before any side effect, and with strict limits.

Protect Secrets, Prompts, and Provider Selection

  • Accept provider names only from a fixed allowlist.
  • Never accept API keys, provider endpoints, or deployment names from the request body.
  • Authenticate the gateway and authorize expensive capabilities.
  • Apply request-size limits and per-user rate limits.
  • Redact prompts, responses, credentials, and tool arguments from logs.
  • Expire conversations and avoid unbounded storage.
  • Encode model output before displaying it as HTML.
  • Treat model output as untrusted data, not authority.
Program.cs
Built-in rate limiting
builder.Services.AddRateLimiter(options =>
{
    options.AddFixedWindowLimiter("ai-gateway", limiter =>
    {
        limiter.PermitLimit = 20;
        limiter.Window = TimeSpan.FromMinutes(1);
        limiter.QueueLimit = 0;
    });
});

app.UseRateLimiter();

Test Without Calling a Real Model

A fake client proves the gateway depends on the abstraction rather than network behavior.

FakeChatClient.cs
Small fake IChatClient
using Microsoft.Extensions.AI;

public sealed class FakeChatClient : IChatClient
{
    public ChatClientMetadata Metadata { get; } =
        new("fake", new Uri("https://fake.invalid"), "fake-model");

    public Task<ChatResponse> GetResponseAsync(
        IEnumerable<ChatMessage> messages,
        ChatOptions? options = null,
        CancellationToken cancellationToken = default)
    {
        cancellationToken.ThrowIfCancellationRequested();
        string prompt = messages.Last().Text;

        return Task.FromResult(new ChatResponse(
            new ChatMessage(ChatRole.Assistant, $"FAKE: {prompt}"))
        {
            ModelId = "fake-model",
            FinishReason = ChatFinishReason.Stop
        });
    }

    public async IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(
        IEnumerable<ChatMessage> messages,
        ChatOptions? options = null,
        [EnumeratorCancellation] CancellationToken cancellationToken = default)
    {
        foreach (string chunk in new[] { "FAKE", ": ", messages.Last().Text })
        {
            cancellationToken.ThrowIfCancellationRequested();
            yield return new ChatResponseUpdate(ChatRole.Assistant, chunk);
            await Task.Yield();
        }
    }

    public object? GetService(Type serviceType, object? serviceKey = null) =>
        serviceType.IsInstanceOfType(this) ? this : null;

    public void Dispose() { }
}
ProviderRegistryTests.cs
Provider selection test
[Fact]
public void Explicit_provider_returns_matching_descriptor()
{
    AiProviderDescriptor descriptor = registry.GetRequired("ollama");

    Assert.Equal("ollama", descriptor.Name);
    Assert.Same(ollamaClient, descriptor.Client);
}

[Fact]
public void Unknown_provider_is_rejected()
{
    Assert.Throws<UnsupportedAiProviderException>(
        () => registry.GetRequired("user-controlled-url"));
}

Add tests for default selection, disabled request-level switching, history truncation, provider-bound conversations, cancellation, streaming order, capability rejection, and secret redaction.

Run the Complete Gateway

Start locally
# Terminal 1
ollama serve

# Terminal 2
dotnet run --project src/SwitchableAiGateway.Api
Send through the default provider
curl -X POST http://localhost:5000/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "message": "Explain dependency injection in simple terms.",
    "conversationId": "demo-123"
  }'
Select a provider explicitly
curl -X POST http://localhost:5000/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "message": "Explain dependency injection in simple terms.",
    "provider": "openai"
  }'
Stream with SSE
curl -N -X POST http://localhost:5000/api/chat/stream \
  -H "Content-Type: application/json" \
  -d '{
    "message": "Give me three reasons to use IChatClient.",
    "provider": "ollama"
  }'
The payoff: change only AiGateway:DefaultProvider, repeat the same HTTP request, and watch the underlying model change without editing endpoint or gateway code.

Production Readiness Checklist

Configuration

  • Validated options
  • Enabled-provider filtering
  • Secrets outside JSON
  • Deterministic package versions

Safety

  • Gateway authentication
  • Provider allowlist
  • Request limits
  • Tool argument validation

Operations

  • Timeouts and cancellation
  • Conservative retries
  • Usage telemetry
  • Conversation expiration
  • Use a distributed store when multiple gateway instances must share conversations.
  • Keep provider integration tests separate and opt-in.
  • Review capability declarations whenever a deployment changes.
  • Test rate limits and failure translation with real provider sandboxes.
  • Set explicit budgets and quotas before exposing the gateway to untrusted users.

What to Build Next

This gateway is intentionally not a RAG system, agent framework, failover router, or billing platform. Those are follow-up concerns that can now build on a clean provider-neutral boundary.

  • Hybrid search: combine keyword and vector retrieval.
  • Evaluation harness: compare prompts and models against golden datasets.
  • Capability and cost routing: choose a provider based on policy.
  • Safe failover: retry only idempotent requests across providers.
  • Budgets and quotas: enforce organizational usage limits.

Frequently Asked Questions

Does IChatClient make every model interchangeable?

No. It standardizes the application contract, not model behavior. Capability metadata remains necessary for streaming, tools, usage reporting, context limits, and provider-managed conversations.

Why use OllamaSharp instead of Microsoft.Extensions.AI.Ollama?

The Microsoft-authored Ollama adapter is deprecated and recommends OllamaSharp. OllamaApiClient implements IChatClient and composes with ChatClientBuilder middleware.

Should the gateway resend history when ConversationId is returned?

No. For provider-managed state, pass the returned identifier through ChatOptions.ConversationId and send only new messages. Keep the conversation bound to that provider.

Can I add a normal HTTP retry policy?

Only cautiously. Repeating a completed generation can incur another charge, and tool calls can duplicate side effects. Restrict retries to known-safe transient failures.

Why are provider package versions not written throughout the tutorial?

Provider adapters evolve quickly. The sample pins tested versions centrally in Directory.Packages.props so one file communicates the verified dependency set.

Primary References

Back to Tutorials