Your payment service is down. Without resilience, 10,000 requests pile up, threads block, and your entire API falls over. With Polly, circuits open in 50ms, fallbacks return cached data, and bulkheads isolate the blast radius. Your API stays up. This isn't defensive coding. It's production reality.
Most Polly tutorials show you WaitAndRetry and stop. Production systems need coordinated timeouts, jittered backoffs, circuit breakers with health checks, bulkhead isolation, and hedged requests. You could patch policies together and hope. Or you could learn the patterns that survive outages. This tutorial shows you the battle-tested patterns.
You'll build a production-ready Orders API that calls Inventory and Payments services with complete resilience:
Cloud services fail. Networks hiccup. Dependencies restart. Without resilience, transient failures cascade into full outages. Your API becomes the weakest link. Resilience patterns keep you up when everything around you is falling down.
Cloud providers guarantee 99.9% uptime. That's 43 minutes of downtime per month. Your app makes 100 dependency calls per request. One failing dependency kills availability. Timeouts prevent hanging. Retries smooth over blips. Circuit breakers stop the bleeding.
If your service promises 99.95% uptime (21 minutes/month), and each dependency is 99.9%, you burn through your error budget fast. Resilience patterns (retries, fallbacks) help you meet SLAs even when dependencies don't.
Polly v8 is a full rewrite. The old ISyncPolicy and IAsyncPolicy are gone. Pipelines replace PolicyWrap. Telemetry is built-in. The API is cleaner, faster, and works seamlessly with HttpClientFactory and OpenTelemetry.
Start with a minimal API that calls a downstream service. You'll add a basic timeout policy first, then layer on retries, circuits, and fallbacks.
Polly v8 is modular. Core policies live in Polly.Core. HTTP integration is in Polly.Extensions.Http. Telemetry hooks into OpenTelemetry via Polly.Extensions.
dotnet new webapi -n OrdersApi --use-controllers false
cd OrdersApi
# Core resilience pipelines and DI integration
dotnet add package Polly.Core --version 8.*
dotnet add package Polly.Extensions --version 8.*
# Recommended HttpClient integration for .NET 8
dotnet add package Microsoft.Extensions.Http.Resilience --version 8.*
Use a typed client for clean dependency injection. Register policies via AddResilienceHandler from the new resilience extensions.
using Polly.Timeout;
var builder = WebApplication.CreateBuilder(args);
builder.Services
.AddHttpClient<InventoryClient>(client =>
{
client.BaseAddress = new Uri("https://inventory-api.example.com");
// Let the resilience pipeline own timeouts.
client.Timeout = Timeout.InfiniteTimeSpan;
})
.AddResilienceHandler("inventory-pipeline", pipeline =>
{
pipeline.AddTimeout(new TimeoutStrategyOptions
{
Timeout = TimeSpan.FromSeconds(3),
Name = "inventory-timeout"
});
});
var app = builder.Build();
app.MapGet("/orders/{id:int}", async (
int id,
InventoryClient inventory,
CancellationToken ct) =>
{
var stock = await inventory.GetStockAsync(id, ct);
return Results.Ok(new { OrderId = id, Stock = stock });
});
app.Run();
public sealed class InventoryClient(HttpClient http)
{
public async Task<int?> GetStockAsync(
int productId,
CancellationToken ct = default)
{
using var response = await http.GetAsync($"/api/stock/{productId}", ct);
response.EnsureSuccessStatusCode();
return await response.Content.ReadFromJsonAsync<int?>(cancellationToken: ct);
}
}HttpClient.Timeout is the overall budget. The Polly timeout inside retries is per-try. Set HttpClient.Timeout higher than (retry_count * per_try_timeout) to avoid premature cancellation.
Timeouts are your first line of defense. Without them, slow dependencies block threads indefinitely. With timeouts, you fail fast and free resources. Polly's timeout strategy integrates with CancellationToken propagation.
Use a timeout outside retries to cap total time spent. This prevents infinite retry loops from consuming all request budget.
builder.Services.AddHttpClient<PaymentsClient>(client =>
{
client.BaseAddress = new Uri("https://payments-api.example.com");
})
.AddResilienceHandler("payments-pipeline", (pipelineBuilder, context) =>
{
// Outer timeout: total request budget
pipelineBuilder.AddTimeout(new TimeoutStrategyOptions
{
Timeout = TimeSpan.FromSeconds(5),
Name = "payments-overall-timeout"
});
// Retry with per-try timeout
pipelineBuilder.AddRetry(new Polly.Retry.RetryStrategyOptions
{
MaxRetryAttempts = 2,
Delay = TimeSpan.FromMilliseconds(500),
BackoffType = Polly.DelayBackoffType.Exponential,
UseJitter = true,
Name = "payments-retry"
});
// Inner timeout: per-try budget
pipelineBuilder.AddTimeout(new TimeoutStrategyOptions
{
Timeout = TimeSpan.FromSeconds(2),
Name = "payments-per-try-timeout"
});
});
Pass CancellationToken through all layers. When a timeout fires, Polly cancels the token. Your code must check ct.IsCancellationRequested and honor it.
public sealed class PaymentsClient(HttpClient http)
{
public async Task<PaymentResult> ChargeAsync(
decimal amount,
CancellationToken ct)
{
using var response = await http.PostAsJsonAsync(
"/api/charge",
new { amount },
ct);
if (!response.IsSuccessStatusCode)
{
throw new PaymentException(
$"Payment failed: {(int)response.StatusCode} {response.StatusCode}");
}
return await response.Content.ReadFromJsonAsync<PaymentResult>(
cancellationToken: ct)
?? throw new PaymentException("Empty response");
}
}
public sealed record PaymentResult(string TransactionId, string Status);Polly v8 throws TimeoutRejectedException on timeout. HttpClient throws TaskCanceledException. Catch both if you need to distinguish timeout from user cancellation.
Retries smooth over transient failures. But naive retries hammer failing services and cause thundering herds. Use decorrelated jitter, bound attempts, and only retry idempotent operations.
Exponential backoff without jitter causes retry storms. All clients retry at the same intervals. Decorrelated jitter spreads retries randomly across time.
using Polly;
using Polly.Retry;
builder.Services
.AddHttpClient<InventoryClient>(client =>
client.BaseAddress = new Uri("https://inventory-api.example.com"))
.AddResilienceHandler("inventory-retry", pipeline =>
{
pipeline.AddRetry(new RetryStrategyOptions<HttpResponseMessage>
{
MaxRetryAttempts = 3,
Delay = TimeSpan.FromMilliseconds(200),
BackoffType = DelayBackoffType.Exponential,
UseJitter = true,
Name = "inventory-retry",
ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
.Handle<HttpRequestException>()
.HandleResult(response =>
(int)response.StatusCode >= 500 ||
response.StatusCode == HttpStatusCode.RequestTimeout ||
response.StatusCode == HttpStatusCode.TooManyRequests),
OnRetry = args =>
{
Console.WriteLine(
$"Retry {args.AttemptNumber + 1} after {args.RetryDelay}");
return ValueTask.CompletedTask;
}
});
});Only retry safe HTTP methods (GET, PUT, DELETE) or requests with idempotency keys. POST without idempotency can create duplicate orders.
public static class HttpRequestMessageExtensions
{
public static bool IsSafeToRetry(this HttpRequestMessage request)
{
if (request.Method is { } method &&
(method == HttpMethod.Get ||
method == HttpMethod.Head ||
method == HttpMethod.Options ||
method == HttpMethod.Trace))
{
return true;
}
// PUT and DELETE are defined as idempotent, but only retry them when
// your server-side implementation preserves that guarantee.
if (request.Method == HttpMethod.Put || request.Method == HttpMethod.Delete)
{
return true;
}
return request.Method == HttpMethod.Post &&
request.Headers.Contains("Idempotency-Key");
}
}
builder.Services
.AddHttpClient<OrdersClient>()
.AddResilienceHandler("orders-retry", pipeline =>
{
pipeline.AddRetry(new RetryStrategyOptions<HttpResponseMessage>
{
MaxRetryAttempts = 2,
ShouldHandle = args =>
{
var request = args.Outcome.Result?.RequestMessage;
if (request is null || !request.IsSafeToRetry())
{
return ValueTask.FromResult(false);
}
var transientResponse = args.Outcome.Result is { } response &&
((int)response.StatusCode >= 500 ||
response.StatusCode == HttpStatusCode.RequestTimeout ||
response.StatusCode == HttpStatusCode.TooManyRequests);
var transientException =
args.Outcome.Exception is HttpRequestException;
return ValueTask.FromResult(transientResponse || transientException);
}
});
});Don't retry all 5xx errors. 500 Internal Server Error often means a bug, not a transient issue. Retry 503 Service Unavailable and network failures. Skip 500, 501, 505.
Circuit breakers fail fast when a service is persistently down. Instead of waiting for timeouts, the breaker opens and rejects requests immediately. This protects your app and gives the downstream service time to recover.
Track failure rate over a rolling window. If failures exceed a threshold, open the circuit. After a cooldown, enter half-open state and test with a single probe request.
using Polly;
using Polly.CircuitBreaker;
builder.Services
.AddHttpClient<PaymentsClient>()
.AddResilienceHandler("payments-circuit", pipeline =>
{
pipeline.AddCircuitBreaker(
new CircuitBreakerStrategyOptions<HttpResponseMessage>
{
FailureRatio = 0.5,
MinimumThroughput = 10,
SamplingDuration = TimeSpan.FromSeconds(30),
BreakDuration = TimeSpan.FromSeconds(15),
Name = "payments-circuit",
ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
.Handle<HttpRequestException>()
.HandleResult(response =>
(int)response.StatusCode >= 500),
OnOpened = args =>
{
Console.WriteLine($"Circuit opened for {args.BreakDuration}");
return ValueTask.CompletedTask;
},
OnClosed = _ =>
{
Console.WriteLine("Circuit closed");
return ValueTask.CompletedTask;
},
OnHalfOpened = _ =>
{
Console.WriteLine("Circuit half-open; testing recovery");
return ValueTask.CompletedTask;
}
});
});Expose circuit breaker state via ASP.NET Core health checks. Monitoring tools can alert when circuits open.
using Microsoft.Extensions.Diagnostics.HealthChecks;
builder.Services.AddHealthChecks()
.AddCheck("payments-circuit", () =>
{
// Access circuit state via DI (requires custom tracking)
// For now, return healthy - production should check actual state
return HealthCheckResult.Healthy("Circuit closed");
});
app.MapHealthChecks("/health");Store breaker state in a singleton service. Update it in OnOpened/OnClosed callbacks. Query state in health checks and expose via /health endpoint.
When all retries fail and circuits open, fallbacks provide degraded responses. Return cached data, default values, or partial results. Never hide persistent failures from users—make degradation visible.
For read-only requests, serve stale cache when the service is down. Mark responses with X-Fallback: true header.
using Microsoft.Extensions.Caching.Memory;
using Polly;
using Polly.Fallback;
builder.Services.AddMemoryCache();
builder.Services
.AddHttpClient<InventoryClient>()
.AddResilienceHandler("inventory-fallback", (pipeline, context) =>
{
var cache = context.ServiceProvider.GetRequiredService<IMemoryCache>();
pipeline.AddFallback(new FallbackStrategyOptions<HttpResponseMessage>
{
ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
.Handle<HttpRequestException>()
.HandleResult(response => !response.IsSuccessStatusCode),
FallbackAction = args =>
{
var key = args.Outcome.Result?.RequestMessage?.RequestUri?.ToString();
if (key is not null &&
cache.TryGetValue(key, out CachedInventoryResponse? cached) &&
cached is not null)
{
var response = new HttpResponseMessage(HttpStatusCode.OK)
{
Content = JsonContent.Create(cached.Payload)
};
response.Headers.TryAddWithoutValidation("X-Fallback", "cache");
return Outcome.FromResultAsValueTask(response);
}
var unavailable = new HttpResponseMessage(
HttpStatusCode.ServiceUnavailable)
{
Content = JsonContent.Create(new
{
error = "Service unavailable and no cached value exists"
})
};
unavailable.Headers.TryAddWithoutValidation("X-Fallback", "empty");
return Outcome.FromResultAsValueTask(unavailable);
}
});
});
public sealed record CachedInventoryResponse(object Payload);For composite endpoints that call multiple services, return partial data when some services fail. Users get what's available instead of total failure.
app.MapGet("/orders/{id:int}/details", async (
int id,
InventoryClient inventory,
PaymentsClient payments,
CancellationToken ct) =>
{
var details = new OrderDetails { OrderId = id };
try
{
details.Stock = await inventory.GetStockAsync(id, ct);
}
catch (Exception ex) when (ex is not OperationCanceledException)
{
details.Warnings.Add("Inventory is temporarily unavailable.");
}
try
{
details.PaymentStatus = await payments.GetStatusAsync(id, ct);
}
catch (Exception ex) when (ex is not OperationCanceledException)
{
details.Warnings.Add("Payment status is temporarily unavailable.");
}
return Results.Ok(details);
});
public sealed class OrderDetails
{
public int OrderId { get; init; }
public int? Stock { get; set; }
public string PaymentStatus { get; set; } = "Unknown";
public List<string> Warnings { get; } = [];
}Fallbacks keep your app alive during brief outages. But if a service is down for hours, users deserve to know. Add timestamps to fallback data and alert thresholds for stale caches.
Bulkheads limit concurrent requests to a service. If one dependency is slow, it can't exhaust your thread pool and starve other services. Use separate bulkheads for third-party vs internal calls.
Polly v8 uses rate limiter strategies for bulkheads. Configure max concurrent requests and queue depth.
using Polly.RateLimiting;
builder.Services.AddHttpClient<PaymentsClient>()
.AddResilienceHandler("payments-bulkhead", (pb, ctx) =>
{
pb.AddConcurrencyLimiter(new ConcurrencyLimiterOptions
{
PermitLimit = 10, // Max 10 concurrent requests
QueueLimit = 5, // Queue up to 5 waiting requests
Name = "payments-bulkhead",
OnRejected = args =>
{
Console.WriteLine($"Request rejected by bulkhead, queue full");
return ValueTask.CompletedTask;
}
});
});External APIs are less reliable than internal services. Give them separate bulkheads so slow third-party calls don't block internal traffic.
// Internal service: generous limits
builder.Services.AddHttpClient<InventoryClient>()
.AddResilienceHandler("inventory-bulkhead", (pb, ctx) =>
{
pb.AddConcurrencyLimiter(new ConcurrencyLimiterOptions
{
PermitLimit = 50,
QueueLimit = 20,
Name = "inventory-bulkhead-internal"
});
});
// Third-party API: tight limits
builder.Services.AddHttpClient<ShippingClient>()
.AddResilienceHandler("shipping-bulkhead", (pb, ctx) =>
{
pb.AddConcurrencyLimiter(new ConcurrencyLimiterOptions
{
PermitLimit = 5,
QueueLimit = 2,
Name = "shipping-bulkhead-external"
});
});Track queued vs rejected requests. High rejection rates mean you need higher limits or the service is too slow. High queue depth means the service can't keep up with load.
Hedging fires parallel requests to reduce tail latency. Wait for p95-p99 latency, then send a second request. The first response wins, and the slower one is cancelled. Use only for idempotent reads.
Configure hedge delay based on observed latency percentiles. Too short and you double load. Too long and hedging doesn't help.
using Polly;
using Polly.Hedging;
builder.Services.AddResiliencePipeline<HttpResponseMessage>(
"catalog-hedging",
pipeline =>
{
pipeline.AddHedging(new HedgingStrategyOptions<HttpResponseMessage>
{
MaxHedgedAttempts = 2,
Delay = TimeSpan.FromMilliseconds(500),
Name = "catalog-hedging",
ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
.Handle<HttpRequestException>()
.HandleResult(response =>
(int)response.StatusCode >= 500),
OnHedging = args =>
{
Console.WriteLine($"Starting hedge {args.AttemptNumber + 1}");
return ValueTask.CompletedTask;
}
});
});
// Hedging is appropriate only for safe, idempotent operations. For HTTP,
// prefer AddStandardHedgingHandler(), which can route attempts across endpoints.Hedging doubles request volume. Only hedge read-only endpoints. Never hedge writes or non-idempotent operations. Monitor hedge rates and adjust delay if load becomes a problem.
Hedging helps latency at the cost of throughput. If hedge delay is too aggressive, you can double the load on a service. Start conservative (p99 delay), monitor backend load, and tune based on capacity.
Policies execute in the order they're added to the pipeline. Order matters. The standard pattern is: Timeout → Retry → Circuit → Fallback. This ensures timeouts apply to retries, circuits track retry failures, and fallbacks catch everything.
Build pipelines inside-out. Inner policies execute first per request. Outer policies wrap inner ones.
builder.Services.AddResiliencePipeline<HttpResponseMessage>(
"payments-full-pipeline",
pipeline =>
{
// Strategies execute in registration order, from outermost to innermost.
pipeline.AddTimeout(TimeSpan.FromSeconds(10));
pipeline.AddFallback(new FallbackStrategyOptions<HttpResponseMessage>
{
ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
.Handle<Exception>(ex => ex is not OperationCanceledException)
.HandleResult(response => !response.IsSuccessStatusCode),
FallbackAction = _ => Outcome.FromResultAsValueTask(
new HttpResponseMessage(HttpStatusCode.ServiceUnavailable))
});
pipeline.AddCircuitBreaker(
new CircuitBreakerStrategyOptions<HttpResponseMessage>
{
FailureRatio = 0.5,
SamplingDuration = TimeSpan.FromSeconds(30),
MinimumThroughput = 10,
BreakDuration = TimeSpan.FromSeconds(15)
});
pipeline.AddRetry(new RetryStrategyOptions<HttpResponseMessage>
{
MaxRetryAttempts = 3,
Delay = TimeSpan.FromMilliseconds(200),
BackoffType = DelayBackoffType.Exponential,
UseJitter = true
});
pipeline.AddTimeout(TimeSpan.FromSeconds(3));
});
Timeout → Retry: Per-try timeout prevents one slow attempt from blocking retries.
Retry → Circuit: Circuit tracks failures after retries exhaust. Don't open circuit on first failure if retries would succeed.
Circuit → Fallback: Fallback catches circuit-open exceptions and serves degraded responses.
Outer Timeout: Caps total time including all retries and fallback execution.
Use the OnRetry, OnCircuitOpened, OnFallback callbacks to log pipeline execution. This shows you which policy handled each request and helps tune thresholds.
Not all endpoints need the same resilience. Critical writes need tight timeouts and no retries. Background jobs tolerate longer delays. Use endpoint metadata to select policies dynamically.
Tag endpoints with metadata, then resolve the policy pipeline based on that metadata.
// Define policy keys as metadata
public record ResiliencePolicyMetadata(string PolicyKey);
// Register named pipelines
builder.Services.AddResiliencePipeline("critical", pb =>
{
pb.AddTimeout(TimeSpan.FromSeconds(2));
// No retries for critical writes
});
builder.Services.AddResiliencePipeline("standard", pb =>
{
pb.AddTimeout(TimeSpan.FromSeconds(5));
pb.AddRetry(new RetryStrategyOptions { MaxRetryAttempts = 3 });
pb.AddCircuitBreaker(new CircuitBreakerStrategyOptions
{
FailureRatio = 0.5,
SamplingDuration = TimeSpan.FromSeconds(30),
MinimumThroughput = 10,
BreakDuration = TimeSpan.FromSeconds(15)
});
});
// Apply to endpoints
app.MapPost("/orders", async (CreateOrderRequest req) =>
{
// Critical path
return Results.Created($"/orders/123", new { Id = 123 });
})
.WithMetadata(new ResiliencePolicyMetadata("critical"));
app.MapGet("/orders/{id}", async (int id) =>
{
// Standard path
return Results.Ok(new { Id = id });
})
.WithMetadata(new ResiliencePolicyMetadata("standard"));Create extension methods for clean policy wiring.
public static class ResilienceExtensions
{
public static RouteHandlerBuilder WithCriticalResilience(this RouteHandlerBuilder builder)
{
return builder.WithMetadata(new ResiliencePolicyMetadata("critical"));
}
public static RouteHandlerBuilder WithStandardResilience(this RouteHandlerBuilder builder)
{
return builder.WithMetadata(new ResiliencePolicyMetadata("standard"));
}
}
// Usage
app.MapPost("/orders", handler).WithCriticalResilience();
app.MapGet("/orders/{id}", handler).WithStandardResilience();Resilience without metrics is blind guessing. Use OpenTelemetry to track policy executions, retry counts, circuit state, and hedge rates. Set up dashboards and alerts. Tune thresholds based on error budgets and SLOs.
Polly v8 emits telemetry events automatically. Wire it into OpenTelemetry for distributed tracing.
using OpenTelemetry.Trace;
using OpenTelemetry.Metrics;
builder.Services.AddOpenTelemetry()
.WithTracing(tracerBuilder =>
{
tracerBuilder
.AddAspNetCoreInstrumentation()
.AddHttpClientInstrumentation()
.AddSource("Polly")
.AddOtlpExporter();
})
.WithMetrics(meterBuilder =>
{
meterBuilder
.AddAspNetCoreInstrumentation()
.AddHttpClientInstrumentation()
.AddMeter("Polly")
.AddOtlpExporter();
});Track retry attempts, circuit opens, and fallback usage with custom metrics.
using System.Diagnostics.Metrics;
public sealed class ResilienceMetrics(IMeterFactory meterFactory)
{
private readonly Meter _meter = meterFactory.Create("OrdersApi.Resilience");
private readonly Counter<long> _retryCounter =
meterFactory.Create("OrdersApi.Resilience")
.CreateCounter<long>("retry_attempts");
public void RecordRetry(string pipelineName) =>
_retryCounter.Add(1, new KeyValuePair<string, object?>(
"resilience.pipeline", pipelineName));
}
builder.Services.AddSingleton<ResilienceMetrics>();
builder.Services.AddResiliencePipeline("payments", (pipeline, context) =>
{
var metrics = context.ServiceProvider.GetRequiredService<ResilienceMetrics>();
pipeline.AddRetry(new RetryStrategyOptions
{
MaxRetryAttempts = 3,
OnRetry = _ =>
{
metrics.RecordRetry("payments");
return ValueTask.CompletedTask;
}
});
});Monitor: retry rate, circuit breaker state changes, fallback usage, request latency (p50/p95/p99), error rate by policy. Set alerts on circuit opens and fallback spikes.
Resilience policies are worthless if they're never tested under failure. Inject faults in staging. Add variable latency. Simulate service restarts. Validate that your SLOs hold up.
Build middleware that randomly injects failures and latency. Enable it via feature flag in non-production environments.
public sealed class ChaosMiddleware(RequestDelegate next)
{
public async Task InvokeAsync(HttpContext context)
{
// Enable only in a controlled test environment.
if (Random.Shared.NextDouble() < 0.10)
{
context.Response.StatusCode = StatusCodes.Status503ServiceUnavailable;
await context.Response.WriteAsync("Chaos: Service Unavailable");
return;
}
if (Random.Shared.NextDouble() < 0.20)
{
await Task.Delay(
Random.Shared.Next(200, 800),
context.RequestAborted);
}
await next(context);
}
}
if (app.Environment.IsStaging())
{
app.UseMiddleware<ChaosMiddleware>();
}Use k6 or Bombardier to generate realistic load. Validate that retries don't cause cascading failures.
import http from 'k6/http';
import { check, sleep } from 'k6';
export const options = {
stages: [
{ duration: '30s', target: 50 }, // Ramp up
{ duration: '2m', target: 100 }, // Sustained load
{ duration: '30s', target: 0 }, // Ramp down
],
thresholds: {
http_req_duration: ['p(95)<500', 'p(99)<1000'],
http_req_failed: ['rate<0.05'], // 95% success
},
};
export default function () {
const res = http.get('https://orders-api.example.com/orders/123');
check(res, {
'status is 200': (r) => r.status === 200,
'response time < 500ms': (r) => r.timings.duration < 500,
});
sleep(1);
}Some teams run chaos in production (Netflix, AWS). Start in staging. Once you trust your resilience, consider controlled production chaos during low-traffic windows. Always have kill switches.
Resilience policies are code. Treat them like production config. Use feature flags for policy changes. Roll out new thresholds with canary deployments. Monitor policy effectiveness and iterate.
Store policy settings in appsettings.json. Use the Options pattern for reloadable config.
{
"Resilience": {
"Payments": {
"TimeoutSeconds": 3,
"RetryCount": 3,
"CircuitBreakerFailureRatio": 0.5,
"CircuitBreakerSamplingSeconds": 30,
"CircuitBreakerBreakSeconds": 15
},
"Inventory": {
"TimeoutSeconds": 2,
"RetryCount": 2,
"CircuitBreakerFailureRatio": 0.3,
"CircuitBreakerSamplingSeconds": 20,
"CircuitBreakerBreakSeconds": 10
}
}
}public sealed class ResilienceOptions
{
public int TimeoutSeconds { get; init; } = 3;
public int RetryCount { get; init; } = 3;
public double CircuitBreakerFailureRatio { get; init; } = 0.5;
public int CircuitBreakerSamplingSeconds { get; init; } = 30;
public int CircuitBreakerBreakSeconds { get; init; } = 15;
}
builder.Services.Configure<ResilienceOptions>(
"Payments",
builder.Configuration.GetSection("Resilience:Payments"));
builder.Services
.AddHttpClient<PaymentsClient>()
.AddResilienceHandler("payments", (pipeline, context) =>
{
var options = context.ServiceProvider
.GetRequiredService<IOptionsMonitor<ResilienceOptions>>()
.Get("Payments");
pipeline.AddTimeout(TimeSpan.FromSeconds(options.TimeoutSeconds));
pipeline.AddRetry(new RetryStrategyOptions<HttpResponseMessage>
{
MaxRetryAttempts = options.RetryCount
});
});Wrap policy registration in feature flag checks. If a policy causes issues, toggle it off without redeploying. Use LaunchDarkly, Azure App Config, or simple environment variables.
Use Polly for external HTTP calls and inter-service communication where transient failures are expected. Internal loopback calls or trusted services with stable SLAs may not need full resilience. Start with circuit breakers for external dependencies, then add retries and timeouts as needed. Avoid over-engineering stable, low-latency services.
Start with 2–3 retries using decorrelated jitter backoff. More retries increase latency under load and can overwhelm failing services. Use retries only for idempotent operations (GET, PUT with idempotency keys). POST without idempotency should fail fast or use hedging instead.
Circuit breakers fail fast when a service is down, preventing cascading failures and long timeouts. Users see immediate errors instead of hanging requests. Pair circuits with fallbacks (cached data, degraded responses) to maintain partial functionality. Monitor breaker state and alert on transitions to open state.
Use hedging for read-only, idempotent requests where tail latency matters more than total request volume. Hedging fires parallel requests after p95-p99 delay, canceling slower ones. Use retries for write operations, non-idempotent requests, or when you want to avoid duplicate work. Hedging increases load; retries don't.
Start with conservative settings (high thresholds, low retry counts). Use feature flags to toggle policies dynamically. Monitor policy events (retries, breaker opens) with OpenTelemetry. Run chaos tests in staging to validate settings. Adjust thresholds based on error budgets and SLOs, not gut feel. Roll out changes gradually with canary deployments.