Why Your Outbox Relay Stops Draining: Execution Strategies, Duplicate Publishing, Poison Messages & Idempotency

The Outbox Row Exists—So Why Is Nothing Moving?

A transactional outbox solves one specific problem: it records the business change and the intent to publish in the same database transaction. That prevents the classic gap where the order commits but the message is never recorded. It does not automatically make the relay reliable.

Many production failures happen after the row is safely in the table. The dispatcher starts an explicit transaction under a retrying execution strategy and throws. Two relay instances claim the same rows. A process publishes successfully and crashes before marking the row complete. One malformed payload becomes the oldest pending row forever. Or the consumer applies the same side effect twice because nobody implemented deduplication.

When an outbox stops draining, the table is usually telling you which part of the reliability chain is missing. The mistakes below cover the most common causes.

Mistake 1: Starting a Transaction Outside EF Core's Execution Strategy

Connection resiliency is commonly enabled with EnableRetryOnFailure. That changes how explicit transactions must be used. A retrying execution strategy cannot safely replay a user-created transaction unless the entire unit of work is handed to the strategy.

The usual symptom is an exception stating that the configured execution strategy does not support user-initiated transactions. Developers sometimes respond by disabling retries, which removes the symptom but also removes useful transient-failure handling. The correct fix is to obtain the strategy from Database.CreateExecutionStrategy() and execute the complete transaction inside ExecuteAsync.

The business row and outbox row must still be persisted together. If a retry repeats the delegate, the operation must also have a stable business key or other protection against an ambiguous commit.

var strategy = db.Database.CreateExecutionStrategy();

await strategy.ExecuteAsync(async () =>
{
    await using var transaction =
        await db.Database.BeginTransactionAsync(cancellationToken);

    db.Orders.Add(order);
    db.OutboxMessages.Add(
        OutboxMessage.From(
            new OrderPlaced(order.Id),
            Activity.Current?.Id));

    // One SaveChanges call writes both rows.
    await db.SaveChangesAsync(cancellationToken);

    await transaction.CommitAsync(cancellationToken);
});

Mistake 2: Treating Publish-Then-Mark as Exactly Once

Every polling relay has a dangerous but unavoidable crash window. It publishes the message to the broker, then updates ProcessedOnUtc. If the process stops after the broker accepts the message but before the database update commits, the row still appears pending.

On restart, the relay publishes the same message again. This is not proof that the outbox failed; it is the expected consequence of at-least-once delivery. Reversing the order does not help. Marking the row processed before publishing creates the opposite failure: a crash can permanently lose the message.

Use the outbox row's stable Id as the message id on the wire. Consumers then record that id in an inbox or processed-message table inside the same transaction as their side effects. A redelivery becomes a harmless lookup rather than a duplicate charge, email, or inventory adjustment.

await using var transaction =
    await consumerDb.Database.BeginTransactionAsync(ct);

bool alreadyProcessed = await consumerDb.ProcessedMessages
    .AnyAsync(x => x.MessageId == message.Id, ct);

if (alreadyProcessed)
{
    await transaction.RollbackAsync(ct);
    return;
}

consumerDb.Shipments.Add(
    Shipment.CreateForOrder(message.OrderId));

consumerDb.ProcessedMessages.Add(new ProcessedMessage
{
    MessageId = message.Id,
    ProcessedOnUtc = timeProvider.GetUtcNow()
});

await consumerDb.SaveChangesAsync(ct);
await transaction.CommitAsync(ct);

Mistake 3: Letting Every Relay Instance Select the Same Pending Rows

A simple query such as WHERE ProcessedOnUtc IS NULL ORDER BY OccurredOnUtc LIMIT 100 works with one dispatcher. Add a second instance and both can read the same batch before either marks it processed. The result is avoidable duplicate publishing and wasted broker traffic.

The rows must be claimed with database locking. PostgreSQL's FOR UPDATE SKIP LOCKED lets each dispatcher lock a different batch while skipping rows already held by another transaction. SQL Server commonly uses locking hints such as UPDLOCK and READPAST for the same queue-style pattern.

Keep the lock transaction short. Claim the batch, publish carefully, and update status without holding database locks through long or unpredictable work when the design can avoid it. For higher scale, introduce an explicit claimed state and lease timeout.

BEGIN;

SELECT *
FROM "OutboxMessages"
WHERE "ProcessedOnUtc" IS NULL
  AND "QuarantinedOnUtc" IS NULL
  AND "NextAttemptOnUtc" <= NOW()
ORDER BY "OccurredOnUtc"
FOR UPDATE SKIP LOCKED
LIMIT 100;

-- Publish the claimed batch and update each row's status
-- according to your chosen claim/lease design.

COMMIT;

Mistake 4: Retrying One Poison Message Forever

A relay often reads the oldest pending rows first. If the oldest row contains invalid JSON, an unknown contract type, or data the publisher can never accept, the same row can fail on every cycle. Without an attempt limit, that poison message can occupy the first slot indefinitely and make the outbox look frozen.

Record Attempts, the last Error, and NextAttemptOnUtc. Retry transient failures with backoff, but classify permanent failures and quarantine them after a defined maximum. The dispatcher should continue processing later rows instead of making one bad message the gatekeeper for the entire table.

Quarantine is not deletion. Keep the payload, error, type, trace id, and timestamps available for investigation and controlled replay after the underlying problem is corrected.

message.Attempts++;
message.Error = exception.Message;

if (message.Attempts >= options.MaxAttempts)
{
    message.QuarantinedOnUtc = timeProvider.GetUtcNow();
    message.NextAttemptOnUtc = null;
}
else
{
    TimeSpan delay = TimeSpan.FromSeconds(
        Math.Min(Math.Pow(2, message.Attempts), 300));

    message.NextAttemptOnUtc =
        timeProvider.GetUtcNow().Add(delay);
}

await db.SaveChangesAsync(ct);

Mistake 5: Assuming a SaveChangesInterceptor Makes the Whole Pattern Safe

A SaveChangesInterceptor is useful because it can convert domain events into outbox rows automatically. It prevents developers from remembering to add an outbox insert in every handler. But the interceptor only solves capture. It does not provide relay locking, retry policy, poison handling, idempotency, or broker delivery guarantees.

Another subtle mistake is clearing domain events before the save succeeds. If SaveChangesAsync fails and the execution strategy retries, the aggregate may no longer expose the event and the second attempt can write the business change without the outbox row. Clear events only after a successful save, or design the capture so retrying the operation is deterministic.

Treat the interceptor as one component in the pipeline, not as the outbox itself.

public sealed class OutboxInterceptor : SaveChangesInterceptor
{
    public override ValueTask<InterceptionResult<int>> SavingChangesAsync(
        DbContextEventData eventData,
        InterceptionResult<int> result,
        CancellationToken cancellationToken = default)
    {
        if (eventData.Context is not OrdersDbContext db)
        {
            return base.SavingChangesAsync(
                eventData, result, cancellationToken);
        }

        var aggregates = db.ChangeTracker
            .Entries<IHasDomainEvents>()
            .Select(x => x.Entity)
            .Where(x => x.DomainEvents.Count != 0)
            .ToArray();

        foreach (var aggregate in aggregates)
        {
            foreach (var domainEvent in aggregate.DomainEvents)
            {
                db.OutboxMessages.Add(
                    OutboxMessage.From(domainEvent, Activity.Current?.Id));
            }

            // Clear only with a retry-safe design.
            // A production implementation may defer clearing until save succeeds.
        }

        return base.SavingChangesAsync(
            eventData, result, cancellationToken);
    }
}

How to Diagnose a Relay That Has Stopped Draining

Start with the oldest pending row, not the newest application log. Its attempt count, next retry time, error text, and age usually reveal whether the relay is blocked, backing off, or never claiming work. Then inspect whether more than one dispatcher is reading the same ids and whether processed timestamps advance after broker acknowledgements.

A healthy outbox is not one with zero rows at every moment. It is one where backlog age remains within an expected window, transient failures recover, poison rows leave the active queue, duplicates are harmless, and processed data is eventually cleaned up.

[ ] Explicit transactions run inside CreateExecutionStrategy().ExecuteAsync
[ ] Business row + outbox row share one transaction and one SaveChanges
[ ] Every outbox row has a stable message id
[ ] Concurrent dispatchers claim rows with SKIP LOCKED / READPAST
[ ] Publish-success / mark-processed crash window is understood
[ ] Consumers deduplicate message ids transactionally
[ ] Attempts, Error, and NextAttemptOnUtc are recorded
[ ] Poison rows are quarantined after MaxAttempts
[ ] Oldest pending row age is monitored
[ ] Processed rows are removed on a retention schedule
[ ] Trace context is preserved without logging full sensitive payloads