JasperFx has helped several of our clients move toward modular monolith architectures and the local queues feature in this post has frequently been a part of that strategy

The software industry has become disillusioned with micro-services and started trying to utilize a modular monolith style instead, and I've written about that before. In a modular monolith you're trying to divide your larger system into logical modules that are deployed in a single process. You're trying to keep the modules decoupled from each other, but there are frequently operations in your system that will result in writing data to more than one module. Consider a system where operations in an Orders module frequently need to tell Shipping and Billing modules that an order was placed or cancelled. How does that message get there?

You could allow the Orders module to just call synchronously straight through to the other modules, but I'd argue that in that case you just have an old-fashioned monolithic codebase with all the same potential problems that led us to micro-services in the first place. More likely, you'll be interested in applying some kind of "Eventual Consistency" where the Orders module uses asynchronous messaging to notify the Shipping and Billing modules of changes with some sort of guarantee that the messages will actually be processed.

In the .NET world, the reflexive answer is usually a MediatR notification (or one of its umpteen dozen copycat projects). At JasperFx we'll argue that that's the wrong tool because of its weak delivery guarantees and lack of durability. Conveniently enough though, Wolverine's local queues are a much better fit. Local queues can be backed by Wolverine's transactional inbox, so they're durable against process failures. The local queues in Wolverine come with the same error handling, retries, and dead lettering as any other Wolverine endpoint. They're instrumented with OpenTelemetry tracing and performance metrics out of the box. And you get real control over sequencing, parallelism, partitioning, and circuit breaking.

Unlike the in-memory queues in other .NET messaging frameworks, Wolverine's durable local queues are designed and documented for production use.

A thirty-second introduction to local queues

In Wolverine, any message that your application handles can be published to an in-process queue. There's no broker, no extra infrastructure, and very little ceremony. Returning a message from a handler "cascades" it, and Wolverine routes it to any local handlers:

public static class PlaceOrderHandler
{
    // Returning OrderPlaced "cascades" it as a new message. Wolverine
    // publishes it to a local queue after this handler completes, and
    // after the transaction commits if you're using transactional middleware
    public static OrderPlaced Handle(PlaceOrder command, IDocumentSession session)
    {
        var order = new Order(command.OrderId, command.Items);
        session.Store(order);

        return new OrderPlaced(command.OrderId);
    }
}

// Over in the Shipping module
public static class OrderPlacedHandler
{
    public static ShipmentRequested Handle(OrderPlaced @event)
        => new ShipmentRequested(@event.OrderId);
}

By default, Wolverine gives every message type its own local queue named after the message type, and every local queue is a real Wolverine endpoint. That last part is the whole story, as it turns out. A local queue gets everything a RabbitMQ or Azure Service Bus listener gets: error handling policies, durability, back pressure, parallelism controls, circuit breakers, and telemetry.

Local queues and modular monoliths

Local queues fit modular monoliths very naturally, and Wolverine has been adding support specifically for that style of architecture for the past couple of years. The most important piece for this discussion is what happens when more than one module handles the same event. Out of the box, Wolverine combines all the handlers for one message type into a single logical handler and a single transaction. That's great for a traditional application, but not so great when the Shipping and Billing modules both react to OrderPlaced and shouldn't be able to fail each other. For that, you'll want the Separated handler behavior:

var builder = Host.CreateApplicationBuilder();

builder.Services.AddMarten(opts =>
    {
        opts.Connection(builder.Configuration.GetConnectionString("marten")!);
    })
    // Adds a PostgreSQL-backed transactional inbox/outbox using the same database
    .IntegrateWithWolverine();

builder.UseWolverine(opts =>
{
    // Each module's handler for the same message type runs separately,
    // on its own local queue, with its own transaction and its own retries
    opts.MultipleHandlerBehavior = MultipleHandlerBehavior.Separated;

    // Treat the same message delivered to two different handlers as two
    // different messages in the inbox, rather than a duplicate
    opts.Durability.MessageIdentity = MessageIdentity.IdAndDestination;

    // Share one envelope storage schema across all the modules
    opts.Durability.MessageStorageSchemaName = "wolverine";

    opts.Policies.AutoApplyTransactions();

    // And make every local queue durable. More on this below!
    opts.Policies.UseDurableLocalQueues();
});

With that configuration, one OrderPlaced event is delivered separately to each module's handler, each on its own local queue. You get:

• Independent transactions per handler, so a failure in Billing doesn't roll back Shipping
• Separate retry loops and potentially different error handling policies per module
• The ability to mix durable and lightweight "fire and forget" semantics module by module
• Granular tracing and metrics per handler

If a module later earns its own deployable service, you can change the routing of a message from a local queue to RabbitMQ, Azure Service Bus, Kafka, or any of the other transports without touching the handler code at all. See the modular monolith tutorial for a lot more detail, including how to give each module its own Marten or EF Core store while keeping the transactional outbox intact.

Durable local queues

The durability does require an application database and integration with Wolverine's message persistence. At this time, Wolverine supports PostgreSQL, SQL Server, MySQL, Oracle, SQLite, RavenDB, and Azure Cosmos DB

Any Wolverine local queue can be made durable, which means every message published to it is persisted in your application's database, in Wolverine's transactional inbox, until it's successfully processed:

builder.UseWolverine(opts =>
{
    // Make every local queue durable
    opts.Policies.UseDurableLocalQueues();

    // Or just a specific queue
    opts.LocalQueueFor<OrderPlaced>().UseDurableInbox();

    // Or by convention, say by the module's namespace
    opts.Policies.ConfigureConventionalLocalRouting().CustomizeQueues((type, queue) =>
    {
        if (type.IsInNamespace("MyApp.Billing"))
        {
            queue.UseDurableInbox();
        }
    });
});

TIP

LocalQueueFor<T>() and CustomizeQueues() configure the queue that Wolverine assigns to a message type. With MultipleHandlerBehavior.Separated, each handler runs on its own queue named after the handler type instead, so neither of those reaches it. In that mode, use opts.Policies.UseDurableLocalQueues(), or implement IConfigureLocalQueue on the handler as shown below.

What does that buy you?

• The inbox and the queue are the same thing. A message cascaded from a handler is written to the inbox in the same database transaction as your business data. If the transaction rolls back, the message never existed.
• Messages survive process failures. If the process crashes, gets killed by a deployment, or has its pod evicted with messages still sitting in the queue, those messages are still in the database. When the node restarts, Wolverine recovers and processes them. In a cluster, Wolverine's durability agent reassigns a dead node's messages to the nodes that are still running, so the work doesn't even have to wait for the restart.
• Failures are persistent too. Retries that are scheduled for later are persisted, and messages that exhaust their error handling land in Wolverine's durable dead letter storage, where you can query them and replay them later.
• It works per module. If your modules use separate Marten stores or separate EF Core DbContext types, the durable local queue persists each message into the store of the handler that's going to process it, so the receiving module's transaction and its inbox stay atomic. See Modular Monoliths in Wolverine for more information, as it does require a little bit of configuration.

Wolverine does still serialize the message to write it to the inbox, but the original object is what gets passed through the local queue to your handler, so you're not paying for a round trip through the serializer on the happy path. The cost is a database write per message. You can choose to pay it only for the messages that matter, and leave everything else as a lightweight, in-memory "buffered" queue.

Error handling, retries, and resiliency

Because a local queue is a real endpoint, all of Wolverine's error handling policies apply to it, globally or per message type:

builder.UseWolverine(opts =>
{
    // Transient database hiccups? Retry in process with a cooldown
    opts.OnException<NpgsqlException>()
        .RetryWithCooldown(50.Milliseconds(), 250.Milliseconds(), 1.Seconds())

        // Then back off and retry a little later. With a durable
        // local queue, these scheduled retries are persisted
        .Then.ScheduleRetry(5.Seconds(), 30.Seconds())

        // And finally give up and move to the dead letter queue
        .Then.MoveToErrorQueue();

    // Some exceptions will never succeed no matter how many times you try
    opts.OnException<InvalidOrderStateException>().MoveToErrorQueue();
});

Contrast that with a MediatR notification handler, where the only error handling is whatever you remember to put in a try/catch block.

Observability you don't have to build

Any time you go asynchronous, even in process, you give up the nice simple call stack in the debugger. When something goes wrong in production, you will want to see what happened to a message. With Wolverine, messages on local queues are instrumented exactly like messages coming in from a broker:

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing => tracing.AddSource("Wolverine"))
    .WithMetrics(metrics => metrics.AddMeter("Wolverine*"))
    .UseOtlpExporter();

That's it. From there, Wolverine emits:

• OpenTelemetry spans for sending and executing every message, with correlation carried from the original HTTP request through every cascading message, so a trace follows one order from the Orders module through Shipping and Billing in your OpenTelemetry tool of choice
• Performance metrics through System.Diagnostics.Metrics: messages sent, execution time, effective time from send to completion, successes, failures tagged by exception type, and dead letters. Every instrument is tagged by message type and destination, so you can see how each local queue, and therefore each module, is performing.
• Inbox, outbox, and scheduled message counts as gauges, so you can see a queue backing up before your users do
• Structured logging for message execution, failures, retries, and circuit breakers opening and closing

See Instrumentation and Metrics for the full list of spans, tags, and metrics. And if you're using CritterWatch, local queues show up there alongside every other endpoint, with dead letter triage, alerting on circuit breakers tripping, and the ability to pause and restart listeners at runtime.

MediatR has none of this built in, and a hand-rolled Channel<T> has exactly the observability you write for it.

Sequencing vs. parallelism

Wolverine gives you a lot of control over message ordering or parallelism within the local queues. Local queues in Wolverine are parallel and unordered by default. Wolverine will happily process messages from a single local queue on multiple threads at the same time, by default up to the number of processors on the machine or five, whichever is greater. That's the right default for throughput, but it's not right for everything.

You have full control, queue by queue:

builder.UseWolverine(opts =>
{
    // Strictly first in, first out, one message at a time
    opts.LocalQueueFor<LedgerEntryPosted>().Sequential();

    // Throttle a queue that calls a rate-limited third-party API
    opts.LocalQueueFor<SyncCustomerToCrm>().MaximumParallelMessages(3);

    // Let the cheap stuff rip
    opts.LocalQueueFor<ProductViewed>().MaximumParallelMessages(20);
});

Or, if you'd rather keep the configuration right next to the handler, a handler type can configure its own local queue by implementing IConfigureLocalQueue:

public class LedgerEntryPostedHandler : IConfigureLocalQueue
{
    public static void Handle(LedgerEntryPosted posted, LedgerAccount account)
    {
        // ...
    }

    public static void Configure(LocalQueueConfiguration configuration)
    {
        configuration.Sequential().UseDurableInbox();
    }
}

Strict sequential processing is a blunt instrument though. What you usually want is sequential processing per entity, event stream, or saga (never process two messages for the same order at the same time) while still processing messages for different orders in parallel. Wolverine's partitioned sequential messaging does exactly that across a set of local queues:

builder.UseWolverine(opts =>
{
    opts.MessagePartitioning
        // Teach Wolverine how to find the "group id" of a message
        .ByMessage<IOrderCommand>(x => x.OrderId)

        // Spread IOrderCommand messages over four local queues, "orders1"
        // through "orders4", by a deterministic hash of the group id.
        // Messages for the same order always land on the same queue, and
        // are processed one at a time, in order
        .PublishToPartitionedLocalMessaging("orders", 4, topology =>
        {
            topology.MessagesImplementing<IOrderCommand>();
            topology.MaxDegreeOfParallelism = PartitionSlots.Five;
        });
});

If your messages are handled by Marten or Polecat aggregate handlers or by sagas, you don't even have to write the grouping rule. UseInferredMessageGrouping() will use the stream id or saga id as the group id automatically.

Global partitioning when you're running more than one node

Everything in the previous section keeps you safe within a single process. The catch is that almost nobody runs only one node in production. Two nodes of your modular monolith can both be processing IOrderCommand messages for the same order at the same time, and each one is perfectly sequential locally.

That's where Global Partitioning comes in. It's how you keep resources that don't tolerate concurrent access, like event streams, sagas, an inventory record, or a third-party API that can't handle two concurrent requests for the same account, from ever seeing concurrent access across the whole cluster. The best part for a modular monolith is that it doesn't need a message broker. If you're already using PostgreSQL or SQL Server for Wolverine's message storage, the shards are just a few more tables in the database you already have:

builder.UseWolverine(opts =>
{
    opts.UsePostgresqlPersistenceAndTransport(connectionString)
        .AutoProvision();

    opts.MessagePartitioning
        // Use the aggregate stream id or saga id as the group id
        .UseInferredMessageGrouping()

        // Or be explicit
        .ByMessage<IOrderCommand>(x => x.OrderId)

        .GlobalPartitioned(topology =>
        {
            // Four sharded, durable PostgreSQL queues, each paired with a
            // companion local queue that does the sequential processing
            topology.UseShardedPostgresqlQueues("orders", 4);
            topology.MessagesImplementing<IOrderCommand>();
        });
});

Each shard is listened to by exactly one node in the cluster at a time, and Wolverine rebalances the shards as nodes come and go. When a message is published, Wolverine checks whether the current node owns the shard for that message's group id. If it does, the message is routed straight to the companion local queue with no network hop at all. Only when another node owns the shard does it go through the database queue. In the single-node case, it all collapses back to plain local queues.

The result is that messages for the same order are never processed concurrently anywhere in your cluster, without distributed locks, without optimistic concurrency retry storms, and without making your whole system single-file. I wrote a lot more about this in Ordered Messaging Without the Locks.

Durable local queues + circuit breakers

Here's a scenario we see all the time in client work. One of your modules depends on something flaky: a payment gateway, a legacy SOAP service, a shared database that falls over during month-end processing. When it goes down, every message on that queue fails, retries, fails again, and eventually lands in the dead letter queue. Now someone has to go replay a few thousand dead letters by hand once the downstream system comes back, and in the meantime you've been pounding a system that was already struggling.

Wolverine's circuit breaker is the answer to that, and it can be applied to local queues:

builder.UseWolverine(opts =>
{
    opts.LocalQueueFor<ProcessPayment>()
        // This is the important part
        .UseDurableInbox()

        // Don't hammer the gateway with more than 5 calls at once
        .MaximumParallelMessages(5)

        .CircuitBreaker(cb =>
        {
            // Don't evaluate the circuit until at least this many
            // messages have been processed in the tracking period
            cb.MinimumThreshold = 10;

            // Trip the circuit if more than 20% of messages fail...
            cb.FailurePercentageThreshold = 20;

            // ...within this rolling window
            cb.TrackingPeriod = 2.Minutes();

            // Stop processing for this long, then try again
            cb.PauseTime = 5.Minutes();

            // Only count the failures that mean "the gateway is down,"
            // not validation errors in the message itself
            cb.Include<TimeoutException>();
            cb.Include<HttpRequestException>();
        });
});

When the failure rate crosses the threshold, Wolverine trips the circuit. It stops pulling new work off that local queue, lets the messages already in flight finish, and pauses the queue for PauseTime before trying again. Only the ProcessPayment queue pauses. Every other module keeps running.

Here's why the durability half matters so much. While the circuit is open, the messages don't go anywhere. Anything published to that queue during the outage is still written to the transactional inbox, where it just sits until the circuit closes and the queue picks back up. There's no unbounded pile of messages growing in memory, and nothing is lost if the process restarts in the middle of the outage. Wolverine's inbox recovery also respects the paused queue, so it won't feed messages into a listener that's deliberately stopped. When the gateway comes back, the backlog drains on its own, and nobody replays dead letters by hand at 2 AM.

WARNING

The circuit breaker only applies to durable local queues. A non-durable, buffered local queue has nowhere safe to put the messages while it's paused, so as of Wolverine 6.36, a circuit breaker on a buffered local queue stops the application from starting with an InvalidListenerConfigurationException. If you want a circuit breaker on a local queue, make that queue durable.

Combine that with the error handling policies above, and the actual behavior is pretty close to what an experienced operator would do by hand: retry the transient blips quickly, back off when things look bad, stop entirely when the downstream system is clearly down, and pick back up by itself afterward. And if you're using CritterWatch, the circuit opening and closing shows up as an alert and on the timeline, so you know it happened even if it fixed itself.

When not to use local queues

There are some limitations to the local queues of course. Messages on a local queue are processed on the node that published them (other than recovery after a failure), so work that's published unevenly across your nodes won't be load balanced across the cluster. This can be a killer problem if you have a system that receives work unevenly from the outside -- for example, if a customer uploads a flat file import that has 100X more data than you ever expected and that sudden surge causes problems because the load all goes to one node.

A backed up buffered queue also lives in memory, and while a durable queue is safe, a very large backlog is still something you'll want to watch for as it can give you memory issues. If you need smooth load distribution across nodes, or you're publishing a large volume of work that could back up, route those messages through RabbitMQ, Azure Service Bus, Amazon SQS, Kafka, or Wolverine's PostgreSQL or SQL Server database queues instead. The handler code doesn't change. You can mix local and external messaging freely in the same application, and moving a message type from one to the other is a one-line routing change.

Wrapping up

The pitch for local queues in a modular monolith isn't that Wolverine has an in-memory message bus. Lots of tools have one of those. The pitch is that Wolverine's local queues are first class endpoints, backed by the same transactional inbox, error handling, dead lettering, telemetry, partitioning, and circuit breakers as everything else in Wolverine. You get to start with the simplicity of in-process communication between modules without giving up any of the reliability you'd expect from "real" messaging, and without having to adopt a message broker before you actually need one. And each module gets its own parallelism, ordering, and durability, all side by side in the same application, instead of everything living with one global setting.

If you're building or migrating to a modular monolith, or you're trying to get off of MediatR notifications that have started losing work in production, JasperFx can help. Our consulting services cover everything from architecture reviews to hands-on delivery, and our support plans get you direct access to the people who wrote this code. And as always, come talk shop with us in the Critter Stack Discord.