Message ordering and concurrency problems are among the most common reasons clients come to JasperFx for help, and we're certainly ready to help your shop as well!

In a recent post about Wolverine's local queues, I promised a follow-up about ordered message processing across a cluster. Here it is.

First, let's get salty. NServiceBus's own tuning documentation says plainly that "sequential processing on the endpoint (logical) level is not possible when scaled out." Its saga documentation says that "sagas must be designed to handle the arrival of out-of-order messages." And Particular has spent years telling the .NET community that you don't need ordered delivery at all.

They're certainly right that designing for out-of-order delivery is good advice on its face -- but what if you had more options for message sequencing?

Wolverine takes the opposite position. Ordering is sometimes a genuine requirement, so the framework should give it to you, even across a scaled-out cluster. And rather than one all-or-nothing switch, Wolverine gives you a range of options with different trade-offs between ordering, throughput, and complexity.

Also see Wolverine's Resequencer Saga feature for collecting messages and putting them back in the correct order in those cases where you get unpredictable messaging from upstream systems.

Why "one at a time" isn't enough

Let's start with the problem. In Wolverine, any queue can be made to process messages one at a time, in order:

opts.LocalQueueFor<LedgerEntryPosted>().Sequential();

opts.ListenToRabbitQueue("ledger").Sequential();

That's strictly ordered, within one process. The catch is that almost nobody runs a single node in production. If three nodes are all listening to the ledger queue, each one is processing its own messages in order, but three messages from the same logical sequence can still be processed at the same time on three different nodes. That's the "competing consumers" pattern, and it's the default for good reason: it's how you scale out. It just isn't ordering.

To get ordering across a cluster, you need something that guarantees that only one node is processing a given sequence of messages at any time, and that some other node picks up the work if that node dies. Wolverine has a number of ways to do that.

Strict ordering: one node, one message at a time

Don't use this anywhere except where you really need it. As a matter of fact, I'd say it's mostly useful for some type of control message flow that's used to coordinate the rest of the application -- and that's exactly where we've seen this used the most so far.

The simplest, bluntest option is ListenWithStrictOrdering():

builder.UseWolverine(opts =>
{
    opts.UseRabbitMq();

    // Any message persistence enables Wolverine's leader
    // election and node agent assignment
    opts.PersistMessagesWithPostgresql(connectionString);

    opts.ListenToRabbitQueue("ledger")
        // Only listen on one node in the whole cluster, and
        // process messages there one at a time, in order
        .ListenWithStrictOrdering();
});

That does two things. It tells Wolverine to only ever run the listener for that queue on a single node in the cluster, and it makes that listener process messages strictly sequentially, on a single thread. The result is strict, global ordering for everything that goes through that queue, no matter how many nodes you're running.

The cost is obvious: that queue's throughput is capped at what one thread on one node can do. That turns out to be perfectly fine for a lot of real work. It's a good fit for fast messages that coordinate slower work elsewhere, for file ingestion processes, for control messages in long-running workflows, or for any queue where the volume is modest and correctness matters more than speed. See the concurrency tutorial for more on when to reach for it.

Exclusive node with parallelism: singleton work, with throughput

Sometimes what you actually need isn't ordering. It's exclusivity. Maybe the handler keeps some in-memory state, or talks to a resource that can only have one client, or does work that only makes sense to do in one place. For that, ExclusiveNodeWithParallelism() keeps the listener on one node, but still lets that node process several messages at once:

opts.ListenToRabbitQueue("inventory-sync")
    // One node in the cluster, up to 5 messages at a time on that node
    .ExclusiveNodeWithParallelism(5);

On its own, that doesn't give you ordering. But it combines nicely with Wolverine's partitioned processing to give you ordering per key on that one node:

opts.MessagePartitioning.ByMessage<IOrderCommand>(x => x.OrderId);

opts.ListenToRabbitQueue("orders")
    // Only one node in the cluster processes this queue...
    .ExclusiveNodeWithParallelism()

    // ...and on that node, messages for the same order are processed
    // one at a time, while different orders are processed in parallel
    .PartitionProcessingByGroupId(PartitionSlots.Seven);

That's a very useful middle ground: messages for the same order can never be processed concurrently anywhere in the cluster, and you still get parallelism across orders. The limit is that it's all happening on one node.

If you're on Azure Service Bus, you can get the same shape with the broker doing the per-key work. Sessions give you FIFO within a session, and Wolverine can pin the session-enabled listener to one node:

opts.ListenToAzureServiceBusQueue("user-events")
    // Enable sessions, process up to 8 sessions in parallel,
    // and only on one node in the cluster
    .ExclusiveNodeWithSessions(8);

Pinned to the leader

Wolverine elects one node in the cluster as the leader, and you can pin a listener to whichever node that is:

opts.ListenToRabbitQueue("control")
    .ListenOnlyAtLeader();

That's handy for coordination work that naturally belongs with the rest of the cluster management. And if what you need isn't a listener at all but a single background process running on exactly one node, Wolverine's SingularAgent gives you that too.

How Wolverine keeps it to one node

None of the options above takes a distributed lock around your message handling. Under the covers, they all use Wolverine's leader election and agent assignment. One node is elected leader, and the leader continuously makes sure that every "agent" in the system, including each exclusive listener, is running on exactly one node. That's the same machinery that distributes Wolverine's durability agents and, with Marten, your async projections and subscriptions. It's on by default as soon as you configure any kind of message persistence, so there's nothing else to turn on.

When the node holding an exclusive listener fails, the other nodes detect it through the persistence layer, the leader reassigns the listener to a surviving node, and processing picks back up there. That typically takes a few seconds.

If you pair an exclusive listener with Wolverine's durable inbox, messages that were in flight on the failed node aren't lost either. Wolverine is deliberate about who recovers them: for exclusive and leader-pinned listeners, only the node that currently holds the listener recovers that endpoint's stranded inbox messages. That rule exists because handing them to any other node would break the exclusivity guarantee you asked for. It covers every database that can hold inbox messages for that listener, including separate tenant databases and the separate stores in a modular monolith. See Inbox Recovery Ownership for the details.

opts.ListenToRabbitQueue("ledger")
    .ListenWithStrictOrdering()

    // Strictly ordered, on one node, and no message is lost
    // if that node goes down mid-flight
    .UseDurableInbox();

Global partitioning: ordering per key, across every node

This is Wolverine's special sauce for sidestepping concurrency issues while maximizing system throughput.

Everything so far keeps the ordered work on one node at a time. That's often enough, but it does mean one node's worth of throughput for that queue. What if you need strict ordering per entity and the throughput of the whole cluster?

That's Global Partitioning:

builder.UseWolverine(opts =>
{
    opts.UseRabbitMq();
    opts.PersistMessagesWithPostgresql(connectionString);

    opts.MessagePartitioning
        // Use the saga id or the aggregate stream id as the group id,
        // or teach Wolverine explicitly with ByMessage<T>()
        .UseInferredMessageGrouping()

        .GlobalPartitioned(topology =>
        {
            // Shard the work across 5 RabbitMQ queues
            topology.UseShardedRabbitQueues("orders", 5);
            topology.MessagesImplementing<IOrderCommand>();
        });
});

Every message is assigned a group id: the order id, the stream id, the saga id, a tenant id, whatever identifies the thing that must not be processed concurrently. The group id is hashed deterministically to one of the shards. Each shard is an exclusive listener, so exactly one node consumes it at a time, and Wolverine spreads the shards across the cluster and rebalances them as nodes come and go. Every message for the same order lands on the same shard, so it's processed one at a time, in order, on whichever node owns that shard. Different orders spread across all the shards and all the nodes.

In other words, it's the exclusive listener idea from above, multiplied across the cluster and routed automatically. When the current node happens to own the shard for a message, Wolverine skips the broker entirely and routes the message straight to a local queue. It even follows cascading messages, so anything your handler publishes about the same order stays in that order's lane.

This works the same way over RabbitMQ, Azure Service Bus, Amazon SQS, Kafka, Pulsar, NATS, Redis Streams, GCP Pub/Sub, and PostgreSQL or SQL Server database queues. The database queues are especially nice for teams that don't want to run a broker at all, since the shards are just a few more tables in the database you already have:

opts.MessagePartitioning.GlobalPartitioned(topology =>
{
    topology.UseShardedPostgresqlQueues("orders", 4);
    topology.MessagesImplementing<IOrderCommand>();
});

A couple of honest details worth knowing:

• The unit of ordering is the shard, not the key. Two unrelated orders that hash to the same shard are serialized against each other. That's stronger than you asked for, and it costs some parallelism, but the number of shards is fixed, so nothing grows with the number of keys you have.
• The default is durable. Each shard is backed by Wolverine's inbox, so every message pays a database write on the way in. For floods of traffic where the database can't keep up, like webhook storms or telemetry, ProcessInParallelWithNativeAcks() drops the database and holds the broker delivery unacknowledged until the handler completes instead. You still get the "never concurrently" guarantee, and you trade the inbox's deduplication for at-least-once delivery that your handlers need to tolerate.
• Failover drains first. When a shard moves between nodes, the outgoing node finishes the messages it's already processing before the new owner is allowed to start. That's what keeps one order from ever being processed in two places at once during a handoff.

I wrote a lot more about global partitioning, and about the companion re-sequencer for messages that arrive out of order, in Ordered Messaging Without the Locks.

Or let the broker do it

Several brokers have their own per-key ordering primitive, and Wolverine supports them directly when you'd rather let the broker do the work:

Broker	Native primitive
Azure Service Bus	Sessions
Amazon SQS	FIFO queues with message groups
GCP Pub/Sub	Ordering keys
Pulsar	KeyShared subscriptions
Kafka	Partitions plus consumer groups, with PropagateGroupIdToPartitionKey()

Wolverine maps its GroupId onto Azure Service Bus session ids, SQS message group ids, and GCP ordering keys, and onto Kafka message keys when you opt in with PropagateGroupIdToPartitionKey(), so the same MessagePartitioning rules can drive the broker's own ordering. The trade-offs versus global partitioning are real, and the partitioning docs lay them out:

• Native primitives usually order per key, so unrelated keys never block each other. But the broker keeps state for every active key, so a system that generates lots of new group ids can run into service limits.
• A poison message under a native per-key primitive blocks its whole key until it's dead-lettered. Under global partitioning, it only ties up one slot of one shard.
• The native primitives tie you to that broker. Global partitioning works the same way on all of them.

And for comparison's sake, NServiceBus's Azure Service Bus and SQS transports currently list message sessions and FIFO queues as "not supported," though to be fair, the Azure Service Bus page says that "support for Azure Service Bus message sessions is being considered."

Ordered event processing

If you're using Marten for event sourcing, there's one more flavor of cluster-wide ordering that comes almost for free. Marten's async daemon runs each projection and subscription on exactly one node, either through its own HotCold mode or through Wolverine-managed distribution, which spreads them across the cluster using the same leader election described above. So you can process the events Marten captures through ordinary Wolverine message handlers, in the strict order they were appended, from a cluster:

opts.Services.AddMarten(connectionString)
    .IntegrateWithWolverine()
    .AddAsyncDaemon(DaemonMode.HotCold)

    // Call Wolverine handlers for each of these events,
    // one at a time, in the order they were appended
    .ProcessEventsWithWolverineHandlersInStrictOrder("Orders", o =>
    {
        o.IncludeType<OrderCreated>();
        o.IncludeType<OrderShipped>();
    });

See Event Subscriptions for the details, including how errors are handled without stopping the subscription.

Choosing between them

Here's my cheat sheet:

Option	Nodes processing	Parallelism	Ordering	Reach for it when...
Competing consumers (the default)	All	Configurable	None	Throughput matters and order doesn't
Sequential()	All, independently	1 per node	Within one node only	You're running a single node, or a local queue
ExclusiveNodeWithParallelism()	One	Configurable	None	You need a singleton, not an order
ExclusiveNodeWithParallelism() + PartitionProcessingByGroupId()	One	Across keys	Per key	Per-entity ordering at one node's throughput
ListenWithStrictOrdering()	One	1	Global	Strict global order at modest volume
ListenOnlyAtLeader()	The leader	Configurable	None	Coordination work that belongs with the leader
Broker sessions, FIFO groups, or keys	All	Across keys	Per key	You're committed to one broker and keys are bounded
Global partitioning	All	Across shards	Per shard	Per-entity ordering at full cluster throughput, on any transport
Marten strict order subscription	One per subscription	1	Append order	Reacting to events in the order they happened

A few rules of thumb from our client work:

• Don't order what doesn't need ordering. Most messages are fine with competing consumers.
• When you do need ordering, you almost always want it per entity, not globally. Global strict ordering is the right answer much less often than people think.
• Favor the options that make concurrent access impossible over the ones that make it recoverable. Optimistic concurrency plus retries is a fine safety net, but a hot entity under load turns it into a retry storm that burns capacity reprocessing work you already did.
• You can mix all of these in the same application. One queue can be strictly ordered while the next one is partitioned and the one after that is wide open.

Wrapping up

"You don't need ordered delivery" is a perfectly good design principle right up until you do. When that happens, the answer shouldn't be to rebuild your handlers around an ordering guarantee your framework can't give you, or to hand-roll a sharded topology with your own routing and failover. With Wolverine, it's a line or two of configuration, backed by leader election, failover, and durable inbox recovery that already exist, and it works on whichever transport you're using.

To be fair to the rest of the ecosystem, MassTransit's SQL transport does have a genuine cluster-wide partitioned ordering mode, and credit to them for it. But in MassTransit's own words, it's "unique to the SQL transport." For everything else, and for most .NET messaging tools, the answer to "can I have ordering in a cluster?" is still some version of "no, and you shouldn't want it." Wolverine's answer is "yes, and here are your options."

If ordering or concurrency problems are hurting your system, whether that's lost updates, retry storms, or a saga that trips over itself, this is exactly the kind of work JasperFx does. Our consulting services cover everything from architecture reviews to hands-on fixes, and our support plans get you direct access to the people who built these features. And as always, come talk shop with us in the Critter Stack Discord.