Almost everything in this post came out of JasperFx client engagements, and if your Critter Stack system is running at a scale where the infrastructure itself starts to creak, that's exactly the kind of work we do. More on that at the end.

Wolverine has a subsystem buried inside of it that most users never think about, right up until it matters a great deal: leadership election and agent assignment. One node in your cluster is elected the leader, the leader decides what background work ("agents") should run on which nodes, and the whole thing quietly fails over when nodes come and go. It's the piece that lets Wolverine distribute Marten projection and subscription work across a cluster, run a listener on exactly one node, keep global partitioning honest about which node owns which slot, and recover the transactional inbox and outbox when a node dies.

That subsystem has been through a lot in the past couple of months. A JasperFx client runs a sharded, per-tenant Marten system across 512 databases with north of 18,000 projection agents in flight, deploys a new version every few days as a blue/green rollout, and has lost its leader mid-rollout more than once. That is a wonderful stress test, and it found every soft spot we had. This post is the story of what we found, what we changed, and where this is going next.

It turns out that a subsystem that was originally tested all in memory with a couple dozen items at a time might spring some leaks when it encounters 18,000+ agents and growing running in Kubernetes type chaos. Who could have guessed that?

Why we built this in the first place

In the beginning, the "Critter Stack" was really just Marten. In early Marten (pre 7.0), the asynchronous projection subsystem only worked in a "Hot/Cold" distribution where every single asynchronous projection executed on a single node within your cluster. The Marten core team most definitely knew that was a severe limiting factor for Marten's potential scalability, and it in fact was a major cause of concern from potential users who expected larger data loads.

As you can probably guess, Marten (and now Polecat and Fisher) projections are stateful and each projection (or subscription now too) can only run on one node at a time. We knew that for better scalability, we'd need to more intelligently distribute asynchronous projection and subscription work across an application cluster to better spread out the work -- but make sure at all times that each projection is running on exactly one node even as nodes get added and removed.

In no small part, Wolverine was rebooted and renamed specifically to be a complement to Marten by adding messaging support including its transactional outbox. A major goal for the 1.0 version of Wolverine was to add a new leader election capability and the ability to do software based assignment of stateful "agents" to different nodes in a cluster. As part of an early JasperFx client engagement, we got the chance to build out what is now Wolverine's managed projection and subscription distribution for Marten that was the original reason for doing this work in the first place.

Flash forward to today, and this subsystem is far more robust and capable.

What the subsystem looks like today

Before getting into the war stories, here's the shape of the thing as it stands.

Every node in a Wolverine cluster heartbeats into a wolverine_nodes table in your message store, along with the capabilities it advertises: the agents that this particular node knows how to run. One node holds the leadership lock. On a schedule (every HealthCheckPollingTime, 10 seconds by default) the leader reads the node table, evaluates what should be running where, and sends start, stop, and reassign commands to the other nodes through a control queue. Nodes that stop heartbeating get ejected and their agents get reassigned.

An "agent" is just a long running IAgent that the leader can start and stop, grouped by an IAgentFamily that knows how to enumerate them and how it wants them spread out. Today that includes:

• Async projection and subscription shards for Marten, Polecat, and Fisher, when you opt into Wolverine managed distribution. With database-per-tenant or sharded multi-tenancy, that's one agent per projection per tenant per database, which is how you get to 18,000 of them.
• The per-database durability agents that recover stranded inbox and outbox messages and poll for scheduled messages
• Exclusive listeners (ListenWithStrictOrdering(), ExclusiveNodeWithParallelism()) and leader-pinned listeners
• The slot ownership behind MessagePartitioning.GlobalPartitioned. Each partition slot is an agent, and the leader guarantees exactly one node is consuming a slot at a time and drains the outgoing node before the incoming one starts
• Recurring schedules, which prefer to run anywhere but the leader

Turning it on for Marten is one line:

builder.Services.AddMarten(opts =>
{
    // Marten configuration...
})
.IntegrateWithWolverine(x =>
{
    // Let Wolverine's leader spread projection and
    // subscription shards across the cluster
    x.UseWolverineManagedEventSubscriptionDistribution = true;
});

The important thing to understand about the leader is that it is making decisions from a snapshot. It reads the node table, computes an assignment grid, and emits commands. Those commands take time to land. The next evaluation reads a new snapshot, which may or may not reflect the commands from the last one yet. Nearly every problem I'm about to describe is some version of the leader making a confident decision from a snapshot that was already stale.

Then the fleet got big

The capability matching inside the assignment grid was a list scan: for each agent, walk each node's capability list. That is O(agents × nodes × capabilities), and it's only exercised when nodes advertise different capability sets, which is to say during every blue/green rollout with a projection version bump. At a few hundred agents nobody notices. At Topicus's scale a freshly elected leader spent 66 minutes of single-core CPU on its first assignment evaluation. Nothing was assigned in the meantime, the leader looked stale to everyone else, and three leaders in a row were replaced before one of them ever finished evaluating.

The capability match was the worst offender but not the only one. The same first evaluation had four more scans of exactly the same shape hiding in it: a List.Remove per command while batching commands (sixty thousand removals from a sixty thousand entry list), a Contains per agent when building a heartbeat record, a per-placement fill of a node's agent list, and a recount of a node's agents on every placement. We replaced all of them with sets and live counters. Measured on the reported shape (62,400 distinct agents across both versions, three blue nodes and two green nodes), the first evaluation of a cold cluster went from 76 seconds to well under 5 seconds for the entire test suite.

Then there were the control messages themselves. A node shedding 7,592 agents at once produced a single 6 MB stop command, which the receiving node refused to read at all. Starts and reassignments had been chunked for a while, but stops never had been, and the leader then politely waited out a reply window computed for that batch size, which came to about 63 hours. Worse, a control queue row that couldn't be deserialized aborted the whole poll, which rolled back the expiry that would have cleared the bad row, so the node re-read it forever and received no control messages at all. Stops are chunked now, a bad row is quarantined on its own, and both the sender and the receiver log enough to actually diagnose this.

And one more that only shows up at this scale: the leader's heartbeat was building the entire event store usage descriptor (every projection, every event type, the highest sequence) for every store, every tick, just to learn the shard names and the database topology. It reads only what it needs now.

Stopping the reassignment swarms

The scaling fixes were the straightforward part. The harder problem was churn. With a few thousand agents, a leader that makes a slightly wrong decision just makes a slightly wrong decision. With 18,000 agents, a slightly wrong decision is a wave of thousands of stops and starts, every one of which opens connection pools and catches up projections, and the next evaluation is happening before that wave lands. Several of these turned into genuine oscillations in production, where the cluster never converged at all.

Here is the honest list of places we had to add hysteresis, meaning some deliberate reluctance to act on the most recent reading:

Node ejection. A single stale read of the node table (replica lag, a GC pause, an aggressive StaleNodeTimeout) used to be enough for a peer to delete a perfectly healthy node's row, along with its agent assignments and in-flight message ownership. The node would re-register, get its agents back, and get ejected again. Ejection now requires several consecutive stale observations, and a follower may never delete the current leader's row. Only a node that actually holds the leadership lock gets to clean up a stale leader.

Leader takeover. The leader's ledger of "starts I've sent but that haven't shown up as rows yet" lives in memory on the leader. When the leader dies mid-wave, its replacement can't see that ledger, so every in-flight agent looks completely unplaced and the new leader starts a second copy somewhere else. Wolverine's duplicate agent detection would heal that a cycle later, but a projection agent ran twice in the meantime. A newly elected leader now holds back first-time placement for one evaluation (LeaderTakeoverHoldEvaluations), long enough for its predecessor's starts to appear as rows and be left where they are. Moves, stops, and anything visibly running proceed immediately, and orphans from a node that actually left are never held.

Cold start. The mirror image. The first node of a cold cluster sees nobody else in the node table and is handed the whole agent universe. On the reported fleet, 37,310 of 37,388 agents started on the first pod before any peer got a share, and every pod that joined afterwards was then served out of the first node's own backlog as a stop-then-start. The first leader now waits one tick (LeaderColdStartHoldEvaluations) when the wave is bigger than a start batch, which is enough for pods that start together to register. For rolling deploys that bring pods up over a longer window, AssignmentSettlePeriod lets the leader wait for the node set to go quiet before it assigns anything.

Group affinity. For a multi-database store, Wolverine keeps all of a database's agents together on one node so the connection pool count scales with databases rather than nodes × databases. The incumbent rule for that only recognized a partition when every member sat on one node. One straggler, say an agent an operator pinned elsewhere or a start that landed on a fallback node mid-rollout, and the whole database was placed from scratch, possibly onto a third node. One moved agent became a database's worth of moves, and with a release cooldown lapsing every ten minutes that repeated as a ten minute oscillation on a cluster whose node set never changed. The host is now whichever node is already running the most members, and only the stragglers move. A settled blue/green split is now a fixed point of the placement, and there's a test that pins it that way, because an earlier version counted roughly 45,000 reassignment decisions during a single rollout ramp.

Capacity. The original failure that kicked off a lot of this was a cascading collapse: three nodes shrinking to one, the survivor dutifully accepting every assignment, running out of memory starting them, and the leader re-deciding forever at about 125 assignment changes a minute. Capacity-aware assignment lets each node advertise its load, and the leader refuses to pile work onto a node over the line. The detail that matters here is that the line where a node starts shedding agents sits ten points above the line where it can start receiving them again. With a single threshold, a node at the line sheds an agent, drops below, looks like a target again, and gets the same agent right back on the next tick. The band gives the reading somewhere to settle. Shedding is also capped per evaluation and only ever happens when some other node can take the work. An overloaded node is still better than no node at all.

Departed nodes. A reassignment runs in the source node's command lane and waits for that node to confirm the stop. A node that was killed confirms nothing, and that lane used to sit out the whole reply window, 25 minutes at the default batch size, while the dispatcher insisted the agents were "still moving." The leader now notices a node's departure and abandons its lane outright.

Warm-up spread. During a blue/green warm-up, the first green node to register got every green-only agent and the green nodes that joined seconds later got nothing, because a pending assignment kept being put back on its pending node by every subsequent family pass. That one is fixed too, and it's a good segue.

If there's a theme to all of this, it's that every one of those knobs exists because we watched a real cluster oscillate. None of them were designed in up front, and I'm not sure they could have been.

How we test this now

Honestly, the biggest change isn't any single fix. It's that we can now reproduce this class of problem on a laptop, deterministically, without a Kubernetes cluster and 512 databases.

The test harness is a SimulatedCluster that drives the leader's real evaluation code against an in-memory node set. One round is one health check tick: in-flight starts advance, the leader evaluates, and the commands it emits are applied to the cluster the way the real agent commands would apply them. Time is measured in rounds, not milliseconds. Everything is seeded, so a failure reproduces exactly. Nodes can join, leave, be replaced as leader, or have the leader killed outright, and a start can be made to take several rounds or to fail against particular databases.

On top of that harness sit three layers of tests:

1. Lifecycle invariants drive hand-written sequences through whole lifecycles: a full blue/green deploy step by step, the leader replaced mid-wave with starts in flight, a scale-down, repeated scale-up and scale-down cycles. At every step, not just at the end, they assert that no agent is ever started on two nodes, no agent is ever sent to a node that can't build it, every transition converges in a bounded number of rounds, and the leader doesn't move agents it has no reason to move.
2. Fleet-scale evaluation gates the cost of the whole leader evaluation at the field shape, 62,400 agents on five nodes, so a regression back to minutes fails the build instead of hanging it. A production shape with 512 databases and roughly 100,000 agents is there too, opt-in behind an environment variable.
3. A chaos soak rolls a seeded random schedule of node joins (same version or a bumped one that kicks off a blue/green warm-up), node departures, leader replacements, and leader deaths for as many rounds as you ask, checking the invariants on every round and requiring full convergence whenever the fleet has been quiet for a while. It's deliberately not in CI. We run it by hand at ten seeds times 600 rounds and three seeds times 5,000 rounds before touching anything in this area.

The soak earned its keep almost immediately. Once it was driving commands through the real per-node dispatcher lanes with field-like batching, it found three separate ways a start could be withdrawn or double-started around a move that hadn't landed yet, including one that only appears when a start takes longer than the pending ledger's TTL, which happens to be the exact shape of Topicus's 27 second median start time against a 20 second TTL. Every one of those had a test that was red before its fix.

And underneath all of it, the leadership election itself has a compliance suite that runs against every message store Wolverine supports: PostgreSQL, SQL Server, MySQL, Oracle, SQLite, RavenDB, and Cosmos DB. That's where the "takeover hold never delays an orphan" bug was caught, in the Cosmos DB and Azure Service Bus lanes before anywhere else.

Blue/green deployment

Wolverine has supported blue/green deployment of projections for a while, in the sense that agents are only ever assigned to nodes that advertise the capability to run them. Bump a projection's version on the new release, and only the new nodes get the new version's agents, while the old nodes keep running the old version. I wrote about the Marten side of that last year.

What the past couple of months taught us is that "capability matching" is necessary but nowhere near sufficient at scale. In a mixed-version fleet with group affinity, the blue and green nodes are now distributed as two pools with their own ceilings, so the projections shared across both versions don't ping-pong between blue nodes every time a green node joins or leaves. The leader warns loudly when it finds itself dispatching an agent to a node that doesn't declare it while some other node does, instead of silently re-sending the same placement every evaluation for twelve hours. And a node can now opt out of leadership entirely:

opts.Durability.LeadershipEligible = false;  // default: true

That last one matters because a green node's own store enumerates the new version's agents, which makes it the single least representative node in the cluster to be evaluating assignments for everybody else. Sure enough, that's what was happening.

Preview: dedicated warm-up nodes

Here's where this is heading, and it's on main now for the next Wolverine release.

Say your next release bumps the version of a big async projection, and rebuilding it from scratch takes hours. You want the green nodes to catch that projection up off to the side before you cut over. The catch is that green has to be part of the same Wolverine cluster to be assigned those agents at all, and a normal node in that cluster will also happily pick up messages from its listeners, recover persisted envelopes, and run scheduled messages with the new code. If that new code appends an event type the blue release doesn't know about yet, every async projection on blue that reads it stops with an UnknownEventTypeException. Ask us how we know.

So the next release adds a node role:

// Only on the green nodes, and only until you cut over
opts.Durability.NodeRole = NodeRole.Warmup;  // default: NodeRole.Serving

A warm-up node joins the agent cluster through its control queue and otherwise stays out of the way:

• It never becomes the leader
• It is only ever assigned the projection agents that no serving node declares, in other words the new versions of your bumped projections. The shared projections stay put on blue, so green joining or leaving doesn't shuffle them
• It never runs event subscriptions, only projections. A subscription is a side effect by definition, and blue keeps running those until the cut over
• It runs its projection agents in the daemon's rebuild mode, so RaiseSideEffects() never fires and nothing gets published out of those projections. Unlike a real rebuild, the agent keeps following the event store, and a restarted warm-up node resumes from its own progress instead of starting over. That required a new mode in the shared JasperFx daemon that Marten, Polecat, and Fisher all build on
• It listens to its own control endpoint and nothing else. No application listeners, no inbox or outbox recovery, no durability agents, no scheduled message polling, no recurring schedules

To cut over, restart the green nodes as NodeRole.Serving, which is usually just flipping a configuration value in your deployment. The projection agents carry on in continuous mode from the same progress, and regular blue/green assignment takes it from there. And if every serving node disappears while green is still warming up, nothing gets reassigned at all, on purpose. A crash during a warm-up should never quietly promote the new release to production.

Wolverine can only enforce the role inside its own runtime, so your own BackgroundService implementations, your load balancer membership, and anything your projection code does directly inside an Apply() method are still on you. The docs for this will carry a checklist.

How JasperFx can help

Very little of what's in this post was built on speculation. The capacity-aware assignment, the takeover and cold start holds, the group affinity incumbent rule, the chunked stop commands, the fleet-scale test harness, and the warm-up node role all came directly out of JasperFx client engagements, with clients who were running the Critter Stack hard enough to find the edges and who were willing to let us instrument their clusters and work the problem with them. Topicus in particular has been a tremendous partner on this.

That's the shape of a JasperFx support contract at its best: you get direct access to the people who wrote the code, we get a production system that teaches us things we couldn't learn any other way, and the fixes go back into the open source tools for everyone. If you're running Marten or Wolverine at a scale where the infrastructure is what's hurting, or you're staring down a blue/green projection rollout and would rather not learn about UnknownEventTypeException the way we did, our consulting services cover everything from an architecture review to hands-on work inside your cluster.

And as always, come talk shop with us in the Critter Stack Discord.

Wrapping up

Leadership election and agent assignment started as a way to spread Marten's projection work across a cluster without making anyone think about it. It still is that, and for most teams the one-line opt-in is all they'll ever touch. But "distribute some work across some nodes" turns into a genuinely hard distributed systems problem the moment the agent count has five digits, nodes come and go on their own schedule, and every decision is made from a snapshot that's already a little out of date. The last couple of months have been about making that leader reluctant in exactly the right places, making it fast enough that reluctance is affordable, and building the tooling to prove it stays that way. The warm-up node role is the next step, and I'm looking forward to it shipping.