JasperFx is here to help you get the most out of a Marten or Wolverine system that is big enough to make performance and scalability a concern, and that's work JasperFx does with clients every week. Talk to us before the pods start restarting if you can, but we'll take the call afterward too!

A lot of what makes Marten and Wolverine dependable today certainly didn't come from all the brilliant ideas we had upfront. Instead, our tools got battle hardened from somebody running the Critter Stack much, much harder than our test suites did, finding where and how things broke down under load, and then working through the problems with us. I think this post is a good example of both how we approach scalability problems and what JasperFx does to help our clients succeed.

This particular issue came from a JasperFx client of course, but the Critter Stack community in general has been fantastic to work with over the years and has helped drive many improvements either directly through pull requests or by taking enough time to give us detailed error reports or valuable feature requests.

The short version is that we ended up with a pretty powerful recipe for massive multi-tenanted systems:

1. Partition the work by tenant, so every tenant's projection work is its own small, independent unit
2. Make the tenants take turns when there isn't room for everybody at once, so nobody is starved
3. Limit the work in flight on each node, so a node can't load more than it can hold in memory. In the end that meant taking the back pressure each subscription agent already had and applying it across every agent in the process

The system

The JasperFx client in question runs a multi-tenanted system that's well beyond what most of us will ever get to work on:

• A little over 2,100 tenants spread across 512 PostgreSQL databases with Marten's sharded multi-tenancy
• Per-tenant event partitioning, so every tenant has its own event sequence and its own projection progress
• 22 asynchronous projections, some of them big multi-stage composite projections
• Wolverine distributing the projection work across the cluster, one agent per projection per tenant. That's roughly 52,000 agents

We've been working with this team for a while now. The agent assignment work in Wolverine 6.50 and the warm-up nodes for rebuilding projections without downtime both came out of the same collaboration.

The problem

The trouble showed up in exactly that blue/green warm-up scenario. A new version of a composite projection with about ten members had to be rebuilt from zero for every tenant on a set of warm-up nodes. Each pod picked up around 400 tenant agents inside of about 70 seconds, and then:

• With a 24 GiB memory limit, the pod was OOMKilled after four or five minutes
• With a 56 GiB limit, it didn't get killed for memory, instead the garbage collector ate nearly every core trying to keep up, and the pod was killed by its liveness check a few minutes later (#sadtrombone)

More memory wasn't going to fix this. The process used whatever it was given.

The client's team did the hard part, which was the profiling. Their heap snapshots put most of the live memory in byte arrays that Marten had allocated while serializing documents into a projection batch that was waiting to be written. Making the page size smaller didn't help either, because in their model a single event can fan out to around 11,000 aggregate documents. Bounding a page by its event count just doesn't bound the work inside it.

Why the existing throttles didn't help

Marten already had governors for this kind of thing. MaxConcurrentEventLoadsPerDatabase limits how many agents may load events from one database at a time, and MaxConcurrentBatchWritesPerDatabase limits how many may write. Both exist to keep the connection count sane, and they do that well.

Neither one bounds memory though, and the client's analysis nailed the reason. The write throttle is taken after the batch has been built. Every one of those 400 agents could load its events, run its projection, serialize every document, and then stand in line for a write slot holding a fully built batch. On top of that, both throttles are per database. With hundreds of databases, "four at a time per database" is not much of a limit on one node.

So the memory in use grew with the number of running tenant agents, and per-tenant partitioning had given us a lot of tenant agents. The feature that makes each tenant's work small and independent was also the thing letting all of that small work happen at the same moment.

Tenants take turns

The client sent us a pull request with their diagnosis and a first fix, a semaphore on the number of batches in flight per node. We took the diagnosis whole and went a slightly different direction on the mechanism over two releases worth of work in JasperFx.Events, which is the shared event sourcing engine underneath Marten, Polecat, and Fisher.

The first step was a queue. With it switched on, each projection gets one single-file line per store on a node, shared across every database. A tenant agent that has a page to process waits for its turn, builds its batch, commits it, and goes to the back of the line. That's one built batch per projection on the node instead of one per tenant, and because the line is first in, first out, it rotates through the tenants.

That bounded the built batches, but every waiting agent could still be holding up to 5,000 loaded events of its own. With 400 agents on a pod that's potentially two million events sitting in memory waiting for their turn.

The second step is the one that worked best in our load testing, and it wasn't a new invention at all. A subscription agent in the async daemon has always had back pressure built in. Each agent keeps a "hopper" of events it has loaded but not finished processing, and when that hopper is full the agent stops fetching more until the projection work catches up. That has kept a single agent from outrunning itself for years. The trouble was only that the limit belonged to one agent, and we now had hundreds of agents in one process, each politely staying inside its own limit while the process as a whole ran out of memory.

So we took the approach we already trusted and widened it from one agent to every agent in the process. Instead of each agent counting its own events in flight, all of a store's tenant agents on a node draw from one shared budget on the work in flight. This is the setting I'd tell you to reach for first:

builder.Services.AddMarten(opts =>
    {
        opts.Connection(connectionString);

        opts.Events.TenancyStyle = TenancyStyle.Conjoined;
        opts.Events.UseTenantPartitionedEvents = true;

        // The most events this store's tenant agents may have loaded,
        // but not yet committed, on this node. Zero is unbounded and the default
        opts.Projections.TenantLoadBudget = 2000;

        // Stop handing out new pages at 90% of the GC's high memory load
        // threshold, whatever the budget still allows. This is the default
        opts.Projections.TenantLoadMemoryThreshold = 0.9;
    })
    .IntegrateWithWolverine(m =>
    {
        m.UseWolverineManagedEventSubscriptionDistribution = true;
    });

Here's how it behaves:

• An agent reserves room for a page before it loads anything, trims the reservation to what the page really held, and gives it back when the page commits
• While there's room and nobody is waiting, loads go straight through. A healthy system that's caught up never notices the budget is there
• When there isn't room, agents wait in line and are let in one page at a time, then go to the back. Every tenant keeps moving and no tenant falls behind forever because a bigger neighbor got there first
• A waiting agent is still running, still assigned, and still reporting its health. Wolverine doesn't see a reason to move it and CritterWatch doesn't show it flapping. It just isn't loading
• The memory threshold is the backstop. If the process gets close to the garbage collector's high memory mark, no new pages are handed out until the work already in flight drains, no matter what the event budget says
• A load is always allowed when nothing at all is in flight, so setting the budget too low makes the node slow instead of stuck

That last point matters more than it might look. A limit that can deadlock under the wrong configuration is worse than no limit.

What we measured

We wrote a stress harness in the Marten repository that rebuilds a two-stage composite projection for 24 tenants at once, where each page turns into about 3,000 upserts of 8 KB documents. On a laptop with workstation GC:

Configuration	Peak memory	Rebuild time
No limits	8,215 MB	8.6 s
TenantLoadBudget = 4000 (8 pages)	6,198 MB	6.7 s
TenantLoadBudget = 2000 (4 pages)	1,185 to 1,410 MB	7.6 to 7.9 s
TenantLoadBudget = 500 (1 page)	351 MB	17.8 s
The queue alone	355 MB	16.4 s

A budget of four pages did the rebuild in about the same time as no limit at all, with roughly a sixth of the memory. Squeeze it down to a single page and you get to a twentieth of the memory, at about twice the time. The budget is a dial between speed and memory, and you get to pick the spot that fits your pod. Under server GC the absolute numbers are different, because server GC holds on to more memory on purpose, but the budget still cut the footprint by a third to nearly two thirds where the queue alone barely moved it.

If you're curious, these tests were just run on my development box.

All of this is opt-in and off by default in JasperFx.Events 2.100 and Marten 9.58 or later, and it's all in the Marten documentation (as of today) under Bounding Daemon Memory with Per-Tenant Agents, including some advice on choosing a budget. The settings only apply to per-tenant agents. If you aren't partitioning by tenant, you already build one batch at a time per projection and none of this changes anything for you.

The rest of the scalability toolbox

Bounding memory on a node is one piece of running a very large event sourced system. The Marten documentation has a guide to optimizing for performance and scalability, and I went through all of it at length in Every Lever Marten Gives You for Scaling a Big Event Sourcing System. The ones I'd call out next to this story:

• Per-tenant event partitioning. The foundation for everything above. Each tenant gets its own partitions, its own event sequence, and its own projection progress, so one tenant's huge import or rebuild doesn't stall the others, and you can rebuild a projection for one tenant at a time.
• Stream archiving. The best thing you can do for a big database is make it a smaller one. Archive the streams that are finished, turn on the hot/cold partitioning, and every query and every projection rebuild only has to look at the events that are still live.
• Stream compacting. For the long-lived streams you can't archive, replace the old events with a snapshot of where the stream stood and optionally move the originals to cheaper storage. Less to load, less to replay, less to hold in memory.
• Read replicas. Send your IQuerySession reads to PostgreSQL standbys and leave the primary for the writes and the async daemon.

Notice that archiving and compacting are memory features too. Every event you don't have to load is an event that never counts against that budget.

Working with JasperFx on this kind of problem

Here's what the collaboration looked like in practice. The client's engineers know their system and their data better than we ever will, and they brought production heap profiles and a precise account of where the memory was going. We know the internals of the async daemon and what every change will cost the other people using it. Inside of a day we'd confirmed the diagnosis, shipped a first mechanism, built a harness to measure it, replaced it with a better one once the measurements told us to (after I was disappointed with the initial numbers), and fixed two unrelated bugs the harness shook loose. The "fix" in the end was just expanding our existing but naive back pressure feature for a single subscription agent to cover every agent in the process. Everything landed in the open source projects, off by default, so nobody else's system changed underneath them.

That's the JasperFx engagement model, and it's the same whether you have 52,000 projection agents or you're just starting to wonder why the daemon is a little further behind every afternoon or when you might or might not want to run projections asynchronously at all. If you're pushing Marten or Wolverine hard, or you're about to and want to get the tenancy and partitioning decisions right the first time:

• Our consulting services include architecture and performance reviews of your real system
• Our support plans put you in direct contact with the people who wrote the code
• And the Critter Stack Discord is always open if you'd just like to talk shop