Helping teams scale Marten systems is a big part of what JasperFx does day to day. If your system is creaking under its own success -- or you'd like to make sure it never does -- we'd love to help.

A new client more or less asked us "Will it scale?" about Marten this week. That's certainly a fair question, and we can happily say that a simple, out-of-the-box Marten configuration on a single PostgreSQL database will take the large majority of systems much further than folks expect. But some of you do have that system. The one with hundreds of millions or billions of events, thousands of tenants, a handful of ridiculously hot streams, or a reporting projection that's falling further and further behind every afternoon.

A lot of the work the Marten community and JasperFx have done over the past few years has been driven by teams with exactly those systems, many of them JasperFx clients. The result is that Marten now has a pretty deep bench of scalability features, but they've shipped a release at a time and are scattered across the documentation and Jeremy's blog. I wanted one place that lays them all out together, so here it is.

Roughly speaking, every lever falls into one of four buckets:

1. Do less work on every write
2. Keep the active data set small
3. Partition the data
4. Spread the work out, and be deliberate about when that work happens

I'm going to go through them in about the order I'd reach for them, from "free" to "you should probably talk to somebody first."

Start with the defaults: Quick Append

This is the default behavior with Marten 9.*, but an "opt in" setting for <= Marten 8.*.

The first lever is one you might already have pulled without knowing it. Marten's original "Rich" append mode did a two-step dance where the session first reserved event sequence numbers and figured out stream versions, then wrote the events. That gives you complete event metadata before the commit, but it's chattier, slower, and under heavy load it was the root cause of the "event skipping" problem where the async daemon's high water mark detection could stall on gaps in the event sequence.

The "Quick Append" mode pushes all of that into a single PostgreSQL function call. In our load testing that was a 40-50% reduction in the time spent appending events, and maybe more importantly, it made the daemon's sequence gap problems all but disappear.

As of Marten 9, Quick Append is the default. A brand new Marten 9 system gets EventAppendMode.QuickWithServerTimestamps without doing anything at all. If you're upgrading a Marten 8 system or you called RestoreV8Defaults() to get through a migration, you're still on Rich and it's worth an explicit opt-in:

builder.Services.AddMarten(opts =>
{
    opts.Connection(builder.Configuration.GetConnectionString("marten"));

    // The Marten 9 default. Only necessary if you're coming from
    // Marten 8 or previously called RestoreV8Defaults()
    opts.Events.AppendMode = EventAppendMode.QuickWithServerTimestamps;
});

The trade-off is that Inline projections and pre-commit session listeners don't get the final IEvent.Version or IEvent.Sequence values, because those are now assigned by the database during the insert. If you have inline projections that need the version, implement IRevisioned on the aggregate and let Marten take care of it. And if you were hand-rolling Append(streamId, expectedVersion, events) for optimistic concurrency, that API needs Rich mode -- but you should be using FetchForWriting() for that anyway, and we'll come back to that API later because it's carrying a lot of weight in this post.

Marten 9 also flipped lightweight sessions, UseIdentityMapForAggregates, and 64-bit event sequences to be the defaults. The migration guide has the full list, and my older post on making event sourcing with Marten go faster explains the reasoning behind each switch back when they were all opt-in.

It's a minor thing, but I consider the decision back in 2015 to have the identity map and its subsequent overhead turned on by default to be a mistake. I've called it "Marten's original sin." -- Jeremy

Keep the active data set small

Honestly, this is one of the best lessons you should learn early if you work on enterprise-y systems. Pay attention to archiving data out of the "hot" data sets if you can.

Here's an unglamorous truth about database performance: the single most effective thing you can do for a large database is to make it a smaller database. Event stores only ever grow, and nobody wants to delete events, so Marten gives you two complementary tools to keep the active data small without throwing anything away.

Archiving with hot/cold storage

Most event streams have a natural end. The order shipped, the incident was closed, the claim was paid. Once that happens, those events are just along for the ride in every index, every vacuum, and every query plan. Marten lets you archive a stream when it's finished:

session.Events.ArchiveStream(orderId);
await session.SaveChangesAsync();

Or better yet, make archiving part of your domain by having the workflow's own completion append Marten's built-in Archived event, in which case any single stream projection for that stream will archive the stream as it processes that event.

Archived events are excluded from event queries by default and ignored by the async daemon. But here's the part that trips people up: by itself, archiving only flips an is_archived flag. The rows are still sitting in the same table and the same indexes. We've heard "we archive aggressively and nothing got faster" from more than one team, and that's why. The lever that actually shrinks the hot data is the hot/cold storage partitioning:

opts.Events.UseArchivedStreamPartitioning = true;

With that flag, Marten uses PostgreSQL table partitioning on is_archived so that archiving a stream physically moves its rows out of the "hot" partition and into a "cold" one. The hot tables and their indexes shrink back down to your working set, and because Marten's default filter for events is exactly the partition key, PostgreSQL never even opens the archived partition for normal operations. Archiving is what gives the partitioning something to do, and the partitioning is what turns archiving into an actual performance win. You want both.

Do be aware that turning this on in an existing system is a real migration. PostgreSQL can't convert a populated table to a partitioned table in place, so plan to run that migration deliberately at a time of your choosing instead of letting it be discovered at application startup. This is much easier to adopt early, but as we'll see later in this post, Marten will do the heavy lifting of that migration for you.

Stream compacting

There's a lot of guidance in the Event Sourcing community about the importance of "keep your streams short," but Marten is hopefully much more flexible and gives you plenty of ways to make your system performant even when your upfront analysis wasn't perfectly omniscient.

Archiving handles streams that are finished. Stream compacting is for the opposite problem: the stream that's very much alive but has tens of thousands of events and keeps growing. Think about a piece of equipment reporting telemetry, or a long-lived account.

await session.Events.CompactStreamAsync<Equipment>(equipmentId, x =>
{
    // Compact everything older than 30 days...
    x.Timestamp = DateTimeOffset.UtcNow.Subtract(30.Days());

    // ...and send the old events off to cheaper storage first
    x.Archiver = coldStorageArchiver;
});

Compacting replaces all the events up to the cutoff point with a single Compacted<T> event carrying a snapshot of the aggregate at that point, and your single stream projections pick right up from that snapshot with no code changes on your part. Please do pay attention to that Archiver though. There is no default, and without one the compacted events are simply deleted. If you want that history later, it's on you to copy it to S3, blob storage, or wherever first. I wrote more about the thinking behind this feature in Stream Compacting in Marten 8.

I'll also say what I say to clients: an enormous stream is frequently a modeling smell. "Closing the books" -- ending a stream at a natural boundary and starting a new one from a summary event -- is often the better answer, and compacting is for when that ship has already sailed.

Make the events themselves smaller with binary serialization

Marten stores events as JSON by default, and you should keep doing that for most event types because it's queryable, human readable, and forgiving about versioning. But for the small number of high-volume event types that dominate a big system, the JSON serialization cost and storage size start to matter. Since Marten 9 you can opt individual event types into binary serialization with MemoryPack, MessagePack, or any format you like behind the small IEventBinarySerializer interface:

[BinaryEvent]
[MemoryPackable]
public partial record TelemetryRecorded(Guid EquipmentId, double Value, DateTimeOffset RecordedAt);

// From the Marten.MemoryPack NuGet
opts.Events.UseMemoryPackSerializer();

What I like most about how this turned out is that it's strictly per event type and purely additive. Binary and JSON events live side by side in the same table and even in the same stream, so you can roll this out to an existing production system for just your three noisiest event types with no data migration at all. The cost is that Marten's JSON-based upcasters don't apply to a binary payload, so you evolve a binary event by introducing a new versioned event type instead. There's more detail in my post on binary event serialization for Marten, which not coincidentally came directly out of optimizing a JasperFx client's system.

Spread out the reads with read replicas

Most systems read far more than they write. If your PostgreSQL primary is sweating from query traffic, Marten can send reads to your read replicas through Npgsql's multi-host data source support:

// Host=primary.db.com,replica-1.db.com,replica-2.db.com;Database=marten;...
services.AddMultiHostNpgsqlDataSource(connectionString);

services.AddMarten(opts =>
    {
        // Prefer the standby nodes for querying
        opts.Advanced.MultiHostSettings.ReadSessionPreference = TargetSessionAttributes.PreferStandby;
    })
    .UseNpgsqlDataSource();

Marten is deliberately conservative here. Only queries made through a read-only IQuerySession get routed to a standby. Anything done through an IDocumentSession, plus all the internal work like the async daemon, stays on the primary so your write side never makes a decision on stale data. That maps nicely onto a CQRS-style codebase where your query endpoints take IQuerySession and your command handlers take IDocumentSession. Just remember that replication lag is one more flavor of eventual consistency, so use this for the queries that can tolerate it. See Scaling Marten with PostgreSQL Read Replicas for more background.

Spread out the data with multi-tenancy

If your system is multi-tenanted, your tenants are the most natural seams you're ever going to get for splitting up data, and Marten gives you a progression of options. There's a lot more on this in my Multi-Tenancy in the Critter Stack post.

Per-tenant partitioning

The cheapest option operationally is "conjoined" tenancy where all the tenants share one database with a tenant_id column. At scale, the weak point of that model is that every tenant contends on the same event tables, the same indexes, and the same global event sequence, and the async daemon has one view of progress for the whole store.

Per-tenant event partitioning fixes that without making you run more databases:

opts.Events.TenancyStyle = TenancyStyle.Conjoined;
opts.Events.UseTenantPartitionedEvents = true;

// Tenants are registered through Marten, which builds out the partitions
await store.Advanced.AddMartenManagedTenantsAsync(token, "tenant-a", "tenant-b", "tenant-c");

Now every tenant has its own physical partitions of the event tables, its own event sequence, and its own projection progress. The async daemon runs each projection independently per tenant, so one tenant doing a monster import or a projection rebuild doesn't stall their neighbors, and you can rebuild a projection for just one tenant.

Know the constraints going in though: this requires conjoined tenancy and one of the Quick append modes, and today it cannot be combined with the hot/cold UseArchivedStreamPartitioning from earlier. Sub-partitioning by both tenant and archived status is on the roadmap, but for now you pick the one that matches your bottleneck.

You can do the same thing for your documents, including projected documents, with Marten-managed table partitioning by tenant.

Separate databases, and sharding across a pool of them

JasperFx built this functionality in cooperation with one of our clients who has a tremendously huge data set to the tune of hundreds of billions of events in their active databases.

The next step up is a database per tenant. This has always been Marten's answer for the strictest data isolation requirements, but it's also a perfectly good scalability play: every tenant database is small, independently tunable, and can live on whatever server you need it to. Marten and Wolverine handle the connection routing, the schema migrations across all the databases, and running the async daemon against each one.

The model breaks down a little when you have thousands of tenants, because nobody wants to run thousands of databases. For that, there's the newer sharded multi-tenancy with database pooling:

opts.MultiTenantedWithShardedDatabases(x =>
{
    // The master database that tracks tenant assignments
    x.ConnectionString = masterConnectionString;

    x.AddDatabase("shard_01", shard1ConnectionString);
    x.AddDatabase("shard_02", shard2ConnectionString);
    x.AddDatabase("shard_03", shard3ConnectionString);

    // Or assign new tenants to the smallest database,
    // or explicitly, or with your own strategy
    x.UseHashAssignment();
});

This spreads your tenants over a pool of databases, partitions each tenant's data within its database, and lets you add databases to the pool at runtime. This model was designed for systems targeting hundreds of billions of events, so if you were wondering how far this whole "PostgreSQL as an event store" idea can be pushed, the answer is "quite a ways."

Segment the events by module with ancillary stores

This strategy is effective for scalability just because it effectively partitions the event data between modules, and to oversimplify things, smaller tables == better performance.

Not every system has tenants, but nearly every big system has modules. If you're building a modular monolith, you can give each module its own completely separate Marten store with ancillary stores:

public interface IOrderStore : IDocumentStore;
public interface IInventoryStore : IDocumentStore;

builder.Services.AddMartenStore<IOrderStore>(opts =>
    {
        opts.Connection(connectionString);
        opts.DatabaseSchemaName = "orders";
    })
    .IntegrateWithWolverine();

builder.Services.AddMartenStore<IInventoryStore>(opts =>
    {
        opts.Connection(inventoryConnectionString);
        opts.DatabaseSchemaName = "inventory";
    })
    .IntegrateWithWolverine();

The scalability angle is that each store has its own event tables, its own event sequence, and its own async daemon work. The high-volume Inventory module is no longer interleaving its events with the low-volume Orders module, projections in one module never have to skip past the other module's events, and a rebuild in one module leaves the other alone. You can start with separate schemas in one database, then later move your busiest module to its own database server by changing a connection string. That's a nice option to have in your back pocket.

With the Wolverine integration, you still get the transactional outbox, the aggregate handler workflow, and everything else against each store. See Wolverine 5 and Modular Monoliths for more.

Be deliberate about projection lifecycles

I think this is the most underused optimization in Marten, and it doesn't involve any infrastructure at all. Every projection in Marten runs with one of three lifecycles, and you get to choose per projection:

Lifecycle	Consistency	Cost on writes	Cost on reads
Live	Strong	None	Loads and replays the stream's events every time
Inline	Strong	Projection is updated in the same transaction as the events	None, the document is already there
Async	Eventual (but keep reading!)	None, the work moves to a background process	None, but the document may lag

There's no universally right answer, which is exactly why it's a per-projection choice:

• Short streams that are written more than they're read? Live is free on the write side and cheap enough on the read side. It's a fine default for "write model" aggregates.
• Streams that are read constantly, or too long to replay on demand? Inline gives you a persisted snapshot that's always exactly current, at the cost of some work in every command.
• Anything that combines multiple streams, or gets hammered by concurrent writes? Async. Multi-stream projections updated inline under concurrent load are a contention problem waiting to happen, and the async daemon batches its work in a way that's flat out more efficient than updating one event at a time.

Since the lifecycle is just configuration, you can start simple and change your mind later case by case as your production metrics tell you where the real pressure is.

Async projections with strong consistency

If you use Wolverine with Marten and the "aggregate handler workflow" (the Decider pattern), Wolverine is using this Marten/Polecat/Fisher API under the covers.

The usual objection to moving a projection to Async is "but my command handlers need the current state." Here's the thing: with Marten they can have both. If you use FetchForWriting() in your command handlers -- and you really, really should -- Marten completely encapsulates the projection lifecycle for you:

public static async Task Handle(ShipOrder command, IDocumentSession session)
{
    // Strongly consistent, optimistic concurrency protected,
    // and it works no matter what the lifecycle of Order is
    var stream = await session.Events.FetchForWriting<Order>(command.OrderId);

    if (stream.Aggregate.CanShip())
    {
        stream.AppendOne(new OrderShipped());
    }

    await session.SaveChangesAsync();
}

When Order is an Async projection, FetchForWriting() fetches the last persisted snapshot and any events captured since that snapshot in one database round trip, then applies just those trailing events in memory to "fast forward" the aggregate to the exact current state. Your command handler gets strong consistency and optimistic concurrency protection. Your commands stop paying for the projection update. The daemon does that work more efficiently in batches in the background. And a stream with 10,000 events only ever replays the handful that came in since the last snapshot. The read side equivalent is FetchLatest().

If you're using Wolverine's aggregate handler workflow, you're already using FetchForWriting() under the covers:

[AggregateHandler]
public static class ShipOrderHandler
{
    public static IEnumerable<object> Handle(ShipOrder command, Order order)
    {
        if (order.CanShip()) yield return new OrderShipped();
    }
}

That same "catch up" ability is what makes zero downtime, blue/green deployments of projection changes possible, which I wrote about in Projections, Consistency Models, and Zero Downtime Deployments with the Critter Stack. And for a truly hot aggregate, Marten 9.26 added an opt-in cache so that FetchForWriting() can skip even the snapshot load:

opts.Projections.Snapshot<Order>(SnapshotLifecycle.Async);
opts.Events.CacheAggregatesForWriting<Order>(sizeLimit: 1000);

The cached snapshot is only ever a baseline. Marten still reads the stream version and any newer events every time, so a stale cache entry costs you a slightly bigger query and never a wrong answer. There's more in the optimization guide, which also covers daemon tuning like CacheLimitPerTenant, BatchSize, and the event type index for faster rebuilds.

Partition your projected data too

It's easy to forget that the documents your projections write are just Marten documents, which means everything in Marten's document database feature set applies to them, including table partitioning:

// Keep the completed orders out of the way of the active ones
opts.Schema.For<OrderSummary>()
    .PartitionOn(x => x.Status, x =>
    {
        x.ByList()
            .AddPartition("completed", "Completed")
            .AddPartition("cancelled", "Cancelled");
    });

// Or spread a huge projected document table over hash partitions
opts.Schema.For<CustomerActivity>()
    .PartitionOn(x => x.CustomerId, x => x.ByHash("one", "two", "three", "four"));

You can partition by list, range, or hash on any document member, by tenant, by soft-deleted status, or on rolling date ranges where your retention policy becomes an instant DROP TABLE on an old partition instead of a giant DELETE. It's the same "keep the active data small" thinking as the hot/cold event storage, applied to your read models. More in Making Marten Faster Through Table Partitioning.

You don't have to write the partitioning SQL

Let's stop for a second, because PostgreSQL table partitioning has now shown up four different times in this post: hot/cold storage for archived events, per-tenant event partitioning, the sharded tenancy model, and your projected documents. In every one of those cases, the entire cost of admission was a line or two of Marten configuration.

That's a much bigger deal than it looks like, because Marten is provisioning and managing all of that partitioning for you. PostgreSQL's native table partitioning is a fantastic feature, but the SQL to set it up can be dauntingly complicated if you don't live in it every day. Doing it by hand means knowing that:

• A partitioned table is declared differently from the start, as in PARTITION BY LIST (...), and then every partition is its own table that you create, name, and attach to the parent
• The partition key has to be part of the primary key and every unique index, which ripples out into your other DDL
• You need a default partition, or a plan for the rows that don't match any partition, or your inserts start failing
• Adding tenant number 501 on a Tuesday afternoon means creating new partitions in every partitioned table, in every database, while the system is running
• And all of it has to be exactly the same in development, testing, staging, and production

With Marten, partitioning is just one more part of the configured database schema that Marten already manages. The same schema migration tooling you're already using for your document tables and indexes will detect the difference between your configuration and the actual database, and either apply the changes or write out the migration script for your DBA to review:

# Write out the SQL for any outstanding changes, including partitions
dotnet run -- db-patch partitioning.sql

# Or just apply the changes
dotnet run -- db-apply

And when partitions have to change at runtime, as with new tenants, that's a method call and not a DDL script:

await store.Advanced.AddMartenManagedTenantsAsync(token, "tenant-501");

You can adapt later

Here's the part I most want you to take away from this section. PostgreSQL can't ALTER a populated, ordinary table into a partitioned one. That's why partitioning has a reputation as a decision you have to get right on day one, before you have any way of knowing where your real bottleneck is going to be.

Marten can retrofit partitioning onto an existing, non-partitioned database. When you turn on hot/cold event storage or add partitioning to a document type that already has data, Marten's migration takes care of the whole dance of copying the existing data off to the side, replacing the table with a partitioned version, and copying the data back into the right partitions. For moving a big existing store to per-tenant event partitioning, there's a purpose-built migration utility that moves one tenant at a time, preserves your event sequence numbers, can be resumed if it's interrupted, and never touches the source tables so that rolling back is just "keep using the old store."

I'm not going to tell you it's free. Rewriting a big table takes real time and disk space, so you'll want to generate the script, run it by hand during a maintenance window, and take a backup first instead of letting it happen as a surprise at application startup. But the difference between "schedule a maintenance window" and "sorry, we'd have to rebuild the system to do that" is enormous. It means you can start with the simplest possible Marten setup today, find out from real production metrics where the pressure actually is, and then adopt exactly the partitioning your system turns out to need.

Spread out the work with Wolverine

Here's where Marten's sibling comes in. Once you've moved your projections to Async, all of that projection and event subscription work has to run somewhere. Marten's built-in "Hot/Cold" daemon mode elects a single node to run everything, which is simple and reliable right up until that one node is your bottleneck and the rest of your cluster is sitting there idle.

Add Wolverine and you can have it manage the distribution instead:

builder.Services.AddMarten(opts =>
    {
        opts.Connection(connectionString);
        opts.Projections.Add<OrderSummaryProjection>(ProjectionLifecycle.Async);
        opts.Projections.Add<CustomerActivityProjection>(ProjectionLifecycle.Async);
    })
    .IntegrateWithWolverine(m =>
    {
        // This replaces AddAsyncDaemon(DaemonMode.HotCold)
        m.UseWolverineManagedEventSubscriptionDistribution = true;
    });

Wolverine takes over with its own leader election and agent assignment to spread the projections and subscriptions evenly across every running node, rebalance as nodes come and go, and fail work over when a node dies. It also composes with everything else in this post:

• With per-tenant partitioning, the work fans out per tenant across the cluster
• With multiple databases, all of one database's agents are grouped on one node to keep your connection counts flat
• Ancillary stores get distributed right along with the main store
• In a blue/green deployment, a new projection version is only assigned to nodes that actually have that code
• It doesn't depend on PostgreSQL advisory locks, which makes life behind PgBouncer a lot less exciting

I wrote about the original motivation in Scaling Event Projections and Subscriptions with the Critter Stack. And yes, CritterWatch will show you where every one of those agents is running across your cluster, how far behind each projection is, and let you pause, restart, or rebuild them.

So which levers should you pull?

Not all of them, and certainly not all at once. If I were triaging a system, I'd roughly go in this order:

1. Get onto Marten 9 and its defaults, especially Quick Append. That one is nearly free.
2. Look at your projection lifecycles. Moving the right projections to Async while keeping FetchForWriting() in your command handlers is a big win for a small change.
3. If you're running multiple nodes with async projections, turn on the Wolverine-managed distribution.
4. Start archiving finished streams and turn on the hot/cold partitioning, the earlier the better.
5. Reach for binary serialization and stream compacting for your specific hot event types and streams.
6. Bring in read replicas if it's query load that's hurting you.
7. When the problem is just sheer volume, use your natural seams: per-tenant partitioning, separate or sharded databases, and ancillary stores per module.

The early items are cheap and safe. The later ones are genuine architectural decisions that are far easier to make with real measurements and some experience behind them, which brings me to...

How JasperFx can help

Every feature in this post exists because a real system needed it, and many of them came directly out of JasperFx client work. The questions we get pulled into are the ones this post has been circling the whole time: why is the async daemon falling behind? Which tenancy and partitioning strategy fits where this system is headed? How do we move a live production database to hot/cold storage without drama? Which handful of event types or streams are responsible for most of the load?

If your Marten system is growing faster than you're comfortable with, or you're about to build something big and want to get these decisions right on day one instead of year three, this is exactly the work we do. Our consulting services include architecture and performance reviews of your actual system, and our support plans give you direct access to the people who built every lever in this post. It's almost always cheaper to talk to us before the system is on fire, but we're happy to help either way. And as always, come talk shop with us in the Critter Stack Discord.