Marten has been beaten up and improved through many years of constant usage from our community. The "Critter Stack" also benefits from being an active OSS community as well as having JasperFx Software managing it. In recent months, we've made several improvements to Marten based on community feedback, contributions, and reported issues. As this post tries to help prove, the big advantage of using Marten for Event Sourcing instead of trying to roll your own framework is that Marten has achieved a great deal of robustness through being constantly beaten up by real life usage and the resulting improvements over time.

Not surprisingly in retrospect, some of the hardest technical challenges we've faced with the Critter Stack is in handling a sudden surge in activity in a system. Like the kind that happens in systems that need to process large file imports that ingest external data in potentially large batches. We've already gone through many rounds of improvements to Wolverine's ability to successfully handle large surges that potentially spawn thousands of outgoing messages all at once.

Last week though, we had to dive into Marten's "Async Daemon" subsystem when we got some unfortunate feedback that the venerable async daemon was vulnerable to "event skipping" when a flood of parallel transactions was happening due to a large import file process.

What the "high water mark" is, and why you should care

Some attempts at a PostgreSQL backed event store try to side step the sequence problem by using a global advisory lock to effectively make all event writes be sequential. That's a great way to hamstring your application performance.

Marten's async projection daemon processes events in order. To do that safely it tracks a high water mark: the highest event sequence number that is known to be both committed and contiguous -- meaning there are no gaps below it. Projections are only ever allowed to run up to that mark. Everything at or below it is settled history; everything above it is still potentially in motion.

The subtlety lives in how PostgreSQL hands out those sequence numbers. Event sequence numbers come from a Postgres sequence, and a transaction reserves its number the moment it appends -- but it might not commit for a while after that. It might take a slow network round trip. It might be doing other work in the same transaction. It might roll back. It might crash. So at any given instant the committed events can look like ... 998, 999, 1001, 1002 with 1000 missing -- a gap -- because the transaction that grabbed 1000 hasn't committed yet.

That gap is the whole ballgame. If the high-water detector advances the mark past 1000 while 1000 is still held by a live transaction, then when 1000 finally commits, it's already below the mark. Projections will never look at it again. The event is real, it's durably stored, and it will never be projected. That is the cardinal sin the detector exists to prevent, and everything below is in service of never committing it.

The correct behavior is deeply conservative: when you see a leading gap, hold the mark just before it (marten#4964 tightened exactly this on the normal path). Wait. The gap either fills in when the slow transaction commits, or it becomes a permanent hole because the transaction died. You do not get to guess which -- not without evidence.

Attain the sequence reservation as late as possible

A simple lesson we learned over time is to simply try to attain the sequence numbers just in time to make the write operation. That all by itself helped tremendously when we introduced the now default QuickAppend mode in Marten. Recently Marten got several contributions from the community that further tightened up our sequence assigning so that optimistic concurrency violations are detected to reject a transaction before any event sequences can be reserved and inevitably lead to gaps.

The subtle READ COMMITTED bug (and why silent is the scary word)

Here's the one that genuinely unsettled me when we found it: marten#4953.

Under PostgreSQL's default READ COMMITTED isolation, each SQL statement takes its own snapshot. That's normally fine and even desirable. But Marten's gap detection was historically spread across multiple statements inside a single command -- the gap detector issued three statements, and the statistics detector two. Each of those statements saw its own, slightly-later view of the database.

Now play out the race. A transaction is sitting on a reserved sequence number. The detector runs:

-- statement 1: is there a leading gap right after the current mark?
--   (runs, sees the in-flight number still uncommitted -> returns null, "no gap here")

-- ...the in-flight transaction COMMITS in this window...

-- statement 2: what does the contiguous tail look like now?
--   (new snapshot -> the gap is gone, the tail looks perfectly contiguous)

-- statement 3: max() fallback
--   (advances the mark right over the just-committed number)

Each statement told the truth about the snapshot it saw. The problem is that they saw different snapshots, and the composite decision was nonsense. The leading-gap probe returns null, the interior query sees a contiguous tail, and the max() fallback happily advances the Normal high-water mark straight over a sequence that -- from the standpoint of the decision as a whole -- was never safely settled.

And the worst part: this skip was silent. No skip record. No log line. No metric. The page loader then came along behind it and advanced the ceiling, sealing the mistake. If you're going to have a bug, at least have the decency to be loud about it. A silent skip is the worst kind, because the first evidence you get is a customer asking why a read model is missing data that provably exists in the event store.

The fix is almost anticlimactic in how correct it is: each detector now issues exactly one statement, using scalar subqueries over a single snapshot. Every reading the detector makes is now mutually consistent by construction -- they all describe the same instant in time. Same hold rule, same gap edges as before. Just consistent. You cannot lose a race between two statements when there is only one statement.

Proof of death before you skip

Consistency fixes the multi-snapshot race, but it doesn't answer the older, harder question: what do you do about a gap that just... sits there? A sequence number reserved and then never committed. Is that a transaction that's merely slow, or one that rolled back or crashed and left a permanent hole? A wall clock genuinely cannot tell those two apart. "It's been ten seconds" describes a dead transaction and a slow one equally well.

The old answer was a heuristic -- a magic "-32" that could, in the wrong conditions, pin the daemon in a Stale state forever whenever a dead tail happened to be shorter than 32. Not great in either direction: it could stall on real dead gaps, and it was reasoning from a number rather than from facts.

marten#4953 also introduced the transaction-evidence model, and it's now on by default via ProjectionOptions.UseTransactionEvidenceForGapSkipping (the old behavior is still there behind the opt-out). The idea is to stop guessing and go look for evidence that the reserving transaction is actually still alive before we ever consider skipping.

When a gap gets stuck, we start a per-gap clock. From that same single-snapshot reading we record three things: when the detector first saw the mark pinned at this spot, the snapshot's xmax, and the reserved ceiling as of that observation. Then, once we're past a stale threshold measured from that first observation, a GapLivenessProbe goes looking for concrete evidence the reserving transaction may still be alive:

• granted row-exclusive locks on the mt_events table lineage held by transactions that started before our observation (from pg_locks, which is visible to all roles),
• open transactions from before the observation (pg_stat_activity),
• in-progress write transaction ids below the observed snapshot xmax (pg_snapshot_xip -- pure MVCC data, no special privileges).

The decision rule is intentionally biased toward caution. Any evidence of life means hold -- a slow append gets waited out, exactly as it should. Only a gap proven dead gets skipped. And even then, we never skip beyond the reserved ceiling recorded at first observation, because any sequence numbers reserved after that instant belong to newer transactions whose liveness we have not yet proven -- they get their own chance. Every skip now logs the exact skipped range at Warning level. No more silent skips. If Marten steps over an event, it says so, out loud, with the range.

The per-tenant path (under UseTenantPartitionedEvents) got the same GapLivenessProbe fencing, so per-tenant stale-gap skips honor transaction evidence too. Same discipline, everywhere gaps can happen.

The watchdog that couldn't watch

All of the above is driven by the HighWaterAgent poll loop -- the thing that actually wakes up and runs the detector. It doesn't matter how correct your detection logic is if the loop running it quietly dies. jasperfx#524 fixed two ways it could.

First, every wakeup await in the loop was unguarded. Marten uses a LISTEN/NOTIFY wakeup, and if that wakeup throws -- say, it's trying to reconnect a dropped LISTEN connection while the database itself is down -- the exception would propagate straight out of the loop and permanently kill high-water progress. One bad wakeup during a database blip, and your projections stop advancing until you restart the process. The wakeup is now treated as untrusted: a throw is logged, the loop falls back to a plain delay, and it keeps going.

Second -- and this one is a lovely little async footgun -- the loop was launched with Task.Factory.StartNew over an async delegate. That returns a Task<Task>, and the outer task completes successfully at the very first await. So the watchdog whose entire job was to restart a faulted loop was inspecting a task that could never fault, and could therefore never trigger a restart. The real loop could be face-down in a ditch and the watchdog would report everything green. It's now .Unwrap()ed, so the fault on the actual loop is observed and the loop actually gets restarted.

The unglamorous work is the point

None of this makes a good conference slide. There's no new API to show off, no benchmark to brag about. It's a consistency fix, an evidence model, and a watchdog that finally watches. But this is exactly the kind of work that determines whether you can trust Event Sourcing in production for years, not weeks. The high water mark is invisible when it works, and catastrophic when it doesn't, and "invisible when it works" is the entire job.

All of these are on by default now, with opt-outs available if you need the previous behavior for some reason -- but I'd encourage you to stay on the new path. It's more conservative in exactly the places that matter and louder in exactly the places you'd want to know about.

If your team is leaning on Marten's async projections for anything that matters -- and especially if you've ever stared at a read model wondering where an event went -- this is the kind of thing the JasperFx team lives in every day. We offer commercial support and consulting for the Critter Stack, and you're always welcome to come talk through your setup with us in Discord.