If your team runs Wolverine in production and wants help proving out its durability story for your own infrastructure, or just wants a second set of eyes on a system that has to not lose messages, that's what our support plans are for.

One of the key points I try to make during workshops to JasperFx clients is that we want to build resilient systems that by and large do not lose ongoing work and hopefully very rarely require manual intervention. To that end, Wolverine's transactional inbox and outbox tries to guarantee that if your handler's transaction commits the message will be handled once and any cascading messages will go out.

That's a pretty standard goal for the transactional outbox pattern, but as the Wolverine community has learned over the years, there's plenty of challenge in that promise due to environmental factors that aren't always easy or obvious to recreate during development like a database being temporarily offline or network hiccups or application failures at exactly the wrong time or the million and one ways things can go sideways when a running process is being shut down with work already in flight.

Long story short, Wolverine has had enough reported issues recently where it lost messages in certain conditions that I felt like it was time for yet another round of exploratory and chaos testing to try to identify any other ways or places where our durability code could conceivably lose in flight work somehow. The result of that multi-day effort was our Wolverine 6.63 release today.

Like I said, this exercise resulted from us receiving several issues where maybe a transactional boundary wasn't quite perfect, or a CancellationToken wasn't carried through exactly right, or order of operation issues in process shutdown, or "what if you get an exception at exactly this point?" I admittedly started three days of testing by roughly telling an AI agent to look at several recent issues we'd already fixed, then to do chaos and exploratory testing investigations of all our relational database backed durability options including database queues and scheduled message delivery and frankly, try to break Wolverine -- then fix whatever it found!

That's this big ol' blog post just to try to explain how we did the testing and also to try to prove out our commitment to deep technical quality within the Critter Stack.

Why our existing tests weren't enough

Wolverine has a very large test suite. Every persistence provider has its own compliance suite for the inbox, the outbox, scheduled messages, dead letters and leader election, and those tests are what let us support PostgreSQL, SQL Server, MySQL, Oracle, SQLite, RavenDb and CosmosDb from one codebase without going insane. That part's all good, but problems were still leaking through that coverage.

But a compliance test runs in one process and asks questions like "after I store this envelope and mark it handled, is it handled?" It wasn't (yet) asking "what happens if the process is SIGKILLed between the second and third statement of this move?" or "what if the database accepted the commit but the client timed out waiting for the reply?" With enough users, it's basically inevitable that somebody is going to make those cases happen.

A couple of user reports earlier this autumn made it clear that "done" was optimistic. There was a SQLite queue that could commit the dequeue before the inbox row was written, so a crash in between lost the message outright. The MQTT transport could silently drop messages across a broker reconnection, and NATS JetStream sends could be lost without a word. The HTTP transport would answer 200 when the durable receiver had failed to store the batch, so the sender thought it was done and the messages were gone. These are the kind of bug you mostly find by having something go wrong at the wrong time, and our users were finding them for us. That's great in the sense that the community is engaged enough to dig into a failure and file a precise report, but it's more fun if the Wolverine core team can find and fix those kinds of things before they can happen out in the wild.

On to how we've tried so far to identify and fix more issues.

The durability chaos rig

This was my first exposure to Toxiproxy and my take is that it's pretty darn useful for exactly what we did with it.

The rig lives in the Wolverine repository at src/Testing/DurabilityChaosRig. It is deliberately outside the solution and outside CI, because it is slow, it is destructive, and it needs Docker. Here's the shape of it:

• A producer process commits batches of ten rows to a ledger table, and in the same transaction publishes a Work message for each row through the outbox. A row is in the ledger exactly when its outbox row is, which is the whole point of the outbox.
• Several worker processes listen on a durable queue. Each Work message's handler writes a processed row for it in the handler's own transaction, the same transaction that marks the inbox row handled. Some messages are flagged to fail on their first attempt and come back through a scheduled retry. Some cascade a local message, some cascade a delayed message, so the outbox and the scheduled message machinery are exercised too.
• A driver starts all of that, then spends two minutes doing unpleasant things to it on a seeded random schedule: SIGKILL a worker, SIGTERM a worker so it has to drain gracefully, SIGSTOP a worker for twenty seconds so the cluster ejects it as a stale node and then SIGCONT it so it wakes up still holding messages in memory, docker pause the database or the broker for a few seconds, and lately, inject latency and connection resets through Toxiproxy between the nodes and PostgreSQL.
• When the chaos is over, the driver stops the producer and waits for the system to settle. Then it reads the ledger and asks the only questions that matter: for every row the producer committed, is there exactly one processed row for each stage? Not zero (lost), not two (duplicated). Is the inbox empty? Is the outbox empty? Are there dead letters?

A clean run looks like this:

CLEAN: sent 16190, lost work 0, lost local 0, lost delayed 0, duplicates 0,
       inbox incoming 0, inbox scheduled 0, outbox 0, dead letters 0

A dirty run names the message ids, which nodes and which attempts committed them, and the envelope ids of each commit, so that you can follow one message through the worker logs and find the exact sequence of events that produced the duplicate. That last part turned out to matter more than anything else. Nearly every bug below was found by following one envelope id through three log files.

The rig runs against every store and transport combination we support: Marten and EF Core on PostgreSQL, EF Core on SQL Server, plain ADO.NET handlers on MySQL, Oracle and SQLite, the PostgreSQL, SQL Server, MySQL, Oracle and SQLite queue transports, RabbitMQ, global partitioning over sharded PostgreSQL queues, and a database-per-tenant configuration where one tenant's database gets paused while the others keep working.

TIP

The rig's README covers how to run it against your own infrastructure. Every knob is an environment variable, and the README spells out which Docker containers each configuration needs. If you want to try to break Wolverine against your own database or broker setup, please do, and please file what you find.

What three rounds found

I'm going to go through these roughly in the order we found them, because the order tells a story: each fix made the next, rarer bug visible.

Round one: the easy ones

The first runs were against the PostgreSQL queue with Marten, and they weren't clean. The biggest single finding wasn't even a durability bug. It was that Wolverine's default envelope ids could collide between two processes on one machine. We generate sequential GUIDs with MassTransit's NewId, and the default configuration folds in the MAC address and a per-process sequence but not the process id, so two Wolverine processes on the same host could draw the same id. In a two-process probe, 9,233 of 200,000 ids were shared. In production that would show up as a primary key violation on the inbox table when a handler cascades, and as a different message being silently discarded as a duplicate delivery. It looked exactly like a durability bug, and it took a while to realize it wasn't one. Wolverine now uses its own NewId generator that includes the process id.

WARNING

If you run several Wolverine processes on one host, this one alone is worth upgrading for. It has been in every version of Wolverine that used NewId.

Round one also found that a dead letter marked replayable while the durability agent was moving it could be deleted without ever reaching the inbox, and that the scheduled retry on the SQL Server, MySQL, Oracle and SQLite queues was two autocommitted statements, so a command timeout between them left the message in neither table.

Round two: the ambiguous commit

The second round moved to SQL Server with EF Core, added five second command timeouts, and started pausing the database. That is where the rig found the bug I find the most interesting of the whole exercise: an EF Core commit that succeeds on the server but reports an error to the client.

Pause the database at the moment the client sends COMMIT. The server applies it when it wakes up. The client times out and throws. Wolverine saw an exception from the handler, so it dead-lettered the message. But the handler's work had committed, including the mark that retired the inbox row, and not including the cascaded messages, which were still sitting in memory on the node that thought it had failed. So the dead letter table said the work was never done, the ledger said it was, and the cascades were stranded on a live node that recovery would never touch, because recovery only reclaims rows from dead nodes.

Wolverine now checks, after a failed commit, whether the inbox row is handled. If it is, the commit went through, the message is not dead-lettered, and the cascades are flushed. This is the kind of bug that no amount of reading the code will find. You have to make the database lie to you.

Round two also fixed a PostgreSQL pop that rolled back its whole batch on one inbox conflict, a SQL Server scheduled-message poll that was a regular deadlock victim under normal load, and a slow drain after PostgreSQL outages where a queue waited a whole polling interval per batch to work through a backlog.

Round three: the resumed node

The third round, which finished yesterday, added MySQL, Oracle and SQLite, dead letter replay racing kills, Toxiproxy, global partitioning and database-per-tenant. It ran twenty-odd two-minute legs over two days and found six more.

The thread running through most of them is the stale node that resumes. Pause a worker long enough and the leader ejects it: its node record is deleted and every inbox row it owned is released for recovery. Another node picks those rows up and handles them. Then the paused worker wakes up, still holding the same envelopes in memory, and finishes them. In the simplest case the inbox row is already marked handled and the mark-as-handled matches zero rows, but the transaction committed anyway, so the handler's work was done twice. Wolverine now treats "my mark-as-handled changed nothing and a Handled row exists for this identity" as proof that someone else finished first, and fails the transaction instead of committing it. That one fix is on every relational store and on the Marten path.

Fixing it exposed the next one. If the resumed node's handler failed instead of succeeding, it asked the queue to schedule a retry, and the retry wasn't fenced on who owned the inbox row. So it deleted the other node's inbox row and wrote a second scheduled copy, and both copies ran. Now the retry's delete is conditioned on the row's owner_id being this node, and the scheduled copy is written only if that delete hit, in one transaction.

And fixing that exposed the last one, which was only reproducible on MySQL with command timeouts and which took the better part of an evening to pin down. The move of an outboxed message from the outbox to a database queue was an INSERT and a DELETE in one command, with no transaction, on MySQL, SQL Server and SQLite. A kill between the two left the message in the queue and in the outbox, and outbox recovery sent it again. Usually the inbox catches the second copy as a duplicate. But if the first copy's first attempt had failed and gone to a scheduled retry, the scheduled retry had already deleted its inbox row, so the second copy sailed through and both ran. One duplicate in fifteen thousand messages, under exactly the right sequence of a timeout, a retry and a kill. PostgreSQL was never affected, because Npgsql runs a multi-statement command in one transaction. The other three now do too.

The rest of round three:

• MySQL's expired-handled cleanup locked live inbox rows and deadlocked with handler commits. It now works from an index on (status, keep_until) in bounded batches.
• Oracle's in-transaction mark-as-handled had no fence at all, so committed work ran again when connections failed. The first Oracle run had 362 duplicates. After the fix, zero.
• SQLite in Balanced mode registered every node as node 1 and parsed health check times as local, so released inbox rows were never recovered after a restart. The first SQLite run lost 4,727 of 18,150 messages. Now it's clean.
• Two tenant databases with the same name on one host but different ports shared one durability agent, because the agent's identity didn't include the port. The second tenant's scheduled messages never ran at all. This one was found by the very first run of the database-per-tenant leg, which is a good argument for adding legs.

Where it stands

Every configuration the rig covers is now clean, meaning zero lost and zero duplicated across repeated runs with different seeds, on main as of 6.63.0. The remaining dead letters in the runs are transient database errors on the handler's own commit, a MySQL deadlock or a connection failure while the database is paused, and they are dead-lettered because the rig deliberately configures no retry policy for them. In an application you'd have an OnException<MySqlException>().RetryWithCooldown(...) rule and never see them.

As always, testing is a way to find and remove problems in code, but I don't think you can ever say that testing proves that the code in question is perfect. I can happily say that the chaos testing we did this round was a very good way to find bugs, and the fact that round three found six of them after rounds one and two had made things "clean" should tell you that there are probably more (#sadtrombone), in combinations we haven't run yet. What we did do is drop deterministic tests to act as regression tests to try to prevent the problems from ever recurring.

As much as the usage of AI changes everything, I still think that many of the lessons about software development I learned from Extreme Programming back in the 00's still ring true. Which definitely includes the old advice to "fix" bugs by dropping in new test coverage that hopefully keeps the problem from ever being reintroduced.

What this says about how we work

I'll end where the Boring on Purpose post from July did. The work in this post added no features. Nobody is going to put "the outbox-to-queue move on MySQL is now one transaction" on a conference slide. But if you have bet a system on Wolverine's durable messaging, this is the work you are actually paying for, whether you pay for it with a support plan or with the time you spend filing a good bug report.

Three things I'd like you to take from it:

1. We go looking. The user reports that kicked this off were real and we fixed them, but the majority of the bugs above were never reported by anyone. They were found because we built a tool whose only purpose is to make Wolverine fail and then went through every failure down to the code path. That is a standing commitment, not a one-off.
2. The community is part of the loop. The reports that pointed us at SQLite, MQTT, NATS and the HTTP transport were precise enough to act on, and several of the people who filed them stuck around to verify the fixes. That is what makes an open source project's quality compound rather than decay.
3. Nothing is "done." Wolverine's inbox and outbox are years old and run in a lot of production systems. The community and JasperFx still found a dozen ways to lose or duplicate a message in them in two months of looking, and I would rather tell you that than pretend otherwise. Marten's async daemon went through the same thing earlier this year. The whole Critter Stack gets this treatment, and it will keep getting it.

Inevitably, we have to talk about the impact of AI on software development. I'm convinced that AI usage is allowing our users to observe and describe latent bugs in libraries and frameworks like Wolverine that probably would have gone a lot longer undetected. You can also observe this from the reports of PostgreSQL 19 being delayed because of how many bugs are being found by AI. The AI tool usage is also making it much cheaper in terms of our time to go build out performance or chaos testing harnesses like what we did to find and fix the issues in the Wolverine 6.63 release -- but, it's still on us the developers and testers of the world to tell an AI agent what to go do and with some hopefully specific advice on where to look and the type of failures to simulate.