JasperFx Software shipped quite a few behind the scenes improvements to the Critter Stack this past week -- mostly brought on by our work with our clients. Much of this was also related to getting CritterWatch and Wolverine's agent assignment subsystems to be performant and usable for an extremely complicated, high volume system. There's not much new functionality here, but the key point we hope to make is that the Critter Stack is constantly curated and improved based on real community and JasperFx client usage.

This also doesn't show up in outwardly facing features, but we're also undergoing a large effort to optimize the asynchronous messaging support and improve both the reliability and runtime of the CI builds for Marten, Polecat, Wolverine, and CritterWatch right now to help us be able to deliver faster.

We shipped a lot between July 26th and today. Twenty-two releases across Marten, Wolverine, Polecat, Weasel, and the shared JasperFx libraries — and enough of it matters that a pile of release notes isn't a fair way to communicate it.

So here's the week, sorted by how much it should change what you do on Monday.

Upgrade first: a PostgreSQL injection fix in Marten 9.22.1

Marten 9.22.1 is a security release, and it's the one item here with an actual deadline attached.

A tenant id was interpolated into a double-quoted PostgreSQL identifier without doubling an embedded double quote — so a tenant id containing " could terminate the identifier and run additional SQL. This is a different class from the two advisories previously published against Marten, both of which were the single-quoted string literal class. Neither of those fixes covered this one.

You are affected only if all three are true: you use sharded tenancy, you have Events.UseTenantPartitionedEvents enabled, and your application passes attacker-influenced input as a tenant id. Applications that draw tenant ids from a trusted fixed set are not exploitable. Affected versions are 9.4.0 through 9.22.0. It's not likely that you are vulnerable in real usage, but still, we got the report, it's valid, and we fixed it. We've also made some other structural changes to centralize more of the SQL identifier scrubbing logic to try to stop this kind of vulnerability across the entire stack for any usage of Critter Stack controlled database partitioning by tenant id.

Worth knowing about the reachable surface: it isn't limited to administrative provisioning calls. GetTenantAsync and FindOrCreateDatabase auto-provision an unknown tenant, so ordinary session resolution reaches the same path.

The fix has three parts. PerTenantEventSequences.QuotedSequenceName now escapes embedded quotes to match quote_ident/%I, so the name still resolves to the object the quick-append function finds — covering create, drop, schema-apply, and cleanup. BulkEventAppender no longer rebuilds an unquoted sequence name from a suffix read back out of the tenants table, which also fixes a real functional bug: PreserveSourceSequence bulk imports had been failing with 42601 for hyphenated and GUID tenant ids under sharded tenancy. And ShardedTenancy now validates tenant ids destined for DDL, closing a long-standing asymmetry with DefaultTenancy.

It needs Weasel.Postgresql 9.21.1, which escapes partition bound values. Both halves are required; the dependency comes along automatically.

Credit to Barak Srour (Apiiro) for the report.

Weasel got a second, related hardening pass in the same week. PostgresqlMigrator.AssertValidIdentifier is the only identifier check in the entire stack — DbObjectName and PostgresqlObjectName deliberately do no validation — and it had been permitting exactly the two characters that let an object name escape the statement it's written into: " closes a quoted identifier and ; starts a new one. Both are rejected now, along with all whitespace rather than just the literal space, so a newline can't smuggle in a -- comment. The other four database providers were brought in line with PostgreSQL, and DbObjectName/PostgresqlObjectName are now documented as not being sanitizing boundaries — because they never were.

Two silent data-loss bugs in Wolverine

Wolverine 6.24.0 closed two bugs that destroyed data quietly instead of failing loudly. Both are unusual usages, but "unusual" is cold comfort if you're the one hitting it.

Durable inbox rows were orphaned when a circuit breaker tripped. DurableReceiver checked its latched flag before calling MarkReceived. The latched path still persists each envelope to the inbox as a safety net — but an envelope that never went through MarkReceived has Status at the enum default (Outgoing) and a null Destination. Both are filter columns for inbox recovery, so those rows landed in a state no recovery sweep on any node could ever see. The null Listener also skipped the nack back to the broker, and the broker's redelivery after restart hit DuplicateIncomingEnvelopeException — which acks and drops. Net effect: genuine message loss under a durable inbox, any time a circuit breaker trip latched the receiver mid-flight. Measured against the circuit-breaker suite, 9 of 1,200 messages were lost per run.

Dropping one tenant from a shared partition bucket destroyed its co-tenants' data. Tenant bucketing — registering several small tenants against one partition suffix so they share a physical partition — is documented and exposed through PartitionPerTenant(p => p.AllowPartitionSharing = true), and it did not work on either PostgreSQL or SQL Server. It had no test coverage, because the documentation sample demonstrating it is compile-only and never executed. That's a lesson we took to heart this week, and it shows up again further down.

Two more from the follow-up patches worth calling out. In 6.24.1, RabbitMQ acks were cumulative — every ack went out as BasicAckAsync(tag, multiple: true), acknowledging every lower delivery tag on the channel. That's only correct when completions happen in delivery order, which they emphatically do not with ConsumerDispatchConcurrency > 1. Acking one message silently acknowledged deliveries whose handlers were still running, and a crash at that moment lost them. Acks are per-message now. And in 6.24.3, the durable receiver now settles unacked deliveries when the inbox database is unavailable, rather than leaving the broker holding them.

New: rolling time-window partitions

This is related to CritterWatch work for usages when you are letting CritterWatch itself track performance metrics from Wolverine instead of using a dedicated metrics tool like Prometheus

This maybe starts to veer into the territory of being an external, software managed equivalent to pg_partman.

The headline feature of the week. Declaring every partition up front only works while the set of partitions is fixed. Real time-series storage needs the set to move — provision next month, drop last year — and until now that meant hand-writing the DDL and the maintenance job that runs it.

ManagedRangePartitions landed in Weasel for both PostgreSQL and SQL Server, and surfaces in Marten 9.22.0 and Polecat 5.9.0 as ByRollingRange:

opts.Schema.For<MetricsSample>()
    .Duplicate(x => x.BucketEnd)
    // Keep 12 months of history, provision 3 months ahead. Marten creates the partitions at the
    // leading edge and drops the aged ones at the trailing edge -- no application-authored DDL.
    .PartitionOn(x => x.BucketEnd,
        x => x.ByRollingRange(PartitionPeriod.Month, periodsAhead: 3, periodsBehind: 12));

The window — periods retained behind, the current period, periods provisioned ahead — is a pure function of the policy and the clock, and that's what makes it safe to run automatically. A window that has rolled forward differs from the database by exactly one new partition at the leading edge and one aged partition at the trailing edge. The schema migration only ever adds the new one, so rolling forward never triggers the destructive table rebuild that a moved list of declared ranges would. Partitions are named after the period they cover (m202607, d20260730, y2026), and a DEFAULT overflow partition always exists, so a row written outside the provisioned window is stored rather than rejected.

PartitionPeriod supports Hour, Day, Week, Month, and Year. The maintenance pass — roll forward, then drop below the retention floor — runs at startup alongside the schema changes Marten already applies, so ApplyAllDatabaseChangesOnStartup() is all the wiring there is. Full details in the Marten storage docs.

This one came out of CritterWatch needing it at field scale, which is becoming a familiar pattern — running our own monitoring console against large production installations keeps turning up features that belong in the underlying libraries rather than in the console.

Global partitioning goes wide

Wolverine 6.24.0 pushed the global partitioning epic considerably further: global partitioning topologies for PostgreSQL and SQL Server queues, plus end-to-end sharded-processing suites for Azure Service Bus, GCP Pub/Sub, NATS, Redis Streams, and Pulsar. The scenario is lifted into Wolverine.ComplianceTests.Partitioning.ShardedProcessing, so a new transport now costs one small test class to cover.

Which is the point, and the new suites proved it immediately by finding two real bugs:

• NATS global partitioning had never worked at all. The topology forces EndpointMode.Durable on every slot, and a NatsEndpoint only supports Durable when it's JetStream-backed — so every UseShardedNatsSubjects() call threw at configuration time. The topology now enables JetStream on its own endpoints and declares a work-queue stream per shard, without which the listener died at startup on stream not found.
• Pulsar named its companion local queues off the full topic path, producing queues like global-persistent://public/default/orders1. They use the topic's short name now, matching every other transport.

Documentation for the whole feature, including per-transport native alternatives, is on the Wolverine partitioning page.

Production hardening for projection agent distribution

This was the largest single body of work in the week, spread across Wolverine 6.23.0, 6.23.1, 6.24.1, and 6.24.3. Nearly all of it came from real production clusters, and much of it from watching CritterWatch observe those clusters.

The short version: Wolverine no longer panics doing agent assignments during Kubernetes rollouts or cold cluster starts. 6.23.0 carried a wave of eight fixes to the assignment plane that together remove the re-assignment churn and livelock that could leave projection agents flapping or wedged — heartbeats no longer block behind command execution, node resurrection restores the node's real identity instead of a skeleton record, a pending-assignment ledger suppresses duplicate AssignAgent floods, batch starts are chunked and bounded-parallel, ejection gets hysteresis plus leader protection, and local agents drain with bounded parallelism on shutdown.

Paused shards now say why. IEventSubscriptionAgent.Failure surfaces a ShardFailure carrying the category, the failing event's sequence and type, and the root exception type — through health checks, a new IWolverineObserver.AgentPaused hook, and a NodeRecordType.AgentPaused record. Failures bound to a specific event (ApplyEvent, EventSerialization, UnknownEventType) or to two processes racing one shard (ProgressionOutOfOrder) are no longer auto-restarted, because they would die on the identical event every time. That classification runs all the way down into JasperFx.Events 2.36.x and into Polecat 5.7.0, so it's uniform across both event stores.

A few of the individual fixes are worth reading on their own:

• Restarting a paused projection appeared to do nothing for a full minute (6.23.1). The pending ledger introduced in 6.23.0 only counted an assignment as confirmed if a later evaluation saw the agent running and still assigned to the same node. A pause makes those two conditions mutually exclusive, so the entry was never confirmed and the restart's AssignAgent was suppressed as a duplicate still in flight — for 2 × CheckAssignmentPeriod, or 60 seconds with the defaults. Long enough that an operator reasonably concludes the agent is never coming back.
• wolverine_node_records grew without bound (6.24.1). A reporting cluster reached 36,135,221 rows and 16 GB in five days, on a diagnostic table nothing on the hot path reads. Three distinct defects stacked up: the delete method was implemented for every relational store and never invoked outside tests; the pruning that did run bounded the table by age only, which is no ceiling at all at high write rates; and that age sweep's hourly throttle was dead, because its backing field was never assigned — so a full-table delete went out every recovery cycle, every five seconds by default. There's a new Durability.NodeRecordRetention (default 10,000 rows) and Durability.NodeRecordPruningPeriod (default hourly), and Sqlite, MySQL, and Oracle gained implementations they'd also been missing.
• A comma in an agent URI voided a whole batch confirmation (6.24.1). Agent URIs embed tenant ids and projection names, and the batch commands joined their Uri[] on a comma, which RFC 3986 permits unescaped in a path segment. One comma shattered an agent into fragments, and because the read side built the array in a single projection, the resulting throw took out the confirmation for the entire batch. Newline is the delimiter now, entries are parsed individually so a bad one names itself, and the comma remains the default on the wire for payloads that don't contain one — so rolling upgrades keep working in both directions.
• Agent commands wait on observed progress, not the clock (6.24.3), and reassignments run in the source node's lane and batch by AgentStartBatchSize — which is what finally takes the shutdown of a Balanced-mode host off a full agent reply window.

Event skipping and the high-water mark

Related, and the direct continuation of the high-water detection work we wrote about earlier.

Marten 9.18 added protection against skipping events over gaps that a slow transaction might still fill. The side effect: a session parked in an open transaction for the life of the process reads as a permanent "possible reserver" and can hold the high-water mark — and therefore every async projection — behind a gap that is genuinely dead forever.

Marten 9.20.1's allocation fence keeps idle advisory-lock sessions from holding gap skips indefinitely, and 9.22.0 extended it to read is_called so that mark 0 is fenceable too. On the other side, Wolverine's long-held locks were already shaped correctly — leader election and node coordination hold session-scoped advisory locks on a dedicated connection with no transaction, so they present as state='idle' with a null xact_start — and those sessions are now also tagged application_name = 'wolverine-advisory-lock:<database>', which turns a pg_stat_activity investigation from guesswork into something you can read at a glance.

There's a trap in there worth knowing for your own code, now pinned in tests and in the Postgres durability docs: never add a keepalive query inside a long-lived open transaction. It bumps state_change, makes the session look active, and re-promotes it to candidate reserver — which is precisely the failure mode you were trying to avoid.

If you're running combined Marten + Wolverine deployments: older guidance suggested Postgres's idle_in_transaction_session_timeout as a dead-gap backstop. Prefer upgrading Marten and using SkipStaleGapsDespiteLiveTransactionsAfter instead.

Multi-tenancy

Polecat 5.8.0 brings SQL Server up to parity with Marten on runtime tenant management: MasterTableTenancy now implements JasperFx.MultiTenancy.IDynamicTenantSource<string>, the store-agnostic abstraction Marten's tenancies already implement, and AddPolecat registers it in DI when master-table tenancy is configured. Add / disable / enable / remove round-trip against SQL Server with no CritterWatch release required — a Polecat-backed service's Tenants tab is simply editable now.

Two details matter if you're adopting it. AllActive() returns tenant database identifiers rather than raw connection strings, so credentials never reach an admin dashboard. And the DI registration is deliberately conditional — single-database and static MultiTenantedDatabases() stores leave GetServices<IDynamicTenantSource<string>>() empty, which is the signal consumers use to fall back to a read-only tenant list. One behavior change came with it: AddPolecat(Action<StoreOptions>) now builds the StoreOptions eagerly and delegates to AddPolecat(StoreOptions), exactly as Marten's AddMarten(Action<StoreOptions>) does, so the configure lambda runs at registration time rather than on first IDocumentStore resolution. Details in the Polecat multi-tenancy docs.

On the Wolverine side, EF Core gained tenant partition back-fill. Routine migration deltas deliberately leave Weasel-managed partitions alone, which means a table joining an existing managed set — a newly deployed service, or a newly mapped ITenanted entity — had no partition for any tenant registered before that table existed. IConjoinedTenantPartitions<T>.MigrateTenantPartitionsAsync() reconciles every partitioned table against the full registered tenant set, with per-table TenantPartitionResult reporting. See back-filling a table that joins late.

Smaller things you'll actually notice

Marten and Polecat both gained FetchStreamStatePlan and FetchStreamPlan — raw event stream fetches expressed as batchable query plans — plus StreamEventState and StreamEvents result types for Marten.AspNetCore and its Polecat equivalent. Thanks to @uniquelau for the Marten implementation.

AddMartenHighWaterHealthCheck's databaseFilter got a provider-aware overload. The existing filter is captured at registration time and therefore can't resolve services, which makes it unable to express "the databases this node currently owns" — precisely the case under Wolverine-managed daemon distribution, where agents are assigned per (database, tenant) and rebalanced over a node's lifetime.

Wolverine also picked up a standalone force-catch-up entry point for Marten under managed distribution, which had previously only been reachable as a TrackActivity() stage:

await host.PauseThenCatchUpOnMartenDaemonActivityAsync();
await host.PauseThenCatchUpOnMartenDaemonActivityAsync(CatchUpMode.AndDoNothing);
await host.PauseThenCatchUpOnMartenDaemonActivityAsync<IMyStore>();

It deliberately never calls IProjectionDaemon.CatchUpAsync — doing that under a live coordinator is what produces the ProgressionProgressOutOfOrderException and pk_mt_event_progression duplicate-key errors that test suites have been retrying around for a while. Resuming the agents that already own the shards means there's only ever one writer.

A grab bag of the rest:

• RabbitMQ consumer dispatch concurrency is now per-endpoint. The client default of 1 was the bottleneck. At 2,000 msg/s offered load over a 30-second window, throughput went from 163.7/s at the default to 1,999.1/s at a concurrency of 20 — with transit p50 at 1.486 ms. The 12× multiple actually understates it, because at 1 and 5 the listener never catches up at all.
• Amazon SQS batches message deletions and chunks outgoing batches against the 256 KB request limit.
• Azure Service Bus session listeners are no longer quadratic — the n² session loops are now n, and MaxConcurrentCalls is surfaced.
• MQTT's v5 authentication method name is configurable. It was hardcoded to "OAUTH2-JWT", and Azure Event Grid's custom JWT authentication requires CUSTOM-JWT — so those brokers couldn't be reached through Wolverine's authentication support at all. You could always set it by hand through MqttClientOptionsBuilder.WithAuthentication(), but that gave up Wolverine's token refresh loop, which is the whole reason to use MqttJwtAuthenticationOptions. You no longer have to choose.
• Wolverine HTTP fails fast at startup when an endpoint advertises a request body its HTTP method can't carry, returns 415 rather than 404 for a missing Content-Type on an [AcceptsContentType] route, parses enums case-insensitively from query strings and form posts, and describes the message body for explicitly-routed PublishMessage<T> / SendMessage<T> chains in OpenAPI.
• Marten mt_quick_append_events returned {NULL} for an empty event array, because array_length('{}', 1) is NULL in PostgreSQL rather than 0. The resulting InvalidCastException displaced whatever exception had actually made the append fail — so callers got an unrelated, non-retryable error instead of the real one, which for the reporter dead-lettered Wolverine messages that would otherwise have been retried. No database migration is required to take the fix; see the migration guide for why.
• OrderBy against a dictionary indexer dropped the key, so OrderBy(x => x.SomeDictionary["key"]) generated SQL that ordered arbitrarily rather than failing loudly.
• Lazy LINQ sequences serialized as objects under Newtonsoft. A document property holding a deferred-execution sequence was written as an iterator object rather than a JSON array, so it wouldn't round-trip.
• NgramIndex now matches NgramSearch's unaccent-aware mt_grams_vector expression — thanks to first-time contributor @dat-honguyen.

One documentation fix is worth flagging even though nothing changed at runtime: if you implement IMessageBatch yourself, know that the async daemon raises projection side effects from multiple threads at once — measured at up to 8 concurrent publishers across 10 threads for a single-stream projection catching up. The interface previously said nothing about this. An implementation that appends to an unsynchronized collection will silently drop messages.

CritterWatch RC 3

CritterWatch reached RC 3, and it was almost entirely about being a good database citizen at field scale. The ServiceSummary document was 78% progression data, rewritten in full on every two-second telemetry push and always TOASTed at 378× the threshold — the mechanism behind a measured 19,266 autovacuums.

Moving that state into per-shard documents keyed on the canonical (service, store, database, tenant, shard) tuple collapsed a 5.2× duplication across four parallel dictionaries:

	before	after
ServiceSummary document	755.6 KB	156.0 KB	−79%
telemetry push (every 2s)	16.6 ms	6.2 ms	−63%
write volume	23.0 MB/min	7.0 MB/min	−70%

Then the largest remaining item — a ShardStatesUpdated event appended every 15 seconds per monitored service — moved to MessagePack + LZ4 in Marten's bdata column:

	JSON	binary
one event on the wire	225.9 KB	4.7 KB	−97.9%
mt_events on disk (incl. TOAST)	424.0 KB	168.0 KB	−60.4%
append p50 / p99	3.92 / 8.68 ms	1.04 / 3.15 ms	−73% / −64%
steady state	904 KB/min	19 KB/min

Read those two rows as different claims: PostgreSQL already LZ-compresses TOASTed jsonb, so JSON never occupied its wire size on disk. The wire number predicts the append win; quote the disk number for storage.

No migration is required — Marten's EventsTable adds the bdata column unconditionally, so it arrived with the 9.20.2 bump and was simply empty. Binary and JSON rows coexist, because Marten discriminates per row on bdata IS NULL, which makes the opt-in reversible. SQL Server / Polecat keeps JSON event serialization for now; Polecat has no IEventBinarySerializer yet. That's lossless, just larger.

Under the hood

Two things happened this week that won't show up in anyone's application, but are worth knowing about if you care how this stuff gets maintained.

Every test suite in the organization moved to xUnit v3, and Marten, Polecat, and JasperFx moved onto the Microsoft Testing Platform. Along the way we reinstated the xUnit1051 analyzer and threaded cancellation tokens through — 504 analyzer warnings cleared in the JasperFx libraries alone.

Cross-store event sourcing compliance moved into a shared package. For a long time, Polecat's test suite carried hand-mirrored copies of Marten's event sourcing tests, which is exactly as durable as it sounds. There's now a JasperFx.Events.ComplianceTests source-only package, and both stores run the same suite. Two waves of migration retired the mirrored pairs, and — like the sharded-processing compliance suites in Wolverine — it found real bugs the moment it ran, including an unwrapped DcbConcurrencyException from SaveChangesAsync in Polecat.

Both of these are the same bet: bugs that only one implementation had test coverage for are the ones that reach production. The tenant bucketing data-loss bug at the top of this post existed because the documentation sample proving the feature worked was compile-only and never executed. We'd rather find those ourselves.

Getting current

Everything here is on NuGet:

dotnet add package Marten --version 9.22.2
dotnet add package WolverineFx --version 6.24.3
dotnet add package Polecat --version 5.9.1

The Marten security fix is the one to prioritize if you're on 9.4.0 through 9.22.0 with sharded tenancy and UseTenantPartitionedEvents. Everything else can move at your normal cadence.

Questions, or something in here that broke for you? Find us in the Critter Stack Discord or through ProductSupport — and if you're running any of this at a scale where a week like this one matters, our support plans exist for exactly that conversation.