Wolverine itself does have the ability to message specific nodes within its internals for agent assignment and leadership election, but we've never emphasized that for normal usage. Yet.

Almost everything about messaging in Wolverine is built around competing consumers. You publish a message, some number of running nodes are listening, and exactly one of them picks it up and handles it. That's what you want for the vast majority of commands and events, and it's what every message broker and every .NET messaging framework is optimized for.

Every so often though, you need the opposite. You've got a message that every running node in your cluster needs to see. The classic examples are:

• Throwing away an in-memory cache on every node after some reference data changed
• Pushing a configuration or feature flag change out to every node
• Resetting some kind of node-local state
• Pushing a notification out to every browser connected to your system, when those browser connections are spread across several servers

That last one is the one that comes up the most, and it's where I'd like to start.

Lead with NATS

A couple folks have recently been asking for extended NATS.io support specifically for being able to broadcast messages from Wolverine to connected user interface clients, and we had some plans laying around to do exactly that anyway, so here you go.

A good example of somebody wanting exactly this is the Stebet.SignalR.NATS project, which uses NATS as a backplane for ASP.NET Core SignalR. The problem it solves is that out of the box, a SignalR hub on one server can only reach the clients that happen to be connected to that server. If you've scaled your web application out to three nodes and you want to tell every connected user that something happened, you need a way to get that message to all three nodes so each can push it down its own websocket connections. Microsoft's scale out guidance points you at a Redis backplane or the Azure SignalR Service for this, but if you've already got NATS in your architecture, why not use it?

NATS is a natural fit here because broadcast is its default behavior. In Core NATS, every subscriber to a subject gets its own copy of every message published to that subject. Load balancing is the opt-in through queue groups, where the members of a group split the messages between them. That's the reverse of most brokers, where you have to do extra work to get a fan out.

Wolverine's NATS transport carries that behavior through. If you listen to a Core NATS subject without a queue group, every running node of your application gets a copy of every message on that subject:

builder.UseWolverine(opts =>
{
    opts.UseNats("nats://localhost:4222").AutoProvision();

    // No queue group, so every running node subscribes
    // independently and gets its own copy of each message
    opts.ListenToNatsSubject("catalog.changed").ProcessInline();

    opts.PublishMessage<CatalogChanged>().ToNatsSubject("catalog.changed");
});

And the handler is just a handler:

public record CatalogChanged(string Region);

public static class CatalogChangedHandler
{
    // This runs on every node, because every node
    // is subscribed to the subject
    public static void Handle(CatalogChanged message, IMemoryCache cache)
    {
        cache.Remove(message.Region);
    }
}

A couple of things to watch out for with this approach:

• If you set the transport-wide DefaultQueueGroup on your NATS configuration, every listener joins that group, including this one, and you're right back to competing consumers. If you need a mix of load balanced and broadcast listeners, leave the default alone and use UseQueueGroup() on the individual listeners that should be load balanced.
• Core NATS is at-most-once. A node that's restarting or disconnected when the message goes out simply misses it. For cache invalidation that's usually fine as long as a node rebuilds its cache on startup anyway, but you should know that going in.
• A named, durable JetStream consumer is shared by every node that connects to it, so that's competing consumers again. For broadcast, stick with Core NATS subjects.

That's a great option if you're on NATS. But what if you aren't, or you don't want to set up a dedicated subject for something you do once in a blue moon?

New in Wolverine 6.50: BroadcastToAllNodesAsync()

Wolverine has always had to talk to every node in its own cluster for its leader election and agent assignment, so the plumbing to reach every node was already sitting there. It just wasn't exposed in any kind of friendly way. As of Wolverine 6.50, it is with the new IMessageBus.BroadcastToAllNodesAsync() method:

public record InvalidateCache(string Region);

public static class InvalidateCacheHandler
{
    // Runs once on every node in the cluster when the message is broadcast
    // with IMessageBus.BroadcastToAllNodesAsync()
    public static void Handle(InvalidateCache message, IMemoryCache cache)
    {
        cache.Remove(message.Region);
    }
}

public static class CatalogEndpoint
{
    [WolverinePost("/api/catalog/{region}/refresh")]
    public static async Task Post(string region, IMessageBus bus)
    {
        // Every node in the cluster, including this one, gets a copy
        // of the message and handles it exactly once
        await bus.BroadcastToAllNodesAsync(new InvalidateCache(region));
    }
}

When you call BroadcastToAllNodesAsync(), Wolverine:

1. Hands the message to the current node's own local queue, where your normal handler for that message type processes it
2. Looks up every other active node from Wolverine's node storage and sends each of them a copy through that node's control endpoint

The control endpoint is the part that makes this transport agnostic. If you've opted into broker control queues with EnableWolverineControlQueues() (Rabbit MQ, Azure Service Bus, and Amazon SQS all support that), the broadcast rides on those per-node queues. Otherwise, it goes through Wolverine's own database-backed control queue in whatever message store you're using -- PostgreSQL, SQL Server, MySQL, Oracle, SQLite, RavenDB, or Cosmos DB.

What that means in practice is that you don't need to set up any broker topology at all to broadcast. That includes NATS. Wolverine's NATS transport doesn't have control queues today, so a broadcast from a Wolverine application using NATS goes through the database control queue, and it works exactly the same way.

A few other things I think are worth calling out:

• It honors the outbox. If you call BroadcastToAllNodesAsync() from inside a handler that's using Wolverine's transactional outbox, the broadcast doesn't go out until the transaction commits. For cache invalidation, that's exactly what you want. Nobody throws away their cache before the new data is actually committed.
• It doesn't leak out to your other routes. If InvalidateCache is also routed to a Rabbit MQ queue for some other reason, the broadcast does not send it there. Wolverine just runs the handler on every node.
• Solo mode just works. In Solo or Serverless durability mode, there's only one node to worry about, so the call just handles the message locally. You don't need a different code path for local development.
• It's testable. TestMessageContext records broadcast messages in a new BroadcastToAllNodes collection, so you can unit test a handler that broadcasts the same way you'd test one that publishes.

Pairing it with SignalR

CritterWatch heavily uses Wolverine's SignalR integration and we're pretty happy with how that's working out.

Now let's bring this back around to the SignalR problem I started with. Wolverine's SignalR transport lets you send and receive strong typed messages over SignalR as just another Wolverine transport. Publishing a message that's routed to SignalR sends it to the clients connected to the node that published it.

That's fine on a single node, but it's the exact same limitation that drove somebody to write a NATS backplane. With BroadcastToAllNodesAsync(), you can broadcast an internal message to every node, and have each node's handler cascade a websocket message down to its own connected clients:

// Internal message that goes to every node
public record AnnounceMaintenance(DateTimeOffset StartsAt);

// What the browser actually receives
public record MaintenanceScheduled(DateTimeOffset StartsAt) : WebSocketMessage;

public static class AnnounceMaintenanceHandler
{
    // Runs on every node, and the cascaded message goes out over
    // SignalR to the clients connected to *this* node
    public static MaintenanceScheduled Handle(AnnounceMaintenance message)
        => new(message.StartsAt);
}

with the SignalR routing configured like so:

builder.UseWolverine(opts =>
{
    opts.UseSignalR();

    opts.PublishMessage<MaintenanceScheduled>().ToSignalR();
});

Kick it off from anywhere with await bus.BroadcastToAllNodesAsync(new AnnounceMaintenance(startsAt));, and every connected browser hears about it, no matter which node it's attached to.

To be clear, I'm not telling you to throw out your Redis backplane. If you're pushing a high volume of messages to clients, a real backplane or the Azure SignalR Service (which Wolverine also supports through UseAzureSignalR()) is the right tool, and Wolverine reads the list of nodes from storage on every broadcast. But for the occasional server driven notification to everybody, this is one less piece of infrastructure to run.

Rabbit MQ and the other transports

Like I said up above, BroadcastToAllNodesAsync() works no matter which transport you're using, or even if you're using no external transport at all. If you're on Rabbit MQ, I'd recommend turning on Rabbit MQ control queues so your node to node traffic moves off of database polling and onto a non-durable Rabbit MQ queue per node:

opts.UseRabbitMq(new Uri("amqp://localhost"))
    // Use Rabbit MQ for inter-node communication,
    // including broadcasts
    .EnableWolverineControlQueues();

The same EnableWolverineControlQueues() switch exists for Azure Service Bus and Amazon SQS.

As for doing broadcast natively in each broker, it varies:

• NATS does it out of the box as we covered above.
• GCP Pub/Sub has a SubscriptionPerNode() option in Wolverine that gives every node its own subscription to a topic, so every node handles every message.
• Rabbit MQ can certainly fan out through an exchange, but you'd need a queue per node bound to that exchange, and Wolverine doesn't build that topology for you today.

Transport-native broadcast for the rest of the brokers is on our list as follow up work, but honestly, for most of the use cases I listed at the top, BroadcastToAllNodesAsync() is all you need.

Some caveats

I'd rather you hear about the rough edges from me than find them in production:

• It's fire and forget. Delivery to the other nodes isn't durable, and there's no acknowledgement or aggregation of results coming back from the other nodes. A "broadcast and wait for every node to answer" variant could come later if folks need it.
• Watch out for blue/green deployments. A node that doesn't have a handler for the message type, like an older version of your application, quietly discards the message, because control endpoints don't have a dead letter queue.
• Don't put it in a hot loop. Wolverine reads the current list of nodes from storage on every call. That's perfectly fine for occasional broadcasts and not so fine for thousands a second. If you need high volume fan out, that's a job for NATS or a real backplane.
• Dead nodes are skipped using the same DurabilitySettings.StaleNodeTimeout that Wolverine's leader uses to detect dead nodes.
• Tracked sessions can't see it. Messages to node control endpoints aren't visible to Wolverine's tracked sessions, so an integration test that verifies delivery on the other nodes has to poll for the expected outcome.

Wrapping up

This was a small feature, but it's one of those things that's surprisingly fiddly to get right by hand against your broker's topology. If you're on NATS, you've already got a great broadcast primitive and Wolverine lets you use it. If you're not, or you just want the simplest possible thing, BroadcastToAllNodesAsync() is now a single line of code on any transport.

If you're scaling out a Wolverine system and wrestling with exactly this kind of problem, that's the kind of thing JasperFx helps with every day through our consulting services and support plans. And as always, come talk shop with us in the Critter Stack Discord.