> For the complete documentation index, see [llms.txt](https://docs.triton.one/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.triton.one/solana/quickstart/streaming/fumarole-cluster-failover.md). # Fumarole cluster failover Set up Fumarole across regional clusters for automatic streaming failover. {% hint style="info" %} This guide shows how to implement cross-region failover for Fumarole from the subscriber side. For an overview of regional clusters and the redundancy model, see [Fumarole Persistent gRPC](/solana/real-time-streaming/fumarole-persistent-streams.md). {% endhint %} Fumarole runs as **independent regional clusters**. Persistent subscriber state does not replicate across regions. This means that if your primary cluster experiences a major outage, you can fail over to another region, but the cutover is fully customer-managed and requires you to recreate your persistent subscriber on the secondary cluster from the last slot you consumed. This guide walks through the full failover lifecycle: preparation, detection, cutover, and failback. ## Scope and assumptions This guide assumes: * You have an active Fumarole integration on one regional cluster (your **primary**). * You have access to a second regional cluster (your **secondary**); the same token works on both. * You can implement client-side logic that tracks per-cluster state and reacts to outage signals. The example throughout this guide uses **EU (`ams.rpcpool.com`)** as primary and **US (`nyc.rpcpool.com`)** as secondary. The procedure is symmetric; it applies in either direction. {% hint style="warning" %} Triton does **not** orchestrate failover for you; failover is **customer-managed**. We do not synchronize subscriber state, track your last-consumed slot, or trigger the cutover. Everything described below runs on your side. This page tells you exactly what to do. {% endhint %} ## Before you start Failover is only possible if you do the prep work ahead of time. Put these in place during normal operation, not during the outage. **1. Track the last fully-consumed slot per cluster.** Your client must keep a record of the highest slot you have successfully processed end-to-end from each cluster. Persist this value durably (e.g. in your database, in Redis, in a file) anywhere that survives a process restart. You will need it to recreate the subscriber on the secondary cluster at the correct position. Fumarole does not return the last slot you consumed through its API, so record it yourself. The client marks the end of each block (slot and blockhash) with a `SlotEnded` event, and several blocks can exist for the same slot. Record a slot once your processing of its block succeeds. **2. Build your outage-detection logic.** Decide what counts as a cluster-level failure for your workload (see Detecting an outage below) and instrument the client to surface it. **3. Make sure your processing pipeline is idempotent.** There will be a small overlap window during failover where the same slot may be delivered from both clusters. Your downstream processing must tolerate seeing the same slot twice without producing duplicate side effects. ## Detecting an outage A failover is only worth the effort when the primary is genuinely down. Transient hiccups should be handled by normal reconnect logic, not failover. Before triggering a failover, make sure the problem is on the cluster and not on your side. Some signals worth tracking: * **Stream interruption that does not recover on reconnect.** Fumarole is built to survive normal node restarts and rolling upgrades. If you have been disconnected for more than a few minutes and reconnect attempts keep failing, that is beyond normal recovery. * **`fume test-config` against the failing endpoint fails or hangs.** This is the most direct check that the cluster is reachable. * **No new slots advancing.** The `fumarole_offset_lag_from_tip` metric on your client should normally trend toward zero. If it is growing unbounded and your network is healthy, the cluster is not advancing the log. * **"Not found" or stale errors** for a persistent subscriber you know was healthy minutes ago. {% hint style="warning" %} **Check whether the problem is differential.** If only some of your clients are affected but others are fine, the problem is likely on your side: network, host, or deployment. Do not fail over. {% endhint %} If `fume test-config` succeeds against the failing endpoint and the subscriber exists, but you are not receiving data, that is worth a CS-channel ping before declaring a full failover. ## Failover procedure When your detection logic concludes the primary cluster is down, execute the following sequence. **Step 1: Stop the primary client** Halt consumption from `ams.rpcpool.com`. Stop reconnect attempts. **Step 2: Read the last fully-consumed slot for the primary** Read the value you have been tracking. This is the slot from which the secondary subscriber will resume. Call this `last_primary_slot`. **Step 3: Create the subscriber on the secondary cluster** Connect to the secondary cluster and create the persistent subscriber there, starting from `last_primary_slot + 1`. ```bash fume --config ~/.fumarole/config-us.yaml create \ --name my-app-failover \ --from-slot 312500001 ``` If a subscriber with that name already exists on the secondary cluster from a previous failover, delete it first: ```bash fume --config ~/.fumarole/config-us.yaml delete --name my-app-failover fume --config ~/.fumarole/config-us.yaml create \ --name my-app-failover \ --from-slot 312500001 ``` The same pattern applies whether you use the Fume CLI or the Rust / TypeScript SDK: create a fresh subscriber pointing at your recovery slot. The recovery slot must be within Fumarole's 4-day buffer. If the cluster no longer stores that slot, the create call does not fail: the subscriber starts at a later slot than the one you asked for, and you miss the gap. Check the stored range first with `fume slot-range` (`get_slot_range` in Rust, `getSlotRange` in TypeScript), and backfill a gap from historical sources. **Step 4: Begin consuming from the secondary** Start your client against `nyc.rpcpool.com`. Resume normal processing. Until the secondary catches up to live tip, you will be receiving historical data. Expect some replay delay proportional to the gap between `last_primary_slot` and the current chain tip. **Step 5: Confirm forward progress** Once the secondary is at live tip and slots are advancing normally, your failover is complete. Log the cutover and update your monitoring to track the new active cluster. ## Failing back to the primary When the primary cluster recovers, you have a choice: * **Stay on the secondary.** Acceptable if the secondary is geographically reasonable for your workload. * **Fail back to the primary.** Necessary if the secondary is far from your backend and you accepted the failover only for availability. If you decide to fail back, run the failover procedure in reverse: 1. Track the last fully-consumed slot on the secondary. 2. Stop the secondary client. 3. Create the primary subscriber starting from `last_secondary_slot + 1` (deleting the existing primary subscriber first if needed). 4. Start consuming from the primary. 5. Confirm forward progress. {% hint style="warning" %} Failback is a planned operation, not an emergency one. Schedule it during a low-traffic window and verify the primary has been stable for some time before cutting back. {% endhint %} ## Common pitfalls * **Failing over on a transient blip.** A single dropped connection is not a cluster outage. Tune your detection thresholds so normal reconnects do not trip failover. * **Not persisting the last-consumed slot.** If you only hold it in memory and your process restarts mid-outage, you have lost your resume point. Persist it durably. * **Forgetting the subscriber starts at `last_slot + 1`.** Starting at `last_slot` itself replays the slot you already processed, which widens the duplicate window. * **Assuming token-or-permission errors mean an outage.** Authentication failures are not outage signals; they indicate a client configuration problem, not a cluster problem. * **Leaving unused subscribers behind.** A subscriber on a cluster you no longer use still counts toward that cluster's [limit of 15 persistent subscribers per subscription](/solana/real-time-streaming/fumarole-persistent-streams.md#limits). Delete it with `fume delete --name `. ## Quick reference | Phase | What you do | | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Prep | Track last-consumed slot per cluster, build detection logic, ensure idempotent processing | | Detect | Run `fume test-config`, watch the `fumarole_offset_lag_from_tip` metric, and check for "not found" / stale errors on your subscriber. See the [Fume CLI reference](https://github.com/rpcpool/yellowstone-fumarole/blob/main/apps/yellowstone-fumarole-cli/README.md). | | Cut over | Stop primary client → read `last_primary_slot` → create subscriber on secondary from `last_primary_slot + 1` → start consuming → confirm progress | | Fail back | Same procedure in reverse, scheduled rather than reactive | *** :life-ring: Contact support by clicking the chat icon in your [customer dashboard](https://customers.triton.one)\ :briefcase: Sales questions? [Contact us](https://triton.one/contact)\ :sparkles: AI agent? Read [llms.txt](https://docs.triton.one/llms.txt)\ :rss: Follow updates: [Blog](https://blog.triton.one) · [X](https://x.com/triton_one) · [YouTube](https://www.youtube.com/@triton_one_ltd) · [Telegram](https://t.me/tritonone) · [GitHub](https://github.com/rpcpool)