> For the complete documentation index, see [llms.txt](https://docs.triton.one/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.triton.one/solana/quickstart/how-tos/index-solana-history-with-superbank.md). # Index Solana history with Superbank Run Superbank on a single node to index the full Solana ledger into ClickHouse and serve a Solana-compatible JSON-RPC endpoint. Superbank is an open-source Solana transaction indexer built by Triton. It ingests the full ledger into ClickHouse and exposes a Solana-compatible JSON-RPC endpoint, giving you your own queryable copy of the chain, with no rate limits and sub-millisecond lookups on historical data. This guide walks through running Superbank on a single node from scratch: ClickHouse setup, DDL, ingestion configuration, and the RPC server. It also covers how to filter the index to only your app's transactions. ## Self-hosting vs Triton managed Self-host when: * You need full control over what's indexed (e.g. filter to only your program's transactions) * You have compliance or data-residency requirements * Your query volume is high enough that operating your own node is cheaper than per-request pricing * You want to run custom queries directly against ClickHouse Use Triton's managed endpoints when: * You want the fastest path to production, with no ClickHouse operations and no infra * You don't need to index historical data beyond what Triton retains * You'd rather not maintain the ingestor and RPC layer yourself Both options expose the same RPC interface, so you can start managed and migrate to self-hosted later without changing your client code. ## How the components fit together Superbank has three parts, each running as a separate process: ``` [Dragon's Mouth / Fumarole / RPC] ↓ superbank (ingestor) ← Rust binary; runs on your host or in a container ↓ ClickHouse ← runs in Docker (this guide) or native install ↑ superbank-rpc ← Rust binary; runs on your host or in a container ↓ Your app / JSON-RPC clients ``` * **ClickHouse** is the storage layer. The two binaries connect to it over HTTP (port 8123). * **`superbank`** reads from your ingestion source and writes rows into ClickHouse. * **`superbank-rpc`** reads from ClickHouse and serves the Solana-compatible RPC API. None of the three components need to run on the same machine; they only need to be network-reachable to each other. For a single-node setup, running all three on one host is the simplest approach. ## What Superbank provides Once running, Superbank serves these RPC methods against your local ClickHouse: | Method | Notes | | ---------------------------------------------------------- | --------------------------------------------------- | | `getSignaturesForAddress` | Standard + `beforeSlot`/`untilSlot` extensions | | `getSignatureStatuses` | | | `getTransaction` | Standard + `slot` hint extension | | `getBlock` | | | `getTransactionsForAddress` | Custom method: pagination + token account filtering | | `getBlockHeight`, `getSlot`, `getLatestBlockhash` | Chain tip queries | | `getBlockTime`, `getBlocks`, `getBlocksWithLimit` | | | `getHealth`, `getFirstAvailableBlock`, `minimumLedgerSlot` | | | `getInflationReward`, `getTransactionCount` | | Adding `processed` commitment (in addition to `confirmed`/`finalized`) requires the optional gRPC head-cache feature, covered at the end of this guide. ## Prerequisites * **ClickHouse**, 26.x or later. Docker is the fastest way to get one running. * **Rust** 1.98.1, which the repository's `rust-toolchain.toml` pins (the workspace uses edition 2024). Install via [rustup.rs](https://rustup.rs). * **A Dragon's Mouth gRPC endpoint or Fumarole subscription**, for live ingestion. Get one at [customers.triton.one](https://customers.triton.one). For backfill only, the JSON-RPC source works with any public endpoint. * Git and standard build tools (`gcc`, `pkg-config`, `libssl-dev` on Linux / Xcode Command Line Tools on macOS). ## Build and run {% stepper %} {% step %} ### Clone and build ```bash git clone --recurse-submodules https://github.com/solana-rpc/superbank.git cd superbank # Build both binaries cargo build --release -p superbank -p superbank-rpc ``` The build produces `./target/release/superbank` (ingestor) and `./target/release/superbank-rpc` (RPC server). {% endstep %} {% step %} ### Start ClickHouse You have three options. **Option A: Docker (recommended for getting started)** The fastest path, with no installation required beyond Docker itself: ```bash docker run -d \ --name superbank-clickhouse \ -p 127.0.0.1:8123:8123 \ -p 127.0.0.1:9000:9000 \ --ulimit nofile=262144:262144 \ -e CLICKHOUSE_DB=default \ -e CLICKHOUSE_USER=default \ -e CLICKHOUSE_PASSWORD="" \ -e CLICKHOUSE_SKIP_USER_SETUP=1 \ clickhouse/clickhouse-server:26.1.2.11 ``` This ClickHouse container has no password and is for local development only. Its ports are bound to `127.0.0.1`, so only this machine can reach them. **Option B: Native install (recommended for production)** For production servers, install ClickHouse directly on the host for better performance and easier disk/memory tuning. Follow the [official install guide](https://clickhouse.com/docs/en/install) for your OS. Use version 26.x or later. **Option C: ClickHouse Cloud or Altinity** If you'd rather not manage ClickHouse infrastructure yourself, both [ClickHouse Cloud](https://clickhouse.cloud) and [Altinity.Cloud](https://altinity.com/cloud-database/) offer hosted ClickHouse. Point `clickhouse-url` at your cloud endpoint and set credentials accordingly. Check it's up: ```bash curl -s http://localhost:8123/ping # → Ok. ``` {% endstep %} {% step %} ### Apply the DDL Superbank uses three base tables and several materialized views. Apply the local (single-node) DDL: ```bash # Base tables cat ddl/local/transactions.sql | docker exec -i superbank-clickhouse clickhouse-client --multiquery cat ddl/local/blocks_metadata.sql | docker exec -i superbank-clickhouse clickhouse-client --multiquery cat ddl/local/entries.sql | docker exec -i superbank-clickhouse clickhouse-client --multiquery # required for the gRPC and Fumarole sources # Materialized views (derived by ClickHouse at insert time - not by the ingestor) cat ddl/local/gsfa.sql | docker exec -i superbank-clickhouse clickhouse-client --multiquery cat ddl/local/signatures.sql | docker exec -i superbank-clickhouse clickhouse-client --multiquery cat ddl/local/token_owner_activity.sql | docker exec -i superbank-clickhouse clickhouse-client --multiquery ``` What each table is for: | Table/View | Type | Contents | | ---------------------- | ----------------- | ---------------------------------------------------------------- | | `transactions` | Base table | Every transaction, partitioned by epoch | | `blocks_metadata` | Base table | Block-level data: blockhash, height, rewards | | `entries` | Base table | PoH entry metadata (Fumarole and gRPC sources only) | | `gsfa` | Materialized view | Powers `getSignaturesForAddress`, 32-bucket address partitioning | | `signatures` | Materialized view | Powers `getSignatureStatuses`, 32-bucket signature partitioning | | `token_owner_activity` | Materialized view | Powers token-filtered `getTransactionsForAddress` | No filtering happens at the ingest layer. The ingestor writes every transaction from the source as-is into the `transactions` base table. The materialized views are computed by ClickHouse at insert time. This means: * If an account doesn't appear in a transaction, it won't appear in `gsfa`. * gSFA exclusions (below) are a DDL concern, not an ingestor concern. * The base `transactions` table is always the full unfiltered ledger. #### gSFA default exclusions By default, four system-level addresses are excluded from the `gsfa` materialized view. They appear in almost every transaction so they'd dominate the index without adding value: * **System Program**: `11111111111111111111111111111111` * **Vote Program**: `Vote111111111111111111111111111111111111111` * **SysvarClock**: `SysvarC1ock11111111111111111111111111111111` * **SysvarSlotHashes**: `SysvarS1otHashes111111111111111111111111111` This exclusion is in `ddl/local/gsfa.sql` as the `gsfa_ignored_addresses` array. If you need any of these addresses indexed, remove them from the array before running the DDL: ```sql -- In ddl/local/gsfa.sql, find this block and remove any address you want to keep: [ CAST(base58Decode('11111111111111111111111111111111') AS FixedString(32)), CAST(base58Decode('Vote111111111111111111111111111111111111111') AS FixedString(32)), CAST(base58Decode('SysvarC1ock11111111111111111111111111111111') AS FixedString(32)), CAST(base58Decode('SysvarS1otHashes111111111111111111111111111') AS FixedString(32)) ] AS gsfa_ignored_addresses ``` {% endstep %} {% step %} ### Configure the ingestor Copy the example config and fill in your endpoint: ```bash cp superbank.example.yaml superbank.yaml ``` For live ingestion via Dragon's Mouth (gRPC): ```yaml source: "grpc" endpoint: "https://.mainnet.rpcpool.com:443" x-token: "" commitment: "finalized" # "*" = resume from the highest slot already in blocks_metadata on restart (recommended for production). # 0 = start from the earliest available slot. dragonsmouth-from-slot: "*" clickhouse-url: "http://localhost:8123" clickhouse-database: "default" clickhouse-user: "default" clickhouse-password: "" transactions-table: "default.transactions" blocks-table: "default.blocks_metadata" entries-table: "default.entries" ``` For live ingestion via **Fumarole** (persistent consumer group, resumes from where it left off): ```yaml source: "fumarole" fumarole-endpoint: "https://ams.rpcpool.com:443" # regional endpoint; or https://nyc.rpcpool.com:443 fumarole-x-token: "" fumarole-consumer-group: "superbank-mainnet" fumarole-create-consumer-group: true # set to false after first run fumarole-data-plane-tcp-connections: 4 # parallel TCP connections for download; max 20 # fumarole-memory-soft-limit-bytes: 25769803776 # 24 GiB soft limit; set 0 to disable clickhouse-url: "http://localhost:8123" clickhouse-database: "default" clickhouse-user: "default" clickhouse-password: "" transactions-table: "default.transactions" blocks-table: "default.blocks_metadata" entries-table: "default.entries" ``` On first run you'll see the consumer group created, then data flowing: ``` INFO superbank::ingest::fumarole: starting superbank fumarole ingest source="fumarole" endpoint=https://ams.rpcpool.com:443 consumer_group=superbank-mainnet INFO superbank::ingest::fumarole: created Fumarole consumer group consumer_group=superbank-mainnet INFO superbank::clickhouse: clickhouse insert committed table="default.transactions" rows=4051 slot_min=424497433 slot_max=424497435 INFO superbank::clickhouse: clickhouse insert committed table="default.blocks_metadata" rows=3 slot_min=424497433 slot_max=424497435 INFO superbank::clickhouse: clickhouse insert committed table="default.entries" rows=3412 slot_min=424497433 slot_max=424497435 ``` After the first run, set `fumarole-create-consumer-group: false`. The group persists on the server and resumes from its last committed position on restart. Fumarole is generally preferred for production: it uses a persistent consumer group that survives restarts, has at-least-once delivery guarantees, and also populates the `entries` table like the gRPC source. Dragon's Mouth gRPC is simpler for local dev. For **historical backfill**, there are two approaches depending on scale. **Large-scale backfill: Jetstreamer + Old Faithful** For ingesting months or years of history, use the Jetstreamer adapter. By default it reads the public [Old Faithful](/solana/historical-data.md) archive at `https://files.old-faithful.net`, with no Triton endpoint or token involved. Old Faithful has full history back to genesis and serves data at wire speed, far faster than polling `getBlock` over HTTP. Bound the range with the epoch or slot arguments, and set the trusted bound and commission era shown in the run commands below; see the [Filtering section](#filtering-to-your-own-transactions) for trade-offs if you also want a program filter. As of Superbank v0.8.0, the adapter uses Agave 4 and records transaction-v1 fields. It backfills only up to the Alpenglow genesis slot, because Jetstreamer doesn't expose the bank and footer data that later blocks need. On mainnet, `getAgGenesisCert` puts that slot at 459,652,999, and the runner clamps a longer range to it. Use the ingestor's `rpc`, Bigtable or gRPC source for later slots; the `rpc` and Bigtable sources also handle transaction-v1 history. The adapter is its own Cargo workspace, outside Superbank's main one, so validate it on a short slot range before a large backfill. **Small-to-medium backfill: JSON-RPC source** For shorter ranges (e.g. a few thousand slots), the `rpc` source works well: ```yaml source: "rpc" rpc-url: "https://.mainnet.rpcpool.com/" rpc-from-slot: 424000000 rpc-slot-count: 10000 # or use rpc-to-slot for an explicit end slot rpc-max-inflight: 16 clickhouse-url: "http://localhost:8123" # ... rest same as above ``` Use a Triton HTTP RPC endpoint (available from your [customers.triton.one](https://customers.triton.one) dashboard). Dragon's Mouth is gRPC, so the `rpc` source needs your regular HTTP JSON-RPC endpoint. The `rpc` source calls `getBlock`, which Triton's shared endpoints rate-limit per method (see [Rate and connection limits](/using-your-endpoint/rate-and-connection-limits.md)), so lower `rpc-max-inflight` (for example to 16) to reduce HTTP 429 errors, or ask support for a higher limit. Triton's endpoints also serve `getBlock` for older slots automatically from their history backends. The public `api.mainnet-beta.solana.com` endpoint rate-limits `getBlock` aggressively; avoid it for any bulk backfill. #### ClickHouse insert retry For gRPC, RPC, and Bigtable sources, the ingestor retries failed ClickHouse inserts with exponential backoff. Fumarole is excluded: it manages durability at the consumer-group level. | Config key | Env var | Default | Description | | ---------------------- | --------------------------------- | ------- | -------------------------------------- | | `insert-max-retries` | `CLICKHOUSE_INSERT_MAX_RETRIES` | `5` | Maximum retry attempts per batch | | `insert-retry-base-ms` | `CLICKHOUSE_INSERT_RETRY_BASE_MS` | `1000` | Initial retry delay (ms) | | `insert-retry-max-ms` | `CLICKHOUSE_INSERT_RETRY_MAX_MS` | `30000` | Maximum retry delay after backoff (ms) | | {% endstep %} | | | | {% step %} ### Start the ingestor ```bash ./target/release/superbank --config superbank.yaml ``` You'll see log lines as blocks are ingested. The ingestor writes to `transactions`, `blocks_metadata`, and `entries`. ClickHouse materializes the views (`gsfa`, `signatures`, `token_owner_activity`) in the background as rows arrive. Example output (RPC backfill source): ``` INFO superbank: starting name="superbank" version="0.8.0" INFO superbank::ingest::rpc: starting superbank ingest source="rpc" from_slot=424274200 to_slot=424274209 INFO superbank::clickhouse: clickhouse insert committed table="default.transactions" rows=1582 slot=424274200 progress_percent=100.0 INFO superbank::clickhouse: clickhouse insert committed table="default.blocks_metadata" rows=1 slot=424274200 ``` Example output (gRPC / Dragon's Mouth live source): ``` INFO superbank: starting name="superbank" version="0.8.0" INFO superbank::ingest::grpc: starting superbank ingest source="grpc" endpoint=https://.mainnet.rpcpool.com:443 INFO superbank::ingest::grpc: gRPC health check passed status="serving" INFO superbank::ingest::grpc: subscribed to gRPC stream from_slot=424432530 INFO superbank::clickhouse: clickhouse insert committed table="default.transactions" rows=2387 slot_min=424432637 slot_max=424432638 INFO superbank::clickhouse: clickhouse insert committed table="default.entries" rows=1221 slot_min=424432637 slot_max=424432638 ``` The gRPC source also populates the `entries` table (PoH entry metadata); the RPC backfill source does not. Check that data is landing: ```bash docker exec superbank-clickhouse clickhouse-client --query "SELECT count() FROM default.transactions" # → 13950 docker exec superbank-clickhouse clickhouse-client --query "SELECT max(slot) FROM default.transactions" # → 424274209 ``` {% endstep %} {% step %} ### Start the RPC server ```bash CLICKHOUSE_URL=http://localhost:8123 \ CLICKHOUSE_DATABASE=default \ CLICKHOUSE_USER=default \ CLICKHOUSE_PASSWORD="" \ CLICKHOUSE_CLUSTER="" \ CLICKHOUSE_GSFA_TABLE=default.gsfa \ CLICKHOUSE_SIGNATURE_STATUSES_TABLE=default.signatures \ CLICKHOUSE_TOKEN_OWNER_ACTIVITY_TABLE=default.token_owner_activity \ CLICKHOUSE_TRANSACTION_TABLE=default.transactions \ CLICKHOUSE_BLOCKS_METADATA_TABLE=default.blocks_metadata \ ./target/release/superbank-rpc ``` `CLICKHOUSE_CLUSTER=""` marks the ClickHouse server as a single node with no cluster. Without it, superbank-rpc resolves the default `{cluster}` macro at startup, and startup fails on a single-node server. For a ClickHouse cluster, keep the default or set the cluster name. By default, all JSON-RPC responses use HTTP `200 OK`, including error bodies. To return HTTP `503 Service Unavailable` on server-side failures (useful for load balancers that route on HTTP status), add `--emit-http-errors` or set `SUPERBANK_RPC_EMIT_HTTP_ERRORS=true`. Only internal errors (`-32603`), timeouts (`-32000`), node unhealthy (`-32005`), and storage unreachable (`-32019`) are promoted to HTTP 503; client and data-condition errors remain 200 OK. Test it: ```bash curl -s http://localhost:8899 \ -X POST -H "Content-Type: application/json" \ -d '{"jsonrpc":"2.0","id":1,"method":"getHealth"}' | jq . # → {"jsonrpc":"2.0","result":"ok","id":1} curl -s http://localhost:8899 \ -X POST -H "Content-Type: application/json" \ -d '{"jsonrpc":"2.0","id":1,"method":"getFirstAvailableBlock"}' | jq . # → {"jsonrpc":"2.0","result":424274200,"id":1} curl -s http://localhost:8899 \ -X POST -H "Content-Type: application/json" \ -d '{"jsonrpc":"2.0","id":1,"method":"getTransaction","params":["",{"encoding":"json","maxSupportedTransactionVersion":1}]}' | jq '{slot: .result.slot, blockTime: .result.blockTime, fee: .result.meta.fee}' # → {"slot": 424274200, "blockTime": 1780588286, "fee": 5000} curl -s http://localhost:8899 \ -X POST -H "Content-Type: application/json" \ -d '{"jsonrpc":"2.0","id":1,"method":"getBlock","params":[424274200,{"encoding":"json","maxSupportedTransactionVersion":1,"transactionDetails":"signatures"}]}' | jq '{blockhash: .result.blockhash, blockHeight: .result.blockHeight, numSignatures: (.result.signatures | length)}' # → {"blockhash": "Hpk2nu2vYpfMVNuW363vtc1JgMWkuCkgkjWVYyKfU14A", "blockHeight": 402357051, "numSignatures": 1582} # Triton extension - paginated transaction history with full tx data curl -s http://localhost:8899 \ -X POST -H "Content-Type: application/json" \ -d '{"jsonrpc":"2.0","id":1,"method":"getTransactionsForAddress","params":["TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA",{"transactionDetails":"full","limit":3,"maxSupportedTransactionVersion":1}]}' | jq '{count: (.result.data | length), firstSlot: .result.data[0].slot}' # → {"count": 3, "firstSlot": 424497440} ``` Pass `maxSupportedTransactionVersion` with full transaction results from `getTransactionsForAddress`. Set it to `1` to accept legacy, v0 and transaction-v1 results; each result then includes a `version` field (`"legacy"`, `0`, or `1`). Without it, legacy results omit `version`, and the request fails with error `-32015` if any result is a v0 or v1 transaction. Your Superbank instance is now serving Solana-compatible RPC at `:8899`. {% endstep %} {% endstepper %} ## Filtering to your own transactions By default, Superbank ingests every transaction on the network. If you want a targeted history, say, only transactions involving your program, there are two approaches with very different trade-offs. ### Recommended: filter by time range, ingest everything The safest approach is to pick the time window you care about and ingest every transaction in it, without any program filter. This keeps your derived tables (`gsfa`, `token_owner_activity`) and synthesized methods (`getBlock`, `getTransactionsForAddress`, `getSignaturesForAddress`) fully accurate: they depend on having the complete transaction set for every block in the indexed range. Use the `rpc-from-slot` / `rpc-to-slot` parameters (or the Jetstreamer epoch argument) to bound the range: ```yaml # superbank.yaml - ingest a specific slot range, no program filter source: "rpc" rpc-url: "https://.mainnet.rpcpool.com/" rpc-from-slot: 420000000 rpc-to-slot: 425000000 rpc-max-inflight: 16 clickhouse-url: "http://localhost:8123" ``` This is the approach to use for any production instance where downstream consumers will call gSFA, gTFA, or `getBlock`. ### Advanced: filter by program (breaks derived methods) {% hint style="warning" %} Filtering by program produces an incomplete transaction set. The following RPC methods will return incorrect or empty results: * `getSignaturesForAddress`: misses signatures for addresses that appeared in filtered-out transactions * `getTransactionsForAddress`: same * `getBlock`: blocks will appear to have fewer transactions than they actually did Use this only if you are building a purpose-specific index where you control the query surface and explicitly do not need those methods. {% endhint %} The Jetstreamer adapter is in `ingest/jetstreamer-clickhouse-plugin/src/lib.rs`. The filter is a 1-2 line addition to the `on_transaction` method. **Before (current `on_transaction`):** ```rust fn on_transaction<'a>( &'a self, thread_id: usize, db: Option>, transaction: &'a TransactionData, ) -> PluginFuture<'a> { async move { let Some(db) = self.resolve_db(db) else { return Ok(()); }; self.ensure_legacy_slot(transaction.slot)?; let row = TransactionRow::from_transaction(transaction); // ... buffers the row for this slot ``` **After: add a program filter at the top of the async block.** The plugin already has a `parse_pubkey_bytes` helper that decodes a base58 address to `[u8; 32]`. Use that to avoid adding a new dependency: ```rust // Add near the bottom of lib.rs (outside the impl block, before parse_pubkey_bytes): fn my_program_id() -> [u8; 32] { parse_pubkey_bytes("YourProgramPubkeyHere") .expect("valid program pubkey") } // In on_transaction, add three lines at the top of the async block: fn on_transaction<'a>( &'a self, thread_id: usize, db: Option>, transaction: &'a TransactionData, ) -> PluginFuture<'a> { async move { // Filter: only ingest transactions that involve our program let program = my_program_id(); if !transaction.transaction.message.static_account_keys().iter().any(|k| k.to_bytes() == program) { return Ok(()); // skip this transaction } let Some(db) = self.resolve_db(db) else { return Ok(()); }; // ... rest of the function unchanged ``` The `static_account_keys()` slice contains every static account referenced in the transaction: fee payer, program IDs, and all account operands. If your program ID appears anywhere, the transaction passes the filter. **To filter by multiple programs** (e.g. your program and a token program it interacts with): ```rust fn is_relevant(transaction: &TransactionData) -> bool { const PROGRAMS: &[&str] = &[ "YourProgramPubkeyHere", "TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA", ]; let keys = transaction.transaction.message.static_account_keys(); PROGRAMS.iter().any(|p| { if let Some(bytes) = parse_pubkey_bytes(p) { keys.iter().any(|k| k.to_bytes() == bytes) } else { false } }) } // In on_transaction: if !is_relevant(transaction) { return Ok(()); } ``` After adding the filter, rebuild and run: ```bash cd ingest/jetstreamer-clickhouse-plugin # Rebuild cargo build --release # Required: exactly one trusted historical bound, the slot from getAgGenesisCert export JETSTREAMER_ALPENGLOW_GENESIS_SLOT= # or, on a cluster where getAgGenesisCert is still null: JETSTREAMER_PREACTIVATION_THROUGH_SLOT= # Required: exactly one block reward commission era export JETSTREAMER_BLOCK_REWARD_COMMISSION_BPS_FROM_SLOT= # or, for a range entirely before SIMD-0291: JETSTREAMER_BLOCK_REWARD_COMMISSION_PERCENT=true # Run for a specific epoch ./target/release/jetstreamer-clickhouse 800 # Or for a specific slot range ./target/release/jetstreamer-clickhouse 424000000:424100000 ``` The runner exits at startup if either setting is missing or ambiguous. On mainnet, the SIMD-0291 feature account (`Eg7tXEwMZzS98xaZ1YHUbdRHsaYZiCsSaR6sKgxreoaj`) records activation at slot 433,296,000. See the [plugin README](https://github.com/solana-rpc/superbank/blob/v0.8.0/ingest/jetstreamer-clickhouse-plugin/README.md) for the evidence each setting needs. ## Optional: processed commitment with the gRPC head cache By default, the RPC server supports `confirmed` and `finalized` commitment. To also serve `processed` commitment (latest tip, sub-slot latency), enable the head cache: ```bash # Build with the feature enabled cargo build --release -p superbank-rpc --features grpc-head-cache ``` Then start the RPC server with the head cache pointed at a Dragon's Mouth endpoint, alongside the env vars from the RPC server step: ```bash HEAD_CACHE_ENABLED=true \ DRAGONSMOUTH_ENDPOINT=https://.mainnet.rpcpool.com:443 \ DRAGONSMOUTH_X_TOKEN= \ HEAD_CACHE_RETAIN_SLOTS=32 \ HEAD_CACHE_MIN_COMMITMENT=processed \ CLICKHOUSE_URL=http://localhost:8123 \ CLICKHOUSE_DATABASE=default \ CLICKHOUSE_USER=default \ CLICKHOUSE_PASSWORD="" \ CLICKHOUSE_CLUSTER="" \ CLICKHOUSE_GSFA_TABLE=default.gsfa \ CLICKHOUSE_SIGNATURE_STATUSES_TABLE=default.signatures \ CLICKHOUSE_TOKEN_OWNER_ACTIVITY_TABLE=default.token_owner_activity \ CLICKHOUSE_TRANSACTION_TABLE=default.transactions \ CLICKHOUSE_BLOCKS_METADATA_TABLE=default.blocks_metadata \ ./target/release/superbank-rpc ``` On startup you'll see the head cache subscribe to Dragon's Mouth: ``` INFO superbank_rpc: starting name="superbank-rpc" version="0.8.0" INFO superbank_rpc::server: RPC server listening on http://0.0.0.0:8899 INFO superbank_rpc::head_cache::dragonsmouth: head cache: subscribed to DragonsMouth endpoint="https://.mainnet.rpcpool.com:443" min_commitment=Processed ``` Test all three commitment levels: ```bash curl -s http://localhost:8899 \ -X POST -H "Content-Type: application/json" \ -d '{"jsonrpc":"2.0","id":1,"method":"getSlot","params":[{"commitment":"processed"}]}' | jq .result # → 424467184 (latest tip) curl -s http://localhost:8899 \ -X POST -H "Content-Type: application/json" \ -d '{"jsonrpc":"2.0","id":1,"method":"getSlot","params":[{"commitment":"confirmed"}]}' | jq .result # → 424467183 (1 slot behind processed) curl -s http://localhost:8899 \ -X POST -H "Content-Type: application/json" \ -d '{"jsonrpc":"2.0","id":1,"method":"getSlot","params":[{"commitment":"finalized"}]}' | jq .result # → 424432638 (from ClickHouse) ``` The cache holds the last `HEAD_CACHE_RETAIN_SLOTS` slots in memory and merges them with ClickHouse results for `getTransaction` and `getSignatureStatuses` queries, plus `getSignaturesForAddress`, `getTransactionsForAddress`, `getBlockHeight`, `getSlot`, `getTransactionCount`, `getLatestBlockhash`, `isBlockhashValid`, `getBlocks`, and `getBlocksWithLimit`. These 11 methods accept `processed`. `getBlock` and `getBlockTime` also read recent blocks from the cache, but `getBlock` still requires `confirmed` or `finalized`. {% hint style="info" %} **RAM requirement.** The head cache itself is lightweight: at the default of 32 slots, it holds \~32 × 1,500 mainnet transactions worth of metadata in memory, typically under 500 MB. The dominant memory cost is ClickHouse: for a full mainnet history, plan for at least 32 GB RAM on the ClickHouse host (64 GB+ for comfortable production operation). The `superbank` and `superbank-rpc` binaries themselves each use well under 2 GB. {% endhint %} {% hint style="info" %} The `grpc-head-cache` feature opens a separate live Dragon's Mouth subscription. This is why it's an opt-in compile feature rather than the default. {% endhint %} ## Optional: gRPC streaming (`grpc-streaming`) When compiled with `--features grpc-streaming` and enabled at runtime, the RPC server exposes a tonic gRPC endpoint alongside JSON-RPC. The current implementation supports historical `StreamBlocks` and `StreamTransactions` over bounded inclusive slot ranges from ClickHouse. `StreamBlocks` returns one message per matching block with block metadata, rewards, and transaction payloads. `StreamTransactions` returns one message per matching transaction. `StreamBlocks` filters only on `account_include`. `StreamTransactions` supports account include/exclude/required filters, vote filtering, and failed/success filtering. Build with the feature: ```bash cargo build --release -p superbank-rpc --features grpc-streaming ``` Then start with gRPC enabled (add alongside the env vars from the RPC server step): ```bash SUPERBANK_GRPC_ENABLED=true \ SUPERBANK_GRPC_PORT=10000 \ ./target/release/superbank-rpc ``` Configuration: | Flag | Env var | Default | Description | | ----------------------------------------- | --------------------------------------- | ----------- | ----------------------------------------------- | | `--superbank-grpc-enabled` | `SUPERBANK_GRPC_ENABLED` | `false` | Enable gRPC endpoint at runtime | | `--superbank-grpc-host` | `SUPERBANK_GRPC_HOST` | `0.0.0.0` | Bind host | | `--superbank-grpc-port` | `SUPERBANK_GRPC_PORT` | `10000` | Bind port | | `--superbank-grpc-max-slot-range` | `SUPERBANK_GRPC_MAX_SLOT_RANGE` | `100` | Maximum inclusive slots per stream request | | `--superbank-grpc-query-timeout-ms` | `SUPERBANK_GRPC_QUERY_TIMEOUT_MS` | `30000` | Per-chunk ClickHouse range-query timeout (ms) | | `--superbank-grpc-chunk-slots` | `SUPERBANK_GRPC_CHUNK_SLOTS` | `8` | Slots fetched from ClickHouse per chunk | | `--superbank-grpc-max-send-bytes` | `SUPERBANK_GRPC_MAX_SEND_BYTES` | `104857600` | Max encoded gRPC response message size (100 MB) | | `--superbank-grpc-max-concurrent-streams` | `SUPERBANK_GRPC_MAX_CONCURRENT_STREAMS` | `20` | HTTP/2 concurrent stream limit | ## Optional: disk cache (`disk-cache`) When compiled with `--features disk-cache` and enabled at runtime, superbank-rpc forwards recent **finalized** slots from the source ClickHouse cluster to a separate ClickHouse instance on localhost. The feature is independent of `grpc-head-cache`. The read tiering becomes: ``` head cache (optional, unfinalized tip) → local ClickHouse cache (finalized, recent slots) → source ClickHouse (full history) ``` At startup, superbank-rpc copies the required schema from the source into a local database that it owns exclusively, then streams `transactions` and `blocks_metadata` into it. Local materialized views build the query indexes, and coverage becomes readable only after it validates. Any miss, hole, local query error, or unavailable local instance falls through to the source cluster. Methods served from the local cache when covered: `getBlock`, `getBlocks`, `getBlocksWithLimit`, `getBlockTime`, `getTransaction`, `getSignatureStatuses`, `getSignaturesForAddress`, and `getTransactionsForAddress` (including `tokenAccounts` filters). {% hint style="warning" %} `DISK_CACHE_RETAIN_SLOTS` is required and has no default. `DISK_CACHE_MAX_BYTES` caps the cache database by evicting complete old partitions. A budget that's too small can purge the newest partition and leave the cache unready. {% endhint %} The source ClickHouse user needs read access to the copied tables and their metadata. The local ClickHouse user needs permission to create and drop the dedicated database, create and alter its tables, and read and insert its data. The local endpoint must use `localhost` or a loopback IP; it can use a different port when the source ClickHouse runs on the same host. ```bash # Build with the independent disk-cache feature cargo build --release -p superbank-rpc --features disk-cache # Run with a source cluster and a separate local ClickHouse HTTP endpoint RPC_HOST=0.0.0.0 RPC_PORT=8899 \ CLICKHOUSE_URL=http://source-clickhouse:8123 CLICKHOUSE_DATABASE=default \ DISK_CACHE_ENABLED=true DISK_CACHE_RETAIN_SLOTS=432000 \ DISK_CACHE_CLICKHOUSE_URL=http://127.0.0.1:8123 \ DISK_CACHE_CLICKHOUSE_DATABASE=superbank_disk_cache \ ./target/release/superbank-rpc ``` Key configuration options: | Option | Environment | Default | Notes | | ---------------------------------- | -------------------------------- | ----------------------- | -------------------------------------------------------------------- | | `--disk-cache-enabled` | `DISK_CACHE_ENABLED` | `false` | Enable at runtime | | `--disk-cache-clickhouse-url` | `DISK_CACHE_CLICKHOUSE_URL` | | Local ClickHouse HTTP endpoint; loopback only | | `--disk-cache-retain-slots` | `DISK_CACHE_RETAIN_SLOTS` | `(none)` | Finalized slots to retain; required when enabled | | `--disk-cache-max-bytes` | `DISK_CACHE_MAX_BYTES` | `0` | Byte budget for the cache database; 0 = unlimited | | `--disk-cache-clickhouse-database` | `DISK_CACHE_CLICKHOUSE_DATABASE` | `superbank_disk_cache` | Dedicated database owned by the cache | | `--disk-cache-required` | `DISK_CACHE_REQUIRED` | `false` | Require local initialization at startup and local health for /health | | `--disk-cache-backfill-enabled` | `DISK_CACHE_BACKFILL_ENABLED` | `true` | Source-to-local forward and repair task | Observability: `superbank_disk_cache_*` metrics cover readiness, local ClickHouse bytes, coverage, read outcomes, forwarding, errors, rebuilds, and partition eviction. The `X-Superbank-Sources` response header reports `disk-cache` when the local tier contributes. Parity validation and performance comparison scripts are in `tests/k6/scenarios/validation/superbank-rpc-disk-cache-parity.js` and `tests/k6/scenarios/performance/superbank-rpc-disk-cache-compare.js`. {% hint style="info" %} The RocksDB settings `DISK_CACHE_PATH`, `DISK_CACHE_BLOCK_CACHE_BYTES`, `DISK_CACHE_WRITE_QUEUE_SLOTS`, and `DISK_CACHE_READ_CONCURRENCY` were removed; setting any of them now fails with a configuration error. By default, local initialization is fail-open: the RPC server starts against the source cluster and retries the local cache in the background. Set `DISK_CACHE_REQUIRED=true` to make initialization and health strict. {% endhint %} ## Going to production ### Use replicated DDL for HA For a production multi-node setup, use `ddl/replicated/` instead of `ddl/local/`. This requires ClickHouse Keeper or ZooKeeper and uses `ReplicatedReplacingMergeTree`. The DDL commands follow the same pattern as the local setup: ```bash cat ddl/replicated/transactions.sql | docker exec -i superbank-clickhouse clickhouse-client --multiquery cat ddl/replicated/blocks_metadata.sql | docker exec -i superbank-clickhouse clickhouse-client --multiquery cat ddl/replicated/gsfa.sql | docker exec -i superbank-clickhouse clickhouse-client --multiquery # ... etc for other tables ``` Note: the replicated tables will fail to create if ClickHouse Keeper or ZooKeeper is not configured and reachable. Set up your Keeper cluster before applying this DDL. The `ddl/local/` variant used in this guide does not have this requirement. ### Use Fumarole as the ingestion source Fumarole is the recommended source for production: * Persistent consumer groups survive ingestor restarts * At-least-once delivery with commit checkpoints * Lower reconnect risk than raw gRPC streaming Set `source: fumarole` in your YAML and configure `fumarole-consumer-group`. After the first run, set `fumarole-create-consumer-group: false`. ### Metrics Both binaries expose Prometheus metrics at `/metrics`: * Ingestor: `http://:9901/metrics` * RPC server: `http://:9900/metrics` **Key RPC server metrics:** | Metric | What it tells you | | --------------------------------------------------- | ---------------------------------------------------------------------------- | | `superbank_rpc_requests_total{method,status}` | Request count per method: track error rate by filtering `status!="200"` | | `superbank_rpc_response_time_seconds{method}` | Full end-to-end latency histogram: use for p50/p95/p99 | | `superbank_rpc_clickhouse_duration_seconds{method}` | Time spent in ClickHouse per method: isolates DB vs overhead | | `superbank_rpc_inflight_requests{method}` | Concurrent in-flight requests: spike indicates backpressure | | `superbank_rpc_route_total_total{method,outcome}` | Routing outcomes: watch for `outcome="backend_error"` or `outcome="timeout"` | | `superbank_head_cache_active` | `1` = head cache connected to Dragon's Mouth; `0` = disconnected | | `superbank_head_cache_latest_slot` | Most recent slot in cache: compare to chain tip to measure lag | | `superbank_process_resident_memory_bytes` | Resident memory used by the RPC process | **Key ingestor metrics (`:9901`):** The ingestor tracks insert throughput, batch sizes, and ClickHouse write latency. Watch for any sustained gap between the max ingested slot and the chain tip: this indicates the ingestor is falling behind. The ingestor also exposes a liveness health check at `http://:9901/health`. It returns `200 OK` while flushing normally, and `503 Service Unavailable` when no successful ClickHouse flush has occurred within `health-stale-secs` seconds (default: 120). Set `health-stale-secs: 0` in your config to disable the staleness check. Add `METRICS_CLUSTER_LABEL` to attach a static `cluster="..."` label to all ingestor metrics, useful for multi-cluster Prometheus setups. When running the Fumarole source, the following additional gauges and counters are emitted: | Metric | Description | | -------------------------------------------------------- | ------------------------------------------------------------------------- | | `superbank_ingest_fumarole_memory_soft_limit_bytes` | Configured memory soft limit | | `superbank_ingest_fumarole_buffered_bytes` | Estimated assembler bytes in memory | | `superbank_ingest_fumarole_pending_slots` | Slots waiting to flush | | `superbank_ingest_fumarole_rss_bytes` | Process RSS sampled by the memory guard | | `superbank_ingest_fumarole_pressure_flushes_total_total` | Backpressure-triggered flushes (rising counter indicates memory pressure) | **Example Prometheus queries:** ```promql # p99 RPC latency per method histogram_quantile(0.99, sum by (le, method) (rate(superbank_rpc_response_time_seconds_bucket[5m]))) # Error rate across all methods sum(rate(superbank_rpc_requests_total{status!="200"}[5m])) / sum(rate(superbank_rpc_requests_total[5m])) # ClickHouse query time p95 histogram_quantile(0.95, sum by (le, method) (rate(superbank_rpc_clickhouse_duration_seconds_bucket[5m]))) ``` ## What's next * **Kubernetes**: `deploy/k8s/` has manifests for local development with Tilt and Kind, covering ClickHouse, the DDL init job, superbank-rpc, and both ingestion modes. * **Load testing**: `tests/k6/` has k6 scenarios for all RPC methods, including validation tests that compare your Superbank instance against a reference RPC. * **Triton hosted**: if you'd rather not operate ClickHouse and the ingestor yourself, Triton runs Superbank as a managed service. Same RPC interface, no infra. *** :life-ring: Contact support by clicking the chat icon in your [customer dashboard](https://customers.triton.one)\ :briefcase: Sales questions? [Contact us](https://triton.one/contact)\ :sparkles: AI agent? Read [llms.txt](https://docs.triton.one/llms.txt)\ :rss: Follow updates: [Blog](https://blog.triton.one) · [X](https://x.com/triton_one) · [YouTube](https://www.youtube.com/@triton_one_ltd) · [Telegram](https://t.me/tritonone) · [GitHub](https://github.com/rpcpool)