Cluster Setup#

RELATA_PROFILE=cluster is alpha (see Deployment — the same caveat applies here: petabyte / very-high-cardinality sharding still needs more work). This page walks through what it actually takes to get multiple cluster-profile nodes talking to each other outside of Kubernetes — local dev, bare metal, or just understanding what the Helm chart is doing under the hood.

Every requirement below was verified against a real 3-node boot (not just read off source) as of this writing. Cluster is under active development — re-check against crates/relata-cli/src/serve.rs before relying on exact error strings in a script.

Topology & roles#

A cluster node advertises a CLUSTER_ROLE: coordinator | reader | writer | indexer (default coordinator; an unrecognized value fails startup).

Important nuance: CLUSTER_ROLE is a routing/registry hint, not a hard access gate. Every node — regardless of role — still exposes the full HTTP surface (/query, /ingest, /health, etc.). The role tells the cluster registry how to characterize a node for cross-node routing decisions (e.g. nodes_of_role(Reader)); it does not, by itself, make a reader node reject writes or a writer node reject reads.

Don't confuse CLUSTER_ROLE with the separate RELATA_ROLE env var (query | indexer | both, default both) — that one controls where deferred secondary/vector index-maintenance work is applied, and is independent of cluster fan-out routing entirely. If you only set one, set CLUSTER_ROLE.

Data is distributed across nodes by consistent hashing (RELATA_CLUSTER_SHARDS, default 8 shards) — every node must derive the same partition key, which is what RELATA_CLUSTER_SEED (below) is for.

Connecting clients to a cluster — one address, not a seed list#

Relata is smart-server, dumb-client (the opposite of MongoDB's architecture). The cluster does fan-out internally — you point any client at one stable address in front of the cluster, the coordinator routes to peers over gRPC (QueryShard), and results are merged. Your client never learns about the other nodes.

You do NOT use mongodb+srv:// — that's a MongoDB Atlas DNS-SRV seed-discovery mechanism, and Relata doesn't implement it (or replica-set hello/isMaster handshake). You don't need it: standard drivers connect to a single host just fine, and a load balancer gives you HA/failover without driver-side seed discovery.

                ┌────────────────────────┐
your client  ──►  LB / VIP / K8s Service  ──►  coordinator node
                └────────────────────────┘            │
                                                      └─► gRPC fan-out to peers (internal)

// MongoDB — any official driver, single host. password = RELATA_BEARER_TOKEN
const c = new MongoClient("mongodb://relata-vip:27017", {
  auth: { username: "relata", password: "<token>" },
});

# Postgres / pgvector / Redis / Neo4j / ClickHouse / S3 — same pattern, one VIP
psql -h relata-vip -p 5433 -U relata              # password = token
redis-cli -h relata-vip -p 6379 -a <token>
cypher-shell -a bolt://relata-vip:7687 -u neo4j -p <token>

Failover: the LB health-checks GET /health/ready (HTTP 9090) and drops any node returning non-200. Every node can serve reads; writes funnel through the planner regardless of which node received them. No driver change, no +srv, no replica-set name.

One constraint: the compat doors default to the same port on every node. Running multiple nodes on one host causes bind collisions (non-fatal WARN, that node loses the door). One node per host/container/pod is the supported topology — which is exactly what an LB in front assumes. See Deploying Protocol Doors and Compatibility & Doors.

Required env vars#

Every cluster-profile node needs all of the following. Names and defaults are current as of this writing (crates/relata-cli/src/serve.rs, serve/cluster.rs, serve/admin_listener.rs, relata-cluster/src/scatter_gather.rs).

A tested local 3-node example#

This exact recipe was booted end-to-end while writing this page: all three nodes reported 200 on /health, and a write via /ingest on the writer replicated to the other two nodes over HTTP with no peer-replication errors in any node's log.

BIN=/path/to/relata   # or `cargo run -p relata-cli --release -- serve`
 
COMMON=(
  RELATA_PROFILE=cluster
  RELATA_CLUSTER_SEED=my-local-cluster-seed   # any string — identical on all 3
  RELATA_KMS_LOCAL_DEV=true                   # local/dev ONLY — never in production
  CLUSTER_AUTH_TOKEN=shared-cluster-secret    # identical on all 3
  RELATA_BEARER_TOKEN=my-client-token         # identical on all 3 (client-facing)
)
 
# node0 — coordinator
env "${COMMON[@]}" \
  NODE_ID=node0 CLUSTER_ROLE=coordinator \
  CLUSTER_PEERS=http://127.0.0.1:29081,http://127.0.0.1:29082 \
  RELATA_PUBLIC_URL=http://127.0.0.1:29080 \
  RELATA_HTTP_BIND=127.0.0.1:29080 RELATA_PORT=29080 \
  RELATA_GRPC_BIND=127.0.0.1 RELATA_GRPC_PORT=29180 \
  RELATA_ADMIN_BIND=127.0.0.1:29280 RELATA_PG_PORT=29380 \
  RELATA_DATA_DIR=/tmp/relata-cluster/node0 \
  "$BIN" serve &
 
# node1 — reader
env "${COMMON[@]}" \
  NODE_ID=node1 CLUSTER_ROLE=reader \
  CLUSTER_PEERS=http://127.0.0.1:29080,http://127.0.0.1:29082 \
  RELATA_PUBLIC_URL=http://127.0.0.1:29081 \
  RELATA_HTTP_BIND=127.0.0.1:29081 RELATA_PORT=29081 \
  RELATA_GRPC_BIND=127.0.0.1 RELATA_GRPC_PORT=29181 \
  RELATA_ADMIN_BIND=127.0.0.1:29281 RELATA_PG_PORT=29381 \
  RELATA_DATA_DIR=/tmp/relata-cluster/node1 \
  "$BIN" serve &
 
# node2 — writer
env "${COMMON[@]}" \
  NODE_ID=node2 CLUSTER_ROLE=writer \
  CLUSTER_PEERS=http://127.0.0.1:29080,http://127.0.0.1:29081 \
  RELATA_PUBLIC_URL=http://127.0.0.1:29082 \
  RELATA_HTTP_BIND=127.0.0.1:29082 RELATA_PORT=29082 \
  RELATA_GRPC_BIND=127.0.0.1 RELATA_GRPC_PORT=29182 \
  RELATA_ADMIN_BIND=127.0.0.1:29282 RELATA_PG_PORT=29382 \
  RELATA_DATA_DIR=/tmp/relata-cluster/node2 \
  "$BIN" serve &
 
wait

Verify boot and write path:

curl http://127.0.0.1:29080/health   # -> 200, all three ports
curl http://127.0.0.1:29081/health
curl http://127.0.0.1:29082/health
 
# Writes go through /ingest, NOT /query — /query is read-only (a plain SQL
# INSERT against /query returns 400: "/query is read-only and does not
# accept INSERT; use POST /ingest?object_type=<Type>&purpose=<P>").
curl -X POST "http://127.0.0.1:29082/ingest?object_type=Person" \
  -H "Authorization: Bearer my-client-token" -H "Content-Type: application/json" \
  -d '{"id": "p-1", "name": "Example Person"}'
# -> 200 {"rows_ingested":1,...} with no peer-replication warnings in any log

This is enough to prove the topology boots cleanly and that HTTP-level peer replication (governed_upsert) works with no CLUSTER_AUTH_TOKEN/connectivity errors. It is not, by itself, enough for cross-node query fan-out (SELECT, including a plain primary-key lookup) to work — see the gRPC gotcha immediately below, which is a separate, deeper issue than peer replication.

Gotchas#

RELATA_GRPC_PORT must match across nodes, not be unique#

This is the one that costs the most time, and it's the opposite of the instinct that "every door needs a unique port to avoid a bind collision."

Cross-node query fan-out (relata-cluster/src/scatter_gather.rs) does not read each peer's real gRPC address. It rewrites the peer's HTTP URL from CLUSTER_PEERS by keeping the peer's host and swapping in this node's own RELATA_GRPC_PORT (rewrite_to_grpc_port()). This is correct and required in the normal production topology — Kubernetes/the Helm chart's StatefulSet gives every replica the same container port for gRPC (.Values.service.ports.grpc, RELATA_GRPC_PORT unset → default 50051 everywhere) and differentiates nodes purely by pod DNS name/IP. In that world, "peer host + my own gRPC port" always resolves correctly.

On a single machine, giving every node a unique RELATA_GRPC_PORT (to dodge the 127.0.0.1:50051 bind collision — this is exactly what the required-vars table above tells you to do to get the process to boot) breaks that assumption: node0's rewrite of a request meant for node1 targets 127.0.0.1:<node0's-own-gRPC-port> (its own gRPC port, not node1's real one). The symptom is not a boot failure — every node reports healthy — it's every cross-node query returning:

{"status":503,"title":"Service Unavailable","detail":"cluster fan-out failed: scatter-gather: all 2 peer dispatches failed"}

This was reproduced directly against the recipe above: /health is 200 on all three nodes and /ingest replicates cleanly, but a SELECT ... WHERE id = '...' against any node returns the 503 above — because data is hash-partitioned across the 3 shards/nodes, so even a single-row primary-key lookup usually isn't fully answerable from local state alone and needs a working peer gRPC connection. The logs show QueryShard peer connect failed targeting the wrong (self) gRPC port.

Fix: give every node the same RELATA_GRPC_PORT, and instead vary the host each node binds/is reached on:

• Real hosts / separate containers / Kubernetes (recommended, matches the Helm chart) — each node is a distinct IP or DNS name; leave RELATA_GRPC_PORT at the default (or any value) as long as it's identical everywhere.
• Single-machine simulation — give each node its own loopback alias IP with a shared gRPC port: sudo ifconfig lo0 alias 127.0.0.2 up (macOS repeated per extra address needed; harmless and local-only) or use distinct 127.x.x.x addresses directly on Linux (the whole 127.0.0.0/8 block routes to loopback there without extra config). Bind each node's RELATA_GRPC_BIND to its own alias and reference that same alias (not 127.0.0.1) in CLUSTER_PEERS and RELATA_PUBLIC_URL, keeping RELATA_GRPC_PORT identical across all three. This variant was not independently re-verified in a sandbox without root — it follows directly from rewrite_to_grpc_port()'s logic and the same host-per-node pattern the Helm chart uses, but confirm it in your own environment.
• Docker Compose — each service gets its own container IP on the compose network; publish distinct host ports for HTTP/admin/pgwire as needed, but leave the container-internal gRPC port identical across services (mirrors Kubernetes exactly).

If you only need to prove nodes boot and replicate writes over HTTP (as in the tested recipe above), unique gRPC ports per node are fine — just know that /query fan-out won't work until the gRPC ports line up.

Three different bind-var conventions#

Passing RELATA_GRPC_BIND=127.0.0.1:29180 (treating it like the other two) does not FATAL cleanly — it gets concatenated with RELATA_GRPC_PORT into something like 127.0.0.1:29180:50051, and the process dies with a raw Error: failed to bind <addr>: invalid socket address rather than a helpful message pointing at the actual mistake.

CLUSTER_AUTH_TOKEN fails silently, not loudly#

Unlike RELATA_CLUSTER_SEED, RELATA_PUBLIC_URL, and NODE_ID — all of which FATAL the process at startup if wrong — an unset CLUSTER_AUTH_TOKEN on a cluster-profile node does not stop that node from starting. It boots, reports 200 on /health, and looks completely healthy. Every peer that tries to replicate a write to it or run a QueryShard RPC against it gets rejected with a 503:

"cluster internal endpoints require RELATA_CLUSTER_AUTH_TOKEN to be configured"

Note the error text itself says RELATA_CLUSTER_AUTH_TOKEN — that env var doesn't exist; the real var the code reads is CLUSTER_AUTH_TOKEN (no RELATA_ prefix). Don't go looking for a var named RELATA_CLUSTER_AUTH_TOKEN — it's a naming slip in the error string, not a second real var.

Because this fails per-request rather than at boot, it's easy to bring up a cluster, watch every node's /health return 200, declare victory, and only discover the misconfiguration later when writes stop propagating or fan-out queries start 503ing. Double-check CLUSTER_AUTH_TOKEN is identical (not just "set") on every node before trusting a green /health.

If the tokens are set on both sides but simply mismatched (not empty), the failure mode is different again: a 401 Unauthorized, not a 503 — the request falls through to a bearer-token check instead.

Every other protocol door defaults to the same port on every node#

RELATA_PG_PORT (pgwire), and the S3/ClickHouse/Neo4j/Redis/MongoDB/Bolt compat doors, all default to a fixed port regardless of NODE_ID. Running 3 nodes on one machine means 2 of them lose those doors to a non-fatal bind failed — disabled WARN unless you give each node distinct values (RELATA_PG_PORT above) or explicitly disable the doors you don't need per-node. This doesn't break the core cluster (HTTP + gRPC fan-out), but it's worth knowing the WARNs in the log are expected and not something to chase.

At-rest encryption's FATAL trigger is broader than "using a remote store"#

RELATA_KMS_LOCAL_DEV=true (or a real RELATA_KMS_KEY_ARN) is required the moment cluster profile has any on-disk persistence — a plain local RELATA_DATA_DIR is enough to trigger the KMS FATAL at startup, not just an S3/remote object store. If you see RELATA_KMS_KEY_ARN is required in the 'cluster' profile; refusing to fall back to the committed dev secret, this is why.

Helm / Kubernetes vs. this guide#

The Helm chart (infra/helm/relata in the main repo) already encodes a correct version of most of this for you:

• The StatefulSet gives every replica the same container ports (so the gRPC-port gotcha above never comes up — pods are differentiated by DNS name, not port).
• CLUSTER_PEERS is generated automatically from the headless-service DNS names for all replicas.
• CLUSTER_AUTH_TOKEN defaults to the same secret as RELATA_BEARER_TOKEN when not set explicitly, so you can't accidentally leave it unset on cluster.enabled: true.
• RELATA_GRPC_PLAINTEXT_OK / RELATA_GRPC_TLS_* are wired for you depending on cluster.grpcPlaintext / tls.enabled.

Use this page to understand the mechanics, debug a Helm-based cluster that isn't fan-out-ing correctly, or run cluster profile somewhere that isn't Kubernetes at all (bare metal, systemd units on separate hosts, Docker Compose). For a real deployment, prefer the Helm chart.

See also#

• Deployment — the three profiles and when to use each
• Kubernetes Deployment — the Helm-chart production path
• Environment Variables — full reference
• Limits & Caveats — capacity & scaling caveats for cluster profile