Scaling#

RelataDB scales from a laptop (free profile, ~1 B entities) to a multi-node cluster (cluster profile, 100 B–1 T+ entities). The key levers: RAM walls that engage automatically on server/cluster, paged backends that spill to object storage, streaming execution that never OOMs, and lazy cold-restart that brings large nodes back in seconds.

Choose the right profile#

# Local dev, CI, demos — unbounded RAM, eager restart
RELATA_PROFILE=free relata serve
 
# Single-node production — 1 GB RAM wall, disk-first walls, lazy restart on
RELATA_PROFILE=server RELATA_BEARER_TOKEN=<your-strong-token> relata serve
 
# Multi-node (alpha) — same as server plus coordination
RELATA_PROFILE=cluster RELATA_BEARER_TOKEN=<your-strong-token> relata serve

Single-node server is the production-recommended profile today. cluster is alpha — petabyte-scale sharding is still in progress (epic #797).

All scaling knobs#

Set any of these to override the profile default. Explicit values always win.

Tune RAM walls for your node#

The 1 GB default is conservative. On a 32 GB node running only RelataDB, raise it:

RELATA_PROFILE=server \
RELATA_BEARER_TOKEN=<your-strong-token> \
RELATA_STORE_MAX_RAM_MB=16384 \
RELATA_GRAPH_RAM_BUDGET_MB=8192 \
RELATA_IDENTITY_RAM_BUDGET_MB=4096 \
relata serve

The caps are byte-aware — small datasets never spill. The wall only bites when you actually exceed the budget.

Disk-first walls and paged backends#

Every large structure has a paged backend. When the RAM budget is exceeded, data pages out to the object store and pages back in on demand. RAM becomes a cache; the object store is the truth.

Paging adds disk-read latency on cold paths but eliminates OOM. On hot paths, the cache hierarchy absorbs most reads.

Streaming execution#

The execution engine never materialises full intermediate results. Big joins, high-cardinality aggregates, and large result sets all stream:

# This query on a 50 M-row table streams results — does not OOM
relata query "SELECT dept, COUNT(*) FROM Person GROUP BY dept"

Hash-join and aggregate operators spill intermediate batches to the object store when they exceed the RAM budget, then merge-read in a streaming pass.

Columnar analytics#

Filter-free GROUP BY uses a vectorised columnar path that reads only the referenced columns:

-- Fast: only reads the "dept" column
SELECT dept, COUNT(*) FROM Person GROUP BY dept;
 
-- Slower: filter needs the full row to evaluate "active"
SELECT dept, COUNT(*) FROM Person WHERE active GROUP BY dept;

For analytical workloads on large types, design queries to avoid per-row filters where possible, or add a bloom-filtered column to narrow the scan before the filter.

Lazy restart#

RELATA_LAZY_RESTART=true (default on server/cluster) loads only the manifest catalog on startup. Rows hydrate on the first query that touches them. A 10 M-row node returns to ready in seconds instead of ~55 s.

# Pre-warm the 5 most recent segments at startup, stay lazy for the rest
RELATA_LAZY_RESTART=true \
RELATA_HYDRATE_RECENT_SEGMENTS=5 \
relata serve

This is the right setting for most production nodes: fast restart with warm cache for recent data.

Cache hierarchy#

S3-FIFO admission at L1 is scan-resistant — a one-off full-table scan does not evict hot working set. L2 consistent-hashing means a request routed to the right reader node hits warm cache without cross-node fetch.

Graph and vector scaling#

Graph — CSR adjacency stays RAM-resident up to RELATA_GRAPH_RAM_BUDGET_MB, then pages to PagedCsrGraph. The incremental degree index keeps DEGREE() queries O(1) regardless of graph size.

Vectors — HNSW is the in-memory graph for recall. DiskANN is the warm tier backed by object-store segments (ADR-185). The IVF cold bucket stages vectors in RAM up to RELATA_VECTOR_COLD_RESIDENT_MAX before spilling to PagedAnnIndex. Increase RELATA_DISKANN_MAX_RESIDENT on nodes with headroom to avoid premature spill warnings.

Cluster mode (alpha)#

Cluster adds coordinator/reader/writer/indexer roles, hash partitioning, and multi-region replication. Single-node server is production-recommended until the petabyte-scale sharding work lands (epic #797); do not run cluster in production for data you cannot afford to lose until the alpha label is removed.

➡️ The canonical cluster recipe — every required env var (plain NODE_ID / CLUSTER_ROLE / CLUSTER_PEERS — not the RELATA_-prefixed names, which are different, unrelated vars), a tested local 3-node example, and the gotchas (gRPC port must match across nodes, CLUSTER_AUTH_TOKEN fails silently, three different bind-var conventions) — lives in Cluster Setup. Scaling-wise, cluster inherits the same RAM walls, paged backends, streaming execution, and lazy restart described above; the cluster-specific levers are the partition-count (RELATA_CLUSTER_SHARDS, default 8) and the cross-region merge cadence (RELATA_CROSS_REGION_MERGE_INTERVAL_SECS).

Performance characteristics#

Measured on representative hardware at 10 M rows. Use these as order-of-magnitude guides.

Cell masking is expensive because it must inspect every field of every returned row. Apply it to specific columns in specific policies, not globally.

Run the benchmark suite#

Before a capacity change, establish a baseline:

# Quick gate — under 1 min, run before every merge
cargo run -p relata-bench --release -- full --scale 100k --gate
 
# Full suite without HNSW builds (~8 min)
cargo run -p relata-bench --release -- full --scale 100k --no-ann
 
# Full suite including HNSW builds (~15 min+)
cargo run -p relata-bench --release -- full --scale 100k
 
# All scales
cargo run -p relata-bench --release -- full
 
# Memory recall at scale
cargo run -p relata-bench --release -- memory --scale 100k --all

Set RELATA_BENCH_NO_SAVE=1 to suppress writing relata-bench.json.

See also#

• Configuration — every scaling env var
• Backup & Restore — object-store setup, lazy restart
• Observability — RAM wall metrics, queue depth