Capacity planning and WAL retention
/v1 prefix; see the route restructure.This page is a first-order sizing model for one node of the open edition. The budgets below constrain the structures they cover; they are not a whole-process memory cap. Choose a container memory limit from measurements of your own workload and keep headroom. Every figure marked as observed comes from a recorded run on a specific host and is not a product limit or a published number.
Resident-memory budgets
When a budget variable is unset, Relata sizes it from effective RAM (host RAM clamped to the smallest finite cgroup-v2 limit visible). The startup banner prints the chosen split.
| Consumer | Variable | Default when unset |
|---|---|---|
| Row store | RELATA_STORE_MAX_RAM_MB | 28% of effective RAM; crossing it spills cold rows to disk |
| Secondary indexes | RELATA_STORE_MAX_INDEX_MB | 9%; 0 disables the cap |
| Cache | (adaptive) | 9% |
| Vectors | RELATA_DISKANN_MAX_RESIDENT | 15%, as a resident vector count |
| Full-text index | RELATA_FTS_MAX_RESIDENT_BYTES, RELATA_FTS_MAX_DOCS | 9% byte allowance; document-count trigger off |
| Queued ingest payloads | RELATA_INGEST_QUEUE_MAX_BYTES | 5% |
These shares total 75% of effective RAM. Materialized views, query scratch, runtime state and in-flight work are not covered by them, and explicit overrides can exceed the pool. Invalid numeric values fail startup with exit code 78.
Rules of thumb:
- A 768-dimension f32 embedding is about 3 KiB resident before graph overhead (
dim x 4bytes). - Size for versions, not ids. History versions stay resident until retained-history maintenance spills or prunes them, so rewriting 100,000 ids three times keeps about 400,000 resident rows.
- A rollover (below) transiently needs memory proportional to the compact checkpoint, about 1 KiB per version in the measured case.
The WAL and generation rollover
A node's durable state is a generation: an immutable checkpoint plus a WAL tail of segment files written since. The tail is never pruned in place, so the file count grows with every acknowledged write that changes data. Re-asserting a byte-identical row appends nothing, so a load generator that cycles a fixed key space shows a flat file count and proves nothing about reclamation.
The tail is bounded by rolling to a new generation whose checkpoint is a fresh capture of live state and whose tail is empty.
| Trigger | Threshold |
|---|---|
| Startup finds the tail past the rollover point | 4,096 segments or 1 GiB |
| Serving node rolls over | 32,768 segments or 4 GiB (50% of the hard limit) |
/health/ready returns 503 generation-wal-near-limit | 80% of either hard limit |
| Writes refused | 65,536 segments or 8 GiB |
The previous generation is kept for one more cycle, so the steady-state file count is a sawtooth peaking near two tails (about 65,000 files).
Each runtime rollover pauses writes. Callers get a retryable 503 with Retry-After: 5. On a store of about 130,000 rows the pause was on the order of 10 seconds in a macOS debug run. Clients must retry 503 with backoff. A due rollover is postponed while the memory watchdog is shedding writes (503 memory-pressure), but never once the tail reaches 80% of a limit.
Operator actions
- Watch
relata_wal_generation_tail_segmentsagainstrelata_wal_generation_tail_segment_limit, andrelata_generation_rollover_failures_total. - A tail climbing past 32,768 segments with no increase in
relata_generation_rollovers_total, or any rise in the failure counter, means rollover is not completing. Restart the node: startup rolls any tail past 4,096 segments before serving. Restart replay time grows with the tail, so do not wait for the hard limit. - File counts below about 32,768 are normal. Never delete WAL files by hand: recovery refuses a tail with missing segments.
What the recorded runs show
The runs below used one shared Apple-silicon host with an aarch64 Linux guest (4 vCPU, 6 GiB, no swap). Timings are indicative only.
- Ingest slows past about 115,000 rows at default index caps. Queued bulk ingest ran at about 1,500 rows/s, then logged "index RAM cap reached during concurrent admission" and fell to about 34 rows/s. The 400,000-row target was not reached at default caps (reproduced twice).
- Raising the index cap moves the limit to memory. With
RELATA_STORE_MAX_INDEX_MB=4096, 400,000 rows ingested in 267 s with RSS near 5.1 GB of the 6 GB guest. With a larger pending-index limit, ingest returned 503memory-pressurenear 352,500 rows. - Restart after a large ingest failed on Linux in three runs (250k, 300k, 400k rows) with
WAL replay failed ... deferred index-work queue full. A fix was verified by unit tests on macOS; it has not been re-verified on Linux. - Restart of a 200,000-vector store did not become ready within 36 minutes in one run. 10 million vectors were not run.
Do not plan on more than roughly 115,000 rows of sustained queued ingest at default caps without measuring your own shape, and rehearse restart after a large load before relying on it. The full list is in 3.0 evidence status.