Data Flow

RelataDB has two dataflows — ingest and query — and they share one storage layout. This page walks both end to end. For the conceptual overview first, read How It Works.

Ingest: eager vs lazy

The single most important idea on the write path is the two-phase split. Writes are divided into an eager phase (synchronous, in the writer's commit critical path) and a lazy phase (asynchronous, re-runnable from the WAL).

SOURCE         Connector         Writer (eager)                 Storage             Indexer (lazy)
──────         ─────────         ──────────────                 ───────             ──────────────
CSV  ─┐                          validate + canonicalize ──►  Parquet segment
JSON  ├─► normalize ────────►    ACL + tenant stamp       ──►  CSR adjacency    ──► IdentityIndex delta
API  ─┤                          WAL append (fsync)        ──►  blob (SHA-256)       embeddings (HNSW)
Kafka ┘                          row write (&self)         ──►  PROV-O assertion     MV refresh
                                                            └──► commit manifest      SmartIngest

The eager phase (on the hot path)

The eager phase does only declared-column work: parse, canonicalize, stamp principal/agency/classification, append the WAL, write the row. It deliberately does not run embeddings, refresh the IdentityIndex, or extract relationship mentions.

Property	Why it matters
Bounded work	Keeps writer p99 in single-digit milliseconds — no probabilistic work blocks the commit.
Backpressure is real	A bounded ingest queue overflows to HTTP 429 rather than an unbounded memory cliff.
Durability before ack	A row is not acknowledged until the WAL is fsync'd; kill-9 recovery replays the WAL with RPO=0 for acked writes.
Provenance is mandatory	Every row carries source/collector/confidence — there is no "ungoverned" write path.
Bi-temporal by construction	Corrections insert a new row and close the prior `valid_to`; the old belief is never overwritten.

The lazy phase (materialized views, later)

The lazy phase is driven by the indexer role. It reads the WAL delta since the last committed offset and runs:

materialized-view refresh,
embedding generation,
SmartIngest identity extraction,
relationship (mention) inference.

Every lazy step is re-runnable: a crashed indexer resumes at the last committed WAL offset. All lazy output is itself bi-temporal — late corrections are first-class, never in-place overwrites.

Why split eager and lazy? Detection, free-text extraction, and identity fusion are probabilistic and evolving. Running them lazily means a detector upgrade triggers a materialized-view refresh, not a source backfill — and a wrong detector's blast radius is one MV recompute, not polluted source data.

Durability is tunable per write: the X-Relata-Durability request header (or RELATA_WAL_SYNC=true cluster-wide) selects async (~10ms background flush) for bulk loads or sync (fsync before ack) for high-value records. Same table, same writer, different durability per row.

The query path

Client                                              Result
──────                                              ──────
psql / gRPC / HTTP / MCP / Arrow Flight             Arrow batches
SPARQL                                              + optional PROV-O columns
compat doors: S3 / pgvector / ClickHouse            + audit log entry
               / Neo4j / Redis / MongoDB / Bolt            ▲
        │                                                    │
        ▼                                                    │
  Wire auth ─► principal (OIDC JWT / mTLS cert)              │
        │                                                    │
        ▼                                                    │
  Coordinator ─► parse + PURPOSE check                       │
               ─► ACL bitmap compilation                     │
               ─► cost-based optimize + MV rewrite           │
        │                                                    │
        ▼                                                    │
  Execution (streaming, no full materialization)             │
    ─► columnar scan (Parquet + mutable tail)                │
    ─► CSR graph traversal (BFS / Pregel)                    │
    ─► vector ANN (custom HNSW / DiskANN warm tier)          │
    ─► BM25 full-text (custom, not Tantivy)                  │
    ─► bitmap intersection (ACL + filter, fused)             │
        │                                                    │
        ▼                                                    │
  Cache ─► L1 foyer (RAM/NVMe) → L2 ring → L3 S3 ───────────┘

The stages:

Wire auth — every door resolves the caller into a principal. Identity is established before any planning.
Coordinator — parse, the optional PURPOSE check, ACL bitmap compilation, cost-based optimization, and materialized-view rewrite. The ACL bitmap for a (principal, type) pair is computed once and cached.
Execution — a streaming plan against columnar, graph, vector, and full-text stores, fused. The executor never holds the full result set in memory; large hash-join build sides spill to local NVMe.
Assemble — rows are packed into Arrow batches. WITH PROVENANCE attaches PROV-O columns. An audit log entry is written.

Why governance doesn't slow you down

ACL predicates are compiled into bitmap scan predicates before execution begins, never applied as a post-filter. The bitmap is intersected in the scan hot loop alongside the filter. The budget gate rejects (with an RFC 7807 error) any query whose estimated ACL cost would exceed a hard cap, preventing a pathological rule from exhausting the reader.

Cache tiers

Tier	Technology	Latency	Scope
L1	foyer (in-process, NVMe-backed)	sub-ms	Single reader node
L2	Consistent-hash ring across reader nodes	~1 ms	N=2 hot replicas
L3	S3 / R2 / GCS / MinIO	20–100 ms	Durable, all data

L1 admission uses S3-FIFO — a scan-resistant policy so a full-table scan cannot evict hot keys. The query result cache memoizes results keyed by (SQL text, principal, AS OF time) and is invalidated by commit manifests.

What this means in practice

Drop-in, then grow. Point an existing client at a door and you are running on a governed, temporal knowledge engine. The eager/lazy split means the fast path stays fast no matter how much lazy enrichment you enable.
Consistency you can reason about. Single-writer-per-branch strong consistency at baseline; readers are always lock-free against the writer.
An audit story, not an afterthought. Because provenance, bi-temporality, and ACL are in the query path, every read and write is attributable by construction — you don't bolt compliance on later.

Data Flow#

Ingest: eager vs lazy#

The eager phase (on the hot path)#

The lazy phase (materialized views, later)#

The query path#

Why governance doesn't slow you down#

Cache tiers#

What this means in practice#

See also#