# RelataDB — full agent reference > Single-file plain-text dump of every rolling-latest docs page. > Built by ZySec AI (https://www.zysec.ai). Product home: https://relatadb.dev > Pages: 107. Each section starts with its canonical URL. > For the curated summary see https://relatadb.dev/llms.txt ============================================================================== # Crate Map URL: https://relatadb.dev/docs/architecture/crate-map ============================================================================== # Crate Map `relata-core` is the dependency root — every other crate in the workspace depends on it. The next ring is storage + query + canonical + ontology. The outer ring is server, CLI, and domain modules. Dependency arrows always point inward toward `relata-core`; no inner crate depends on an outer crate. ## Dependency graph ```text ┌──────────────┐ │ relata-core │ ← every crate depends on this └──────┬───────┘ ┌─────────────────┼─────────────────┐ ▼ ▼ ▼ relata-canonical relata-storage relata-ontology │ │ │ └────────┬────────┘ │ ▼ │ relata-query ◄────────────────────┘ │ ┌─────────────┼──────────────┐ ▼ ▼ ▼ relata-graph relata-acl relata-identity │ │ │ └─────┬───────┴──────┬───────┘ ▼ ▼ relata-prov relata-detect │ │ └──────┬───────┘ ▼ relata-server ◄── relata-cluster, relata-cache │ ▼ relata-cli ``` ## Foundation | Crate | Layer | Role | |---|---|---| | `relata-core` | Foundation | Core types: bi-temporal row model, `ObjectId` / `LinkId` / `EventId` / `RowId` (16-byte newtypes; production `RowId` is a server-minted counter — see [Data Model](/docs/architecture/data-model#identifier-types)), `Identity` datatype, HLC timestamps. The one crate that depends on nothing else in the workspace. | ## Storage and data plane | Crate | Layer | Role | |---|---|---| | `relata-storage` | Storage | In-memory bi-temporal store with per-type interior locking (`&self` writes via `PlRwLock`). WAL. Custom BM25 full-text (integer posting lists + q-gram prefix + trigram suffix + 15 hand-rolled stemmers (en/fr/de/es/pt/it/nl/sv/no/da/fi/hu/ro + Russian Cyrillic, Turkish, Arabic) + CJK pass-through + subword tokenization (camelCase/digit); NOT Tantivy). Vectors: custom HNSW (`vector.rs`) + DiskANN warm tier (`vector_diskann.rs`). Backup/restore. Per-column bloom filters. Summary index for O(1) `COUNT`/`SUM`. HLL + CMS cardinality sketches. Pure-Rust Kafka ingest adapter. Arena allocation via `bumpalo`. | | `relata-canonical` | Storage | ~76 canonical type validators shipped (email, IBAN, MMSI, VIN, IMEI, MSISDN, Aadhaar, GeoPoint, and more). Deterministic binary encoding. The target catalogue is ~170; the shipped count is in `crates/relata-canonical/src/lib.rs`. | | `relata-ontology` | Storage | Schema-as-code. Git-branched ontology with HEAD pointer. State-machine constraints on `PropertySpec`. Computed columns. | ## Query and compute | Crate | Layer | Role | |---|---|---| | `relata-query` | Query | SQL parser → planner → executor. DataFusion bridge for columnar aggregation. Cost-based optimizer (join ordering + index selection). MV refresh (`ON COMMIT`, `IncrementalAggregate + HLL`). Watch subscriptions. Query result cache. Pattern tracker + speculative prefetch. `ReadOptions` / `WITH CACHE`. | | `relata-graph` | Query | CSR adjacency. BFS/DFS. Pregel-style iterative BFS. PLL distance index (`pll.rs`) wired into `GRAPH_SSSP` (`algo => 'pll'`) and `GRAPH_DIJKSTRA` (reachability pre-check); `PATHS_BETWEEN` uses BFS/DFS. 10+ SQL graph operators (`GRAPH_DIJKSTRA`, `GRAPH_LINK_PREDICT`, `GRAPH_SCC`, `GRAPH_CYCLES`, `GRAPH_SSSP`, `GRAPH_SPANNING_TREE`, `GRAPH_APSP`, `GRAPH_DIAMETER`, `GRAPH_SIMILARITY`, `GRAPH_NODE_METRIC`). Incremental degree index, exposed as the `DEGREE()` SQL function. | ## Identity and detection | Crate | Layer | Role | |---|---|---| | `relata-identity` | Domain | `IdentityIndex` materialized view. `RESOLVE_IDENTITY`, `IDENTITY_CLUSTER`, and `SAME_IDENTITY` SQL operators. Substrate for cross-source fusion. | | `relata-detect` | Domain | Two-phase SmartIngest: eager (validate + canonicalize in the writer's commit path) and lazy (`DETECT_IDENTITIES` operator run by the indexer). Configurable detector packs: `network`, `contact`, `crypto` (default on); `financial`, `payment`, `social`, `transport`, `device`, `ics` (opt-in); `all`, `none`. | ## Governance and trust | Crate | Layer | Role | |---|---|---| | `relata-acl` | Trust | Cedar-inspired ABAC. Deny-wins evaluation. Bitmap row filtering (precomputed per `(principal, type)`). Cell masking (redact / hash / partial). | | `relata-prov` | Trust | PROV-O assertions. Hash-chained commit manifests. Content-addressed blobs. Audit log replay. | ## Server and clients | Crate | Layer | Role | |---|---|---| | `relata-server` | Server | Postgres wire protocol + gRPC + Arrow Flight + MCP. Auth: OIDC, mTLS. Hosts the query coordinator on `server`/`cluster` profiles. | | `relata-cli` | Server | The `relata` binary entry point. Profiles: `free` / `server` / `cluster` (`lite` is a removed legacy alias — rejected at startup). Hosts the protocol-compatibility servers: `s3_server.rs`, `pgwire_listener.rs`, `clickhouse_server.rs`, `neo4j_server.rs`, `redis_server.rs`, `mongo_server.rs`, `bolt_server.rs`. | | `relata-sdk-rust` | Client | Internal/reference client (not a published consumer SDK — Python/TypeScript/Go are the published SDKs). gRPC + HTTP + in-memory client. Arrow-IPC zero-copy. RFC 7807 error shape. Multi-tenant headers. `SearchBuilder` for the `/search` API. Used by the server binary, the tray app, the test harness, and `relata-bench`. | ## Cluster and cache | Crate | Layer | Role | |---|---|---| | `relata-cluster` | Distribution | Coordinator / reader / writer / indexer roles. Hash partitioning. Multi-region replication. Branch-level writer leases. Status: alpha — not recommended for production. | | `relata-cache` | Cache | RAM-only foyer cache. S3-FIFO admission (scan-resistant). | ## Intelligence and feeds | Crate | Layer | Role | |---|---|---| | `relata-feed`, `relata-feed-broker` | Intelligence | RIFN intelligence feed network. Inbound feed ingestion and fan-out. | | `relata-jobs` | Intelligence | Continuous pattern-detection jobs (C2 beacon, convoy, transaction ring). Governance-aware DAG workflow engine. | | `relata-intelligence` | Intelligence | Incident clustering. Anomaly detection. LLM interpretation. Detection-rule tuning: snooze, suppression, exception lists. | ## Extension points These crates define the contract that external extension crates must implement. They are not run directly. | Crate | Role | |---|---| | `relata-connector-stub` | Trait shapes for external `relata-connector-*` crates (data source connectors). | | `relata-pack-stub` | Trait shapes for external `relata-pack-*` crates (domain-specific intelligence packs). | Domain packs and connectors live in separate repositories (`relata-pack-*`, `relata-connector-*`). They are not part of this workspace. ## Benchmarks | Crate | Role | |---|---| | `relata-bench` | 50+ benchmarks + conformance runner. `--gate` quick pre-merge (~30s). `--no-ann` skips HNSW builds (~8 min vs. ~15 min). Protocol coverage for all 8 compat doors + 5 native protocols. Not in the workspace test gate; run separately. | ## Testing | Crate | Role | |---|---| | `relata-testing` | In-process test fixtures. `TestFixture` spins up an in-memory `ObjectStore`; `spawn_ephemeral` (behind the `ephemeral` feature) starts a real `relata` process on a random port for SDK integration tests. Consumed by the workspace test gate, not shipped at runtime. | ## Apps > **Console and Portal live in SEPARATE repositories** (`github.com/relatadb/console` and `github.com/relatadb/portal`), not in this workspace. Only `apps/relata-tray` ships in-repo. They are listed here for orientation; they are not part of the Rust workspace test gate. | App | Repo | Role | |---|---|---| | Console | `github.com/relatadb/console` (separate repo) | Next.js showcase + ops surface. 68 pages, 19 tutorials, 24 docs, 8 interactive showcases. Force-directed graph explorer, universal search, CDR/SDR analyzer, case investigator. Talks to the server via `/api/relata/*` proxy. | | Portal | `github.com/relatadb/portal` (separate repo — this site) | Next.js docs and landing site. Hand-authored rolling-latest surface with per-version snapshots. | | `apps/relata-tray` | in-repo | macOS menu-bar app (Tauri 2). Click-to-launch server + popup health WebView. Bundled in the DMG release. | ## Build invariants These are enforced at the workspace level, not per-crate. > **No `unsafe` in library code.** Every `lib.rs` compiles with `unsafe_code` forbidden or denied — most carry `#![forbid(unsafe_code)]`; `relata-query` and `relata-storage` carry `#![deny(unsafe_code)]`; `relata-cluster` forbids it outside tests. The single workspace exception is the `relata` CLI binary, whose smoke-test harness uses a minimal `libc::kill` FFI shim. > **`missing_docs` is denied.** Every public item has a doc comment. Builds fail otherwise. > **Workspace resolver `"3"`**, **edition 2024**, **MSRV 1.85**. > **`parking_lot` locks only.** `PlMutex` and `PlRwLock` type aliases are the only mutexes used. parking_lot does not poison on panic. > **Workspace dependency pinning.** All shared dependency versions live in `[workspace.dependencies]`. Use `dep.workspace = true`, not inline version strings. > **`cargo deny check`** enforces a license allowlist and bans unknown registries and git dependencies. ## See also - [Architecture Overview](/docs/architecture/overview) - [Data Model](/docs/architecture/data-model) - [Storage Engine](/docs/architecture/storage) - [Query Engine](/docs/architecture/query-engine) ============================================================================== # Data Flow URL: https://relatadb.dev/docs/architecture/data-flow ============================================================================== # Data Flow RelataDB has two dataflows — **ingest** and **query** — and they share one storage layout. This page walks both end to end. For the conceptual overview first, read [How It Works](/docs/architecture/how-it-works). ## Ingest: eager vs lazy The single most important idea on the write path is the **two-phase split**. Writes are divided into an *eager* phase (synchronous, in the writer's commit critical path) and a *lazy* phase (asynchronous, re-runnable from the WAL). ```text SOURCE Connector Writer (eager) Storage Indexer (lazy) ────── ───────── ────────────── ─────── ────────────── CSV ─┐ validate + canonicalize ──► Parquet segment JSON ├─► normalize ────────► ACL + tenant stamp ──► CSR adjacency ──► IdentityIndex delta API ─┤ WAL append (fsync) ──► blob (SHA-256) embeddings (HNSW) Kafka ┘ row write (&self) ──► PROV-O assertion MV refresh └──► commit manifest SmartIngest ``` ### The eager phase (on the hot path) The eager phase does **only declared-column work**: parse, canonicalize, stamp principal/agency/classification, append the WAL, write the row. It deliberately does **not** run embeddings, refresh the `IdentityIndex`, or extract relationship mentions. | Property | Why it matters | |---|---| | Bounded work | Keeps writer p99 in single-digit milliseconds — no probabilistic work blocks the commit. | | Backpressure is real | A bounded ingest queue overflows to HTTP 429 rather than an unbounded memory cliff. | | Durability before ack | A row is not acknowledged until the WAL is fsync'd; kill-9 recovery replays the WAL with RPO=0 for acked writes. | | Provenance is mandatory | Every row carries source/collector/confidence — there is no "ungoverned" write path. | | Bi-temporal by construction | Corrections insert a new row and close the prior `valid_to`; the old belief is never overwritten. | ### The lazy phase (materialized views, later) The lazy phase is driven by the **indexer** role. It reads the WAL delta since the last committed offset and runs: - materialized-view refresh, - embedding generation, - SmartIngest identity extraction, - relationship (mention) inference. Every lazy step is **re-runnable**: a crashed indexer resumes at the last committed WAL offset. All lazy output is itself bi-temporal — late corrections are first-class, never in-place overwrites. > **Why split eager and lazy?** Detection, free-text extraction, and identity fusion are probabilistic and evolving. Running them lazily means a detector upgrade triggers a materialized-view refresh, not a source backfill — and a wrong detector's blast radius is one MV recompute, not polluted source data. Durability is tunable per write: the `X-Relata-Durability` request header (or `RELATA_WAL_SYNC=true` cluster-wide) selects `async` (~10ms background flush) for bulk loads or `sync` (fsync before ack) for high-value records. Same table, same writer, different durability per row. ## The query path ```text Client Result ────── ────── psql / gRPC / HTTP / MCP / Arrow Flight Arrow batches SPARQL + optional PROV-O columns compat doors: S3 / pgvector / ClickHouse + audit log entry / Neo4j / Redis / MongoDB / Bolt ▲ │ │ ▼ │ Wire auth ─► principal (OIDC JWT / mTLS cert) │ │ │ ▼ │ Coordinator ─► parse + PURPOSE check │ ─► ACL bitmap compilation │ ─► cost-based optimize + MV rewrite │ │ │ ▼ │ Execution (streaming, no full materialization) │ ─► columnar scan (Parquet + mutable tail) │ ─► CSR graph traversal (BFS / Pregel) │ ─► vector ANN (custom HNSW / DiskANN warm tier) │ ─► BM25 full-text (custom, not Tantivy) │ ─► bitmap intersection (ACL + filter, fused) │ │ │ ▼ │ Cache ─► L1 foyer (RAM/NVMe) → L2 ring → L3 S3 ───────────┘ ``` The stages: 1. **Wire auth** — every door resolves the caller into a principal. Identity is established before any planning. 2. **Coordinator** — parse, the optional `PURPOSE` check, ACL bitmap compilation, cost-based optimization, and materialized-view rewrite. The ACL bitmap for a `(principal, type)` pair is computed once and cached. 3. **Execution** — a streaming plan against columnar, graph, vector, and full-text stores, **fused**. The executor never holds the full result set in memory; large hash-join build sides spill to local NVMe. 4. **Assemble** — rows are packed into Arrow batches. `WITH PROVENANCE` attaches PROV-O columns. An audit log entry is written. ### Why governance doesn't slow you down ACL predicates are compiled into bitmap scan predicates **before execution begins**, never applied as a post-filter. The bitmap is intersected in the scan hot loop alongside the filter. The budget gate rejects (with an RFC 7807 error) any query whose estimated ACL cost would exceed a hard cap, preventing a pathological rule from exhausting the reader. ### Cache tiers | Tier | Technology | Latency | Scope | |---|---|---|---| | L1 | foyer (in-process, NVMe-backed) | sub-ms | Single reader node | | L2 | Consistent-hash ring across reader nodes | ~1 ms | N=2 hot replicas | | L3 | S3 / R2 / GCS / MinIO | 20–100 ms | Durable, all data | L1 admission uses S3-FIFO — a scan-resistant policy so a full-table scan cannot evict hot keys. The query result cache memoizes results keyed by `(SQL text, principal, AS OF time)` and is invalidated by commit manifests. ## What this means in practice - **Drop-in, then grow.** Point an existing client at a door and you are running on a governed, temporal knowledge engine. The eager/lazy split means the fast path stays fast no matter how much lazy enrichment you enable. - **Consistency you can reason about.** Single-writer-per-branch strong consistency at baseline; readers are always lock-free against the writer. - **An audit story, not an afterthought.** Because provenance, bi-temporality, and ACL are in the query path, every read and write is attributable by construction — you don't bolt compliance on later. ## See also - [How It Works](/docs/architecture/how-it-works) — the conceptual mental model. - [SmartIngest](/docs/architecture/smartingest) — what the lazy identity phase actually does. - [Jobs & Triggers](/docs/architecture/jobs-and-triggers) — the typed-Job engine behind the lazy phase. - [Architecture Overview](/docs/architecture/overview) · [Storage Engine](/docs/architecture/storage) · [Query Engine](/docs/architecture/query-engine) ============================================================================== # Data Model URL: https://relatadb.dev/docs/architecture/data-model ============================================================================== # Data Model The ontology is the schema. You do not write `CREATE TABLE`. You declare `ObjectType`, `EventType`, `LinkType`, and `ActionType`. The storage layout, planner, SDKs, and protocol compatibility doors all derive from those declarations — there is no separate schema definition per surface. ## Identifier types Every entity, link, and event in the store has a stable, typed identifier. Four wrapper newtypes exist at the core level: | Type | Underlying | Minted by | Purpose | |---|---|---|---| | `ObjectId` | 16 bytes | Derived (byte-borrow of a `RowId`) | Stable identifier for an instance of an `ObjectType` | | `LinkId` | 16 bytes | Currently unminted in production (type defined for future wired edges) | Stable identifier for a typed, directional edge (`LinkType` instance) | | `EventId` | 16 bytes | Currently unminted in production (type defined for the causal-event graph) | Stable identifier for a time-anchored occurrence (`EventType` instance) | | `RowId` | 16 bytes | `alloc_row_id()` (`store/mod.rs:4300`) — 8-byte big-endian `AtomicU64` counter, zero-padded | Internal identifier for one materialised storage row (one bi-temporal version) | These are distinct Rust newtypes — `ObjectId` cannot be passed where a `LinkId` is expected. **Note the implementation reality:** the wire format renders as a 36-char UUID-style string (8-4-4-4-12 hex) for tooling compatibility, but production `RowId`s are **server-minted monotonic counters** (`alloc_row_id`, `store/mod.rs:4300`), not random UUID v4 values — version/variant nibbles are deliberately left zero so the counter stays injective (a v4 mask would collide every 4096 rows). `ObjectId` is a non-copy byte-borrow of the underlying `RowId`, so it shares the counter layout. Implications users should know: - **Sequential, not random** — `RowId` increases monotonically with insert order (counter big-endian in bytes 0–7). - **Server-minted, not client-generatable** — clients receive IDs from the server; supplying your own is not supported on the INSERT path. - **Stable within a process lifetime and across disk-spill** — cold-spill rows encode `row.id` (`spill_encode_row`, `store/mod.rs:6720`) so they survive restart. - **Re-minted on WAL replay** — `wal_encode_record` (`store/mod.rs:6578`) does not serialise `row_id`, so a row reconstructed purely from WAL during crash recovery gets a fresh counter value. References to a specific `RowId` from outside the store (e.g. in a downstream system) are not guaranteed to match post-recovery. The original type-level doc-comment in `crates/relata-core/src/id.rs` calls these "UUID-v4 wrappers"; that description is overdue for a correction (tracked separately) — the implementation has been counter-based since the v4-mask collision fix at `store/mod.rs:4304-4309`. ## The three ontology primitives | Primitive | Purpose | Examples | |---|---|---| | **ObjectType** | A thing with stable identity. Survives property changes via bi-temporal rows. | `Profile`, `Subscriber`, `Account`, `Device` | | **EventType** | A time-anchored occurrence. Bi-temporal by construction. | `CallEvent`, `Post`, `Transaction`, `LoginAttempt` | | **LinkType** | A typed, directional edge between two instances. The substrate of the graph plane. | `CALLED`, `MENTIONS`, `OWNS`, `TRANSFERRED_TO` | Declare them in TOML or Rust. The ontology is schema-as-code, versioned in git, branched with a HEAD pointer — schema changes go through the same review process as application code. ```rust // Example ObjectType declaration ObjectType: name = "Profile" [properties] handle = { type = "String", identity = "Handle" } msisdn = { type = "Identity", canonical = "Msisdn" } bio_text = { type = "String", indexed = "bm25" } avatar = { type = "BlobRef" } verified = { type = "Bool" } location = { type = "GeoPoint" } ``` ### Schema features **State-machine constraints on `PropertySpec`** — a property can declare a finite state machine over its allowed values. Transitions that violate the declared machine are rejected at write time by the planner. **Computed columns** — a property can be declared as a deterministic expression over other properties of the same type. The value is computed at read time and cached; writes do not store it. ## Bi-temporal row shape Every row, regardless of primitive, carries the same structural columns. There is no opt-out. | Column | Type | Meaning | |---|---|---| | `id` | `RowId` | Stable identifier for this materialised row version (16-byte counter — see [Identifier types](#identifier-types)) | | `object_type` | `Arc` | The type this row belongs to (e.g. `"Person"`) | | `valid_from` | `i64` ns UTC | When the fact became true in the real world | | `valid_to` | `i64` ns UTC | When the fact stopped being true (`i64::MAX` = currently true) | | `system_from` | `i64` ns UTC | When the database first recorded this version of the fact | | `system_to` | `i64` ns UTC | When this version was superseded (`i64::MAX` = current version) | | `prov` | `ProvenanceRef` | 32-byte hash pointer to the creating PROV-O assertion; the full `ProvAssertion` lives in the `relata-prov` crate and may be attached inline at insert | | `tenant_id` | `Option` | Organisation / agency scope for cross-tenant isolation | | `data` | `RowData` | Property bag — field name → `CanonicalValue` | - **Valid time** answers "when was this true in the world?" A subscription ran from 2024-03-01 to 2024-09-15. - **System time** answers "when did the database know about it?" The subscription row was ingested 2024-03-02 and a correction was recorded 2024-09-20. A `SELECT` without `AS OF` reads the current valid-time slice at current system time. See [Bi-Temporal](/docs/concepts/bitemporal) for query semantics. ## Timestamps All timestamps are `i64` nanoseconds UTC. The parser accepts: - Decimal nanoseconds: `1762828800000000000` - UTC ISO-8601: `2025-11-11T00:00:00Z` - UTC ISO-8601 with fractional seconds: `2025-11-11T00:00:00.500Z` > Non-UTC offsets (`+05:30`, `-08:00`) are **rejected at parse time**. Convert to UTC before writing. The engine will not silently normalize. ## Canonical types Canonical types are deterministic, validated binary encodings of real-world identifiers. They make cross-source fusion cheap: the same phone number, encoded the same way in every row, joins without normalization at query time. | Canonical type | Encoding | Validation rule | |---|---|---| | `IPv4` | `uint32` big-endian | Range check | | `IPv6` | `uint128` | Range check | | `Msisdn` | E.164 `uint64` | Length, leading digit | | `Imei` | `uint64` | Luhn check | | `Iban` | Rearranged string + mod-97 | ISO 13616 check digits | | `Aadhaar` | `uint64` | Verhoeff check | | `GeoPoint` | S2 cell `uint64` | Lat/lon range | | `Hash_SHA256` | 32 bytes | Length check | | `Email` | Normalized lowercased bytes | RFC 5322 minimal | | `Vin` | 17 bytes | ISO 3779 check digit | | `Mmsi` | `uint32` (9 digits) | ITU mid-range check | > **~76 Tier-1 canonical types ship today.** The often-quoted ~170 is the target catalogue size, not the shipped count. The authoritative enum is `crates/relata-canonical/src/lib.rs`. Why canonical types matter: - **5–10× smaller storage** — a phone number is 8 bytes, not a 15-character string. - **SIMD-friendly joins** — fixed-width integers vectorize; variable-length strings do not. - **Cross-source matching** — the same phone number written by three different feeds has one byte representation. No `LOWER()`, no `TRIM()`, no normalization functions in the join predicate. - **Bloom filter effectiveness** — identical byte representations compress to the same bloom entry. ## Identity — the umbrella type `Identity` wraps `CanonicalKind + bytes`. Any property on any primitive can declare its type as `Identity`, and that value automatically participates in the universal `IdentityIndex`. ```rust pub struct Identity { pub kind: CanonicalKind, // Msisdn, Iban, Email, Imei, Hash_SHA256, ... pub bytes: Vec, // canonical-encoded payload } ``` A property declared `Identity` does two things: it stores the typed bytes on the row, and it writes an entry into the `IdentityIndex` materialized view that maps `(kind, bytes) → (object_id, source_table, source_column, observed_at)`. One index, every observation, every source. This is the substrate for `RESOLVE_IDENTITY`, `IDENTITY_CLUSTER`, and `SAME_IDENTITY` SQL operators. ## Locking model - **`parking_lot` locks** — `PlMutex` and `PlRwLock` type aliases. parking_lot does not poison on panic, which eliminates the `recover_poison()` pattern. - **Per-type interior locking** — writes take `&self`, not `&mut self`. Each object type owns its own `RwLock`. A write to `Profile` does not block a write to `Post`. - **Branch-level writer lease** — at most one active writer per branch, enforced by the coordinator. Readers are always lock-free against the writer. ## See also - [Architecture Overview](/docs/architecture/overview) — the five planes and design invariants - [Storage Engine](/docs/architecture/storage) — how rows land on disk - [Identity](/docs/concepts/identity) — the umbrella type and cross-source fusion in depth - [Bi-Temporal](/docs/concepts/bitemporal) — valid time vs. system time query semantics ============================================================================== # How It Works URL: https://relatadb.dev/docs/architecture/how-it-works ============================================================================== # How It Works This page is the conceptual entry point for engineers. It explains *what RelataDB does to your data* and *why it is built that way*, without dumping you straight into file paths and ADR numbers. Read this once and the rest of the architecture set makes sense. > For the deeper, reference-oriented version (planes, invariants, ADRs), see [Architecture Overview](/docs/architecture/overview). ## The 90-second mental model RelataDB is a single Rust binary that replaces a stack of databases — relational, graph, vector, full-text search, and an audit ledger — with one engine. The trade-off is deliberate: the engineering cost of building every plane in-process is accepted in exchange for eliminating the consistency, governance, and operational seams that fragment a polyglot stack. Whatever door data enters, three things happen to every row, automatically: 1. **Recognize** — identifiers (emails, phones, IBANs, VINs, IMEIs, MMSIs, hashes, …) are detected, validated, and stored as compact canonical types, then shaped to your declared ontology. 2. **Govern** — every row is stamped with principal/agency/classification, carries provenance (where it came from), and is written to a tamper-evident audit hash chain. 3. **Remember in time** — every row is bi-temporal. Updates *supersede*; nothing is overwritten. You can replay exactly what the database believed at any moment. Reads go through the same lens: access control compiles into the scan itself (not a post-filter), cells can be masked, and every read is attributed and logged. ## One binary, three profiles The same binary (`relata`) runs in three profiles, selected at startup with `RELATA_PROFILE=free|server|cluster`. The wire protocol, on-disk format, SQL dialect, and ACL semantics are **identical across all three** — only capacity knobs and which subsystems boot change. | Profile | Shape | Headroom | |---|---|---| | `free` | Embedded, local disk | ~1 B entities | | `server` | One node + S3-compatible object store | ~10 B entities | | `cluster` | Coordinator + reader(s) + writer + indexer | 100 B – 1 T+ entities | This is why the path from a laptop prototype to a cluster deployment is a config change, not a rewrite. ## The five planes Every feature belongs to exactly one of five planes. Knowing which plane a feature lives in predicts its operational surface — its latency, its failure mode, and its audit story. | Plane | Owns | |---|---| | **Data plane** | Ontology, typed tables, link tables, the universal `IdentityIndex`, bitmaps, vectors, full-text postings | | **Compute plane** | Planner, executor, cost-based optimizer, materialized-view refresh, the DAG job engine | | **Trust plane** | Cedar-inspired access control, classification labels, the audit log, PROV-O provenance, hash-chained manifests | | **Identity plane** | The `Identity` umbrella type, the `IdentityIndex` materialized view, identity-resolution operators | | **Distribution plane** | Coordinator/reader/writer/indexer roles, write leases, hash partitioning, replication | Cross-plane work goes through typed interfaces. There are no shared mutable globals between planes — a property that keeps the blast radius of any change bounded. ## The journey of a row The clearest way to understand RelataDB is to follow one row from the outside world to a query result. ```text You connect Recognize Govern Remember in time ─────────── ───────── ────── ──────────────── psql · S3 · Mongo · ──► canonicalize ──► ACL + tenant ──► bi-temporal rows Redis · ClickHouse · validate identities provenance stamp WAL (hash-chained) Neo4j · Bolt · shape to ontology audit entry Parquet segment Arrow · HTTP · MCP IdentityIndex delta ``` ### On the way in 1. **A door receives the payload.** The door is the source of truth — an S3 object, a Mongo document, or a Redis key *is* a governed row. Each door is opt-in with one env var and a port. 2. **The gateway authenticates** the caller into a principal (OIDC JWT / mTLS cert), resolves the tenant, and records any declared `PURPOSE`. 3. **The eager phase** validates and canonicalizes declared columns, appends the write-ahead log (fsync), and writes the bi-temporal row. It does *only* declared-column work and stays in single-digit-millisecond p99. 4. **The lazy phase** (asynchronous, driven by the indexer) refreshes materialized views, generates embeddings, runs SmartIngest identity extraction, and infers relationship edges. Every lazy step is re-runnable from the last committed WAL offset. See [Data Flow](/docs/architecture/data-flow) for the full eager/lazy split and the query path. ### On the way out 1. A request arrives over any door and is authenticated into a principal. 2. The coordinator parses, checks the optional `PURPOSE`, and compiles the principal's ACL into per-type **bitmap predicates**. 3. The cost-based optimizer picks join order, scan strategy, and applies materialized-view rewrite. 4. Execution is **streaming** — columnar scan, graph traversal, vector ANN, and BM25 full-text run fused, with no full intermediate materialization. 5. Results ship as Arrow batches (optionally with PROV-O columns), and an audit entry is written. The key idea: **policy lives in the query path**, not as a post-filter. ACL decisions are evaluated at the vectorized loop level alongside the filter, which is why governance does not mean a slow database. ## SmartIngest, in one paragraph SmartIngest is the deterministic identity pipeline that runs on ingest. It detects identifiers using per-type shape gates plus checksum validators (so an IBAN's mod-97 or an IMEI's Luhn must pass), indexes them into the universal `IdentityIndex`, and fuses records that share a validated identifier into one cluster. This is what lets the same human — appearing as a phone number in one feed, an email in another, and an account ID in a third — resolve to one entity, automatically. > **The honest boundary:** RelataDB does *identifier* extraction, not general named-entity recognition. There is no in-tree transformer. For names-in-prose, intent, or sentiment, you register an external scorer; RelataDB writes its output as governed, provenance-stamped assertions. See [SmartIngest](/docs/architecture/smartingest). ## Jobs and what triggers them A core invariant: **every background work item is a typed `Job`.** No opaque daemons, no cron, no shell scripts. Indexing, materialized-view refresh, compaction, and pattern detection all surface in `system.jobs` with state, owner, and progress. Triggers include the WAL-delta-driven lazy indexer, `ON COMMIT` and `INCREMENTAL` materialized-view refresh, scheduled compaction, and the continuous pattern-detection engine. See [Jobs & Triggers](/docs/architecture/jobs-and-triggers). ## Design invariants that shape everything These are non-negotiable. A change that breaks any of them produces a different database: - **Bi-temporal everywhere** — every row carries four timestamps. No opt-out. - **The ontology is the schema** — you declare types, you never `CREATE TABLE`. - **Cell-level ACL is pushed into scans** — bounded under ~2.5× raw scan p50. - **Object-store native** — open formats (Parquet, Arrow IPC, content-addressed blobs); no proprietary on-disk format. A file written by RelataDB is readable by any Parquet reader. - **Hash-chained, tamper-evident commit manifests** — tampering with any historical file breaks the chain, detectable offline. ## Where to go next - [Data Flow](/docs/architecture/data-flow) — ingest (eager/lazy) and the query path, in depth. - [SmartIngest](/docs/architecture/smartingest) — identity detection and its honest boundary. - [Jobs & Triggers](/docs/architecture/jobs-and-triggers) — the typed-Job engine and what fires it. - [Architecture Overview](/docs/architecture/overview) — the reference version: planes, ADRs, invariants. - [Data Model](/docs/architecture/data-model) · [Storage Engine](/docs/architecture/storage) · [Query Engine](/docs/architecture/query-engine) ============================================================================== # Architecture URL: https://relatadb.dev/docs/architecture ============================================================================== # Architecture How the engine is put together — from a row arriving at a protocol door to a streamed query result. Read these top-down to get the full mental model, or jump to a single plane. ## In this section - [How It Works](/docs/architecture/how-it-works) — the journey of one row, from ingest to query (start here) - [Overview](/docs/architecture/overview) — the five planes, deployment profiles, and design invariants - [Data Flow](/docs/architecture/data-flow) — the eager/lazy ingest split and the read path - [SmartIngest](/docs/architecture/smartingest) — deterministic identity detection and the honest boundary with general NER - [Jobs & Triggers](/docs/architecture/jobs-and-triggers) — every background work item is a typed Job; what they are and what fires them - [Data Model](/docs/architecture/data-model) — ontology primitives, the bi-temporal row shape, and canonical identifier types - [Storage Engine](/docs/architecture/storage) — object-store native, WAL, Parquet segments, three-tier cache, tamper-evident manifests - [Query Engine](/docs/architecture/query-engine) — parse, plan, cost-based optimize, streaming execute - [Crate Map](/docs/architecture/crate-map) — the Rust workspace laid out, crate by crate See also: [Concepts](/docs/concepts) for the *why* behind each plane, and [Deployment](/docs/deployment) for running it in production. ============================================================================== # Jobs & Triggers URL: https://relatadb.dev/docs/architecture/jobs-and-triggers ============================================================================== # Jobs & Triggers RelataDB has a hard design invariant: **every background work item is a typed `Job`.** There are no opaque daemons, no hidden cron, and no shell scripts glued to the side. Anything that runs asynchronously — indexing, materialized-view refresh, compaction, pattern detection — is a first-class, observable Job. This page explains the job model, what triggers each kind of work, and how the workflow and detection engines fit in. It pairs with the operational guide at [Jobs, Workflows & Detection](/docs/guides/jobs-workflows). ## The model: typed jobs, not daemons A Job is a typed unit of background work. Every job surfaces in the `system.jobs` table with: - **state** — `pending` · `running` · `completed` · `failed` - **owner** — which role/worker is executing it - **progress** — observable, not a black box ```bash relata jobs # list all jobs relata jobs status indexer # check a specific job ``` ```bash curl -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \ http://localhost:9090/jobs # GET — list jobs curl -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \ http://localhost:9090/jobs/indexer # GET — one job's status ``` Because jobs are typed and resumable, a crashed worker is recoverable: the indexer, for example, resumes at the last committed WAL offset rather than re-scanning from zero. ## What the jobs are | Job | Owns | Triggered by | |---|---|---| | **Indexer** | Lazy ingest — materialized-view refresh, embedding generation, SmartIngest identity extraction, mention inference | WAL delta since the last committed offset | | **Materialized-view refresh** | Keeps MVs current | `ON COMMIT` (synchronous) or `INCREMENTAL` (lazy, WAL-driven) | | **Compaction** | Merges Parquet segments; reclaims space | Segment count / size thresholds | | **Pattern detection** | Continuously evaluates detection rules against new data | New commits that touch a rule's target type | ### Materialized-view refresh modes | Refresh mode | Trigger | Trade-off | |---|---|---| | `ON COMMIT` | Synchronous in the writer's commit | Higher write latency, zero query staleness | | `INCREMENTAL` | Lazy; indexer reads the WAL delta | Approximate aggregates, lower write cost | | `FULL` | Scheduled or manual | Consistent, expensive | This is the same mechanism behind the lazy side of [Data Flow](/docs/architecture/data-flow) — the indexer *is* a job, and the WAL offset is its resumable cursor. ## Detection rules The pattern-detection engine evaluates rules continuously against new data. Rules are defined in YAML (Sigma-compatible) or SQL, and each rule declares a trigger (a type + condition) and an action. ```bash curl -X POST http://localhost:9090/rules \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "large-transfer-flag", "trigger": { "type": "Transaction", "condition": "amount > 100000 AND currency = '\''USD'\''" }, "action": "flag" }' ``` ### Detection modes - **`live`** — alerts fire immediately when data matches. - **`shadow`** — alerts are logged but not surfaced (for validation). - **`disabled`** — rule is inactive. Rules must pass a precision/recall gate against a golden dataset before promotion from `shadow` to `live`. This is what keeps the detection engine trustworthy rather than noisy. ## Workflows (governance-aware DAGs) Workflows are directed acyclic graphs of steps that automate multi-stage analysis — for example: *detect* (a rule fires) → *enrich* (a lookup) → *report* (a templated deliverable). ```bash curl -X POST http://localhost:9090/workflows \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "fraud-investigation", "steps": [ {"name": "detect", "type": "rule", "rule": "large-transfer-flag"}, {"name": "enrich", "type": "lookup", "table": "sanctions_list"}, {"name": "report", "type": "report", "template": "sars-template"} ] }' ``` The critical property: **every workflow step inherits the tenant context, PURPOSE, and ACL of the triggering request.** Steps that would violate governance are blocked. Workflows cannot bypass policy — they run inside the same trust plane as a human-issued query. ## Why "everything is a typed Job" matters - **Observability** — there is one place (`system.jobs`) to see what the system is doing. No secret timers. - **Resumability** — jobs checkpoint (e.g. the indexer's WAL offset), so recovery is cheap and bounded. - **Governance** — because jobs and workflows run inside the trust plane, asynchronous work is subject to the same ACL, provenance, and audit rules as synchronous queries. - **Honesty about partial results** — when a job or a cluster read cannot complete fully, the system says so (partial reads return `206` with warnings) rather than silently truncating. ## See also - [Jobs, Workflows & Detection](/docs/guides/jobs-workflows) — the operational guide (full endpoint reference). - [Data Flow](/docs/architecture/data-flow) — where the indexer job sits in the eager/lazy split. - [Governance](/docs/concepts/governance) — the trust plane that jobs run inside. - [Observability](/docs/guides/observability) — monitoring jobs and alerts. ============================================================================== # Architecture Overview URL: https://relatadb.dev/docs/architecture/overview ============================================================================== # Architecture Overview RelataDB is a governed temporal knowledge database. It collapses the multi-database stack — relational, graph, vector, full-text search, and audit — into a single Rust binary. Policy, provenance, and bi-temporal history are in the query path, not bolted on after the fact. The trade-off is deliberate: the engineering cost of building all planes in-process is accepted in exchange for eliminating the consistency, governance, and operational seams that fragment a polyglot stack. ## Deployment profiles One binary (`relata`), three profiles. The profile changes capacity knobs and which subsystems boot. The wire protocol, on-disk format, SQL dialect, and ACL semantics are identical across all three. | Profile | Topology | Headroom | Status | |---|---|---|---| | `free` (`lite` alias removed — rejected at startup) | Embedded, single binary, local disk | ~1 B entities | Stable | | `server` | Single node + S3-compatible object store | ~10 B entities | Stable | | `cluster` | Coordinator + reader(s) + writer + indexer | 100 B – 1 T+ entities | Alpha — not recommended for production | Set with `RELATA_PROFILE=free|server|cluster`. ## The five planes Every feature belongs to exactly one plane. Cross-plane work goes through typed interfaces; no shared mutable globals exist between planes. | Plane | Owns | |---|---| | **Data plane** | Ontology, typed tables, link tables, IdentityIndex, bitmaps, vectors, full-text postings | | **Compute plane** | Planner, executor, cost-based optimizer, MV refresh, DAG job engine | | **Trust plane** | Cedar-inspired ABAC, classification labels, audit log, PROV-O provenance, hash-chained manifests | | **Identity plane** | `Identity` umbrella type, IdentityIndex MV, `RESOLVE_IDENTITY`, `IDENTITY_CLUSTER`, `SAME_IDENTITY` | | **Distribution plane** | Coordinator/reader/writer/indexer roles, write leases, hash partitioning, multi-region replication | ## Data flow ### Ingest Writes are split into an **eager** phase (synchronous, in the writer's commit critical path) and a **lazy** phase (asynchronous, re-runnable from the WAL). ```text SOURCE Connector Writer (eager) Storage Indexer (lazy) ────── ───────── ────────────── ─────── ────────────── CSV ─┐ validate + canonicalize ──► Parquet segment JSON ├─► normalize ────────► ACL + tenant stamp ──► CSR adjacency ──► IdentityIndex delta API ─┤ WAL append (fsync/async) ──► blob (SHA-256) embeddings (HNSW) Kafka ┘ row write (&self) ──► PROV-O assertion MV refresh └──► commit manifest SmartIngest ``` The eager phase does only declared-column work: parse, canonicalize, stamp principal/agency/classification, append WAL, write row. It does not run embeddings, refresh the IdentityIndex, or extract PostMentions. Writer p99 stays in single-digit milliseconds. The lazy phase is driven by the indexer role. It reads the WAL delta since the last committed offset and runs MV refresh, embedding generation, SmartIngest identity extraction, and PostMention inference. Every lazy step is re-runnable: a crashed indexer resumes at the last committed WAL offset. All lazy output is itself bi-temporal — late corrections are first-class, never in-place overwrites. ### Query ```text Client Result ────── ────── psql / gRPC / HTTP / MCP / Arrow Flight Arrow batches SPARQL + optional PROV-O columns compat: S3 / pgvector / ClickHouse + audit log entry / Neo4j / Redis / MongoDB / Bolt ▲ │ │ ▼ │ Wire auth ─► principal (OIDC JWT / mTLS cert) │ │ │ ▼ │ Coordinator ─► parse + PURPOSE check │ ─► ACL bitmap compilation │ ─► cost-based optimize + MV rewrite │ │ │ ▼ │ Execution (streaming, no full materialization) │ ─► columnar scan (Parquet + mutable tail) │ ─► CSR graph traversal (BFS / Pregel) │ ─► vector ANN (custom HNSW / DiskANN warm tier) │ ─► BM25 full-text (custom, not Tantivy) │ ─► bitmap intersection (ACL + filter, fused) │ │ │ ▼ │ Cache ─► L1 foyer (RAM/NVMe) → L2 ring → L3 S3 ───────┘ ``` ACL predicates are compiled into bitmap scan predicates before execution begins, never applied as a post-filter. ## Design invariants These are non-negotiable. A change that breaks any of them produces a different database. > **Bi-temporal everywhere.** Every row carries `valid_from`, `valid_to`, `system_from`, `system_to` as `i64` nanoseconds UTC. No opt-out. A table without bi-temporality is not a RelataDB table. > **Time is `i64` nanoseconds UTC.** No timezone is stored. Wire format is RFC 3339 with a `Z` suffix. Non-UTC offsets are rejected at parse time. > **`unsafe` is forbidden in library crates.** Every `lib.rs` compiles with `unsafe_code` forbidden or denied (`relata-query` and `relata-storage` use `#![deny]`; the rest use `#![forbid]`). The only workspace `unsafe` is the `relata` CLI binary's `libc::kill` FFI shim for the test harness. Vector hot loops rely on the compiler's auto-vectorizer rather than hand-written SIMD. > **`missing_docs` is denied.** Every public item — function, struct, enum variant, trait, module — has a doc comment. Builds fail otherwise. > **Locking via `parking_lot`.** `PlMutex` and `PlRwLock` type aliases are the only mutexes in the workspace. parking_lot does not poison on panic, eliminating the `recover_poison()` pattern that pollutes `std::sync::Mutex` codebases. > **Per-type interior locking.** Writes take `&self`, not `&mut self`. Each object type owns its own `RwLock`, so a write to `Profile` does not block a write to `Post`. > **Hash-chained commit manifests.** Every commit references the SHA-256 of the previous manifest. Tampering with any historical file breaks the chain, detectable offline via `relata doctor` or online via `GET /audit/count`. > **Object-store native.** Durable bytes are Parquet, Arrow IPC, and content-addressed blobs. Readable by any S3 client, any Parquet reader, any Arrow consumer. > **Single-writer-per-branch strong consistency.** At most one active writer lease per branch at a time, enforced by the coordinator. > **PURPOSE is optional.** Declared `PURPOSE ''` is recorded in audit and may drive policy, but the engine runs without one. > **Cell-level ACL is pushed into scans.** ACL decisions compile to bitmap predicates evaluated at vectorized loop level alongside the filter. Cost is bounded under 2.5× raw-scan p50. > **One ontology is the schema.** No `CREATE TABLE`. Declare `ObjectType`, `EventType`, `LinkType`, `ActionType`; the planner, storage layout, and SDKs derive from those declarations. > **Every background work item is a typed Job.** No opaque daemons. Indexing, MV refresh, compaction, and pattern detection all surface in `system.jobs` with state, owner, and progress. ## Workspace invariants - **Workspace resolver `"3"`**, **edition 2024**, **MSRV 1.85** - All workspace dependency versions are pinned in the root `[workspace.dependencies]` table. Use `dep.workspace = true`, not inline version strings. - `cargo deny check` enforces a license allowlist and bans unknown registries and git dependencies. ## See also - [Data Model](/docs/architecture/data-model) — ontology primitives, row shape, canonical types - [Storage Engine](/docs/architecture/storage) — WAL, Parquet segments, three-tier cache, manifests - [Query Engine](/docs/architecture/query-engine) — parse → plan → optimize → execute - [Crate Map](/docs/architecture/crate-map) — workspace crate graph ============================================================================== # Query Engine URL: https://relatadb.dev/docs/architecture/query-engine ============================================================================== # Query Engine The query engine lives in `relata-query`. It parses RelataDB SQL (ANSI SQL with extensions), plans with ACL pushdown, optimizes with a cost-based join-ordering and index-selection pass, and executes against columnar, graph, vector, and full-text stores. Execution is streaming end-to-end — no intermediate result is fully materialized in memory unless it spills to disk. ## Query lifecycle 1. **Parse** — extract statement shape, optional `PURPOSE` prefix, optional `AS OF` temporal clause, and any graph/vector/identity operators in the projection list. 2. **Purpose check** — when `PURPOSE ''` is declared, it is validated against the `PurposeRegistry` and recorded for audit. Purpose is optional; when absent, the engine runs normally. 3. **Plan** — resolve type references against the ontology, compile the principal's ACL into per-type bitmap predicates, generate a physical plan. The ACL bitmap is computed once per `(principal, type)` pair and cached. 4. **Optimize** — the cost-based optimizer picks join order, chooses scan strategy, picks hash-join build/probe sides, and applies MV rewrite when a materialized view covers the query. 5. **Execute** — streaming scan, hash join, aggregation, graph BFS, vector ANN, and BM25 run fused. The executor never holds the full result set in memory. 6. **Assemble** — result rows are packed into Arrow batches. If `WITH PROVENANCE` was requested, PROV-O columns are attached. An audit log entry is written. ## Statement shape ```sql [EXPLAIN POLICY] [PURPOSE ''] SELECT FROM [AS OF ''] [WHERE ] [ORDER BY ] [LIMIT n [AFTER '']] [WITH CACHE | WITH NOCACHE] [WITH PROVENANCE] ``` - **`EXPLAIN POLICY`** — prints the ACL decisions the planner made for the current principal without executing. Use to debug "why can't I see this row?" - **`PURPOSE ''`** — optional. Recorded in audit; may drive policy enforcement. The engine runs without it. - **`AS OF ''`** — selects the **valid-time** axis (what was true at that timestamp). For the system-time axis use `AS OF SYSTEM TIME ''`; `AS OF CURRENT` is sugar for both axes at `NOW`. See [Bi-temporal reference](/docs/reference/bitemporal). - **`WITH PROVENANCE`** — attaches the `prov` struct of each result row, so the caller can trace every fact to its source agent and activity. - **`WITH CACHE` / `WITH NOCACHE`** — overrides the default result-cache behavior for this query. - **`LIMIT n AFTER ''`** — keyset pagination. Cursor is opaque and stable across concurrent writes. ## ACL pushdown Access control compiles into scan-level bitmap predicates, not a post-filter. | Mode | p50 overhead vs. raw scan | Notes | |---|---|---| | Bitmap row filtering | ~1.0× | Precomputed bitmap intersected in the scan hot loop | | Conditional ACL (per-row predicate) | ~1.32× | Rule evaluates a predicate on each row | | Cell masking (redact / hash / partial) | ~2.6× | Transform applied to output cells post-scan | | Budget gate | Hard cap under 2.5× | Query rejected if estimated ACL cost exceeds the gate | The budget gate prevents a pathological ACL rule from OOM-ing the reader. Rejected queries return RFC 7807 errors with the offending rule ID. ## Cost-based optimizer The optimizer runs between planning and execution. It is not a stub or a roadmap item. It picks: - **Scan strategy** — sequential scan vs. index lookup (bloom filter, range index, IdentityIndex, vector ANN). - **Join order** — build/probe sides based on cardinality estimates from HLL sketches and zone maps. - **Join algorithm** — hash join (default), spill-to-disk hash join for large build sides, or nested-loop for non-equi joins. - **MV rewrite** — if a materialized view covers the query, rewrite to read from the MV instead of the base table. Cardinality estimates come from HLL (HyperLogLog, approximate `COUNT(DISTINCT)`) and Count-Min-Sketch sketches maintained per column. Query-level access: `HLL_COUNT(col)` and `CMS_ESTIMATE(col, value)`. ## DataFusion bridge Columnar aggregation uses Apache DataFusion as a compute bridge. The planner emits a DataFusion logical plan for aggregation-heavy queries; the result is streamed back through the same Arrow batch pipeline as the native executor. DataFusion is used for its vectorized aggregation kernel, not as a SQL parser or planner — those remain in `relata-query`. ## Execution — streaming, no full materialization | Mechanism | Effect | |---|---| | Streaming scan | Scan emits Arrow record batches to the downstream operator as they are produced | | Spilling hash join | Build side spills to local NVMe if it exceeds RAM; probe continues without blocking | | Streaming results | Rows are sent to the client as they are produced, not after the query finishes | | Spill-to-disk RAM release | Spilled pages are evicted from RAM, re-read on demand | ## Aggregation - **Columnar path** — used for full-table `GROUP BY` (no filter on the aggregate input). Vectorized over Arrow columns. - **Row path** — used when the aggregate input is filtered. Row-at-a-time evaluation. - **Supported**: `GROUP BY` + `COUNT(*)`, `SUM`, `MIN`, `MAX`, `AVG`, `COLLECT_LIST`, `HLL_COUNT`. - **`ORDER BY`** multi-column with `ASC`/`DESC` tiebreakers (`OrderBy.tiebreakers`, `ast.rs:1316`; applied at `executor.rs:2665`). ## Graph execution The graph plane is CSR (compressed sparse row) adjacency. Two production traversal engines: | Engine | Use case | |---|---| | Bidirectional BFS | Shortest path between two known nodes (`GRAPH_DIJKSTRA`, `GRAPH_SSSP`) | | Pregel-style iterative BFS | Multi-hop, multi-source traversal over paged CSR | > The PLL (pruned landmark labeling) distance index (`crates/relata-graph/src/pll.rs`) is wired into `GRAPH_SSSP` (when `algo => 'pll'`, `investigation_ops.rs:1938`) and into `GRAPH_DIJKSTRA` as an undirected reachability pre-check (`investigation_ops.rs:2200`). `PATHS_BETWEEN` and default single-pair shortest-path enumeration still use bidirectional BFS / DFS over paged CSR. 10+ SQL graph operators are implemented: `GRAPH_DIJKSTRA`, `GRAPH_LINK_PREDICT`, `GRAPH_SCC`, `GRAPH_CYCLES`, `GRAPH_SSSP`, `GRAPH_SPANNING_TREE`, `GRAPH_APSP`, `GRAPH_DIAMETER`, `GRAPH_SIMILARITY`, `GRAPH_NODE_METRIC`. The incremental degree index maintains per-node in/out degree counts as edges are written. `DEGREE(node)` is O(1) at query time — no CSR re-scan. ## Vector search The primary ANN index is a custom HNSW implementation in `vector.rs` — not a wrapper around an external library. | Component | Location | Role | |---|---|---| | HNSW graph | RAM | Navigation layer | | Vectors (hot) | RAM | Distance computation for visited nodes | | Vectors (warm) | Disk / object store | `PagedAnnIndex` / DiskANN warm tier | | Vectors (cold) | Object store | IVF cold bucket; spills from warm tier | Supported distance metrics: cosine (primary), L2, dot product. pgvector-compatible operators `<=>`, `<->`, `<#>` parse and route to the HNSW index. `search_filtered` applies ACL to vector results. Adaptive pre-filter prevents the recall cliff on selective ACLs by filtering the candidate set before HNSW traversal, rather than filtering ANN output post-hoc. ## Full-text search (BM25) The full-text engine is **custom BM25** — not Tantivy. It is purpose-built for integer posting lists and token interning to fuse cleanly with the vector path. | Feature | Detail | |---|---| | Token interning | Tokens → `u32` IDs at ingest; postings are integer lists | | Integer posting lists | `Vec` per token, SIMD-intersectable | | Q-gram index | Prefix and infix matching | | Trigram index | Suffix search and fuzzy matching | | Stemming | 15 hand-rolled (en/fr/de/es/pt/it/nl/sv/no/da/fi/hu/ro + Russian Cyrillic, Turkish, Arabic) + CJK pass-through (`search.rs:3189-3745`) | | Stop words | Per-language default lists | | Synonyms | User-declared synonym sets | | Highlighting | ``-wrapped match spans in output | | Faceted search | Per-facet counts returned alongside hits | | Ranking rules | Custom, user-declarable | `RELATA_SEARCH_PRESET` (`strict` / `balanced` / `lenient`, default `balanced`) controls edit-distance fuzzy expansion cluster-wide. ## Hybrid retrieval BM25 and vector similarity fuse via reciprocal-rank fusion (RRF). One `SELECT` handles both textual relevance and semantic similarity in one pass. ```sql SELECT id, text, score FROM Post WHERE BM25(text, 'financial fraud') OR VECTOR_SIMILAR(embedding, '[0.12, ...]', 0.8) ORDER BY RRF(BM25_score, vector_score, k = 60) LIMIT 20; ``` ## Materialized views ```sql CREATE MATERIALIZED VIEW ProfileEngagement ON COMMIT AS SELECT author_id, COUNT(*) AS post_count, SUM(likes) AS total_likes FROM Post GROUP BY author_id; ``` | Refresh mode | Trigger | Trade-off | |---|---|---| | `ON COMMIT` | Synchronous in the writer's commit | Higher write latency, zero query staleness | | `INCREMENTAL` | Lazy, indexer reads WAL delta; `IncrementalAggregate + HLL` | Approximate aggregates, lower write cost | | `FULL` | Scheduled or manual | Consistent, expensive | `RELATA_MV_MAX_ROWS` (default 1,000,000) caps cached rows in an incrementally-refreshed MV. Past the cap, the MV falls back to the base table — queries still run, just slower. ## Query result cache Result cache memoizes query results keyed by `(SQL text, principal, AS OF time)`. Invalidation is driven by commit manifests: any commit touching a referenced table invalidates that query's cache entries. Per-query override: `WITH CACHE` or `WITH NOCACHE`. **Pattern tracker + speculative prefetch** — observes query patterns and pre-fetches likely-next pages into L1 before the client asks. ## Watch subscriptions A query can be registered as a **watch subscription**. The engine re-evaluates the query on every commit that touches a referenced table and pushes the diff to subscribed clients over a long-lived gRPC or HTTP stream. Watch subscriptions use the same planner and ACL path as regular queries — the result set is always policy-filtered. ## Per-column bloom filters Each Identity-typed column has a bloom filter per Parquet row group. The planner consults the bloom before opening a row group. Point lookups on phone numbers or IBANs typically skip 90–99% of row groups before reading any data. ## See also - [Architecture Overview](/docs/architecture/overview) - [Data Model](/docs/architecture/data-model) - [SQL Reference](/docs/reference/sql) - [Hybrid Search](/docs/concepts/hybrid-search) ============================================================================== # SmartIngest URL: https://relatadb.dev/docs/architecture/smartingest ============================================================================== # SmartIngest SmartIngest is RelataDB's identity-resolution pipeline. It runs on the lazy side of ingest (see [Data Flow](/docs/architecture/data-flow)) and is what turns a pile of records from different sources into one connected identity graph — without you writing matching logic. This page explains its role, how it works, and — just as importantly — what it deliberately does **not** do. ## The role: "who is this, really?" The same entity shows up across your sources wearing different disguises: an email at signup, a phone number in a call log, an IBAN in a transfer, an account ID in billing. Without identity resolution, these are unrelated strings in unrelated tables. SmartIngest recognizes each identifier, validates it, and fuses records that share a validated identifier into **one cluster** — even when they arrived through different doors. The graph that powers `PATHS_BETWEEN`, PageRank, and community detection forms itself out of these fused identities. ## How it works: deterministic and checksum-gated SmartIngest is **not** a probabilistic matcher. It is a strict-first, checksum-gated detection pipeline. ```text raw text / cell value │ ▼ tokenize ──► per-token shape gate (regex) │ pass │ fail ──► skip ▼ format + checksum validate │ pass │ fail ──► skip ▼ DetectionHit (CanonicalKind + Identity) │ ▼ IdentityIndex (bloom-pruned lookup) │ ▼ identity cluster (fuse same entity) ``` Each canonical type has its own gate plus a validator. The pipeline tries types in order of certainty: 1. **Checksum-gated types** — Aadhaar (Verhoeff), PAN (Luhn), IBAN (mod-97), IMEI, VIN. A value that fails its checksum is skipped. This keeps false positives near zero on identifiers that carry integrity digits. 2. **Regex-only types** — emails, E.164 phone numbers, MAC addresses, URLs, domains. 3. **Heuristics** — last, with a string fallback. Because the inputs are checksum-validated, the fusion is high-precision: you trust the merge because you trust each link. ## What gets detected (and what does not) | Detected automatically (deterministic) | Not auto-detected — bring your own scorer | |---|---| | email · E.164 phone · MAC | arbitrary person/org/place names in prose | | IPv4 / IPv6 · URL · domain | intent · sentiment · stance | | Aadhaar · PAN · GSTIN · IFSC | "Alice met Bob at the hotel" | | IBAN · SWIFT · card (Luhn) | | | IMEI · IMSI · VIN · MMSI · IMO | | | SHA-1/256 hashes · UUID | | | UPI handle · social handles | | > **The honest boundary:** RelataDB does *identifier* extraction, not general named-entity recognition (NER). There is no in-tree spaCy/BERT/transformer. This is a deliberate design choice — identifiers are verifiable, names in prose are not. ### When you need general NER For names-in-prose, intent, or sentiment, you register an **external scorer** (configured via an accel endpoint). RelataDB takes that scorer's output and writes it back as governed, provenance-stamped typed assertions, then fuses it into the identity graph. RelataDB hosts the governed graph and identity layer; you bring the model for the fuzzy parts. ## From detection to a resolvable identity ```text Source A Source B ├─ alice@x.com ├─ alice@x.com └─ +14155550111 └─ device D7 │ │ └────────┬─────────────────┘ ▼ IdentityIndex fuse │ ▼ one identity cluster: Alice = {email, phone, device} ``` Three SQL operators consume the result: - `RESOLVE_IDENTITY(value)` — returns the full cluster for an identifier. - `IDENTITY_CLUSTER(value)` — expands a value into its cluster. - `SAME_IDENTITY(a, b)` — a real-time gate: are these two values the same entity? ## Where declared columns fit in SmartIngest also handles the **eager** path for declared columns. When you declare a property as `Identity` on your ontology, that column is validated and canonicalized on the write hot path (synchronously), and an entry is written into the universal `IdentityIndex`: ```text IdentityIndex: (CanonicalKind, bytes) → (object_id, source_table, source_column, observed_at) ``` One index, every observation, every source. Free-text fields are mined lazily; declared identity fields are typed eagerly. Either way they end up in the same resolvable graph. ## Why this design pays off - **Cross-source fusion is cheap.** The same phone number written by three feeds has one byte representation, so joins need no `LOWER()`/`TRIM()`/normalization in the predicate. - **You trust the merge.** Checksum validation makes each link high-precision; the cluster inherits that trust. - **Upgrades are safe.** Because detection runs lazily as materialized views, improving a detector triggers an MV refresh — not a source backfill. ## See also - [Data Flow](/docs/architecture/data-flow) — where SmartIngest sits in the eager/lazy split. - [Identity](/docs/concepts/identity) — the umbrella type and resolution operators in depth. - [Ingestion & SmartIngest](/docs/guides/ingestion) — the operational guide (CLI, HTTP, endpoints). - [How It Works](/docs/architecture/how-it-works) · [Data Model](/docs/architecture/data-model) ============================================================================== # Storage Engine URL: https://relatadb.dev/docs/architecture/storage ============================================================================== # Storage Engine One S3-compatible bucket (or local directory on the `free` profile). Open formats throughout — Parquet, Arrow IPC, JSON manifests, content-addressed blobs. Hash-chained for tamper evidence. No proprietary on-disk format. A file written by RelataDB is readable by any Parquet reader, any S3 client, any Arrow consumer. ## The bucket is the database All durable data lives in one S3-compatible bucket or, on `free`, under one local directory (`RELATA_DATA_DIR`). The layout is identical across profiles; only the backend changes. Supported object stores: AWS S3, Cloudflare R2, Google Cloud Storage, Azure Blob (via S3 compatibility), MinIO, local disk. ```bash RELATA_PROFILE=free RELATA_DATA_DIR=/var/lib/relata cargo run -p relata-cli -- serve RELATA_PROFILE=server AWS_ENDPOINT_URL=https://... \ AWS_BUCKET=relata-prod cargo run -p relata-cli -- serve ``` ## On-disk layout ```text relata-data/ ├── ontology/ # schema-as-code, git-branched, one file per type ├── tables/ # one directory per ObjectType / EventType │ ├── Profile/ │ │ ├── 2025-11-11.parquet │ │ ├── 2025-11-12.parquet │ │ └── _bloom/ # per-column bloom filters │ └── CallEvent/ ├── graph/ # CSR segments + incremental degree index, one dir per LinkType ├── vectors/ # HNSW graph + DiskANN warm tier, one dir per (type, column) │ ├── Post.embedding/ │ └── MediaContent.clip/ ├── fulltext/ # custom BM25 inverted index (integer posting lists) ├── identity_index/ # universal lookup MV: (CanonicalKind, bytes) → observations ├── provenance/ # PROV-O assertions, content-addressed by assertion hash ├── audit/ # append-only, hash-chained log ├── blobs/ # content-addressed by SHA-256 (media, attachments) └── manifests/ # branch HEAD pointer + commit chain, one JSON per commit ``` ## In-memory store and per-type interior locking The primary read path is an in-memory bi-temporal store built in `relata-storage`. Writes take `&self`, not `&mut self`. Each object type owns its own `RwLock`, implemented with `parking_lot::RwLock` (`PlRwLock` type alias). A write to `Profile` does not block a read or write to `Post`. parking_lot does not poison on panic, which eliminates the `recover_poison()` pattern. ## Write-Ahead Log (WAL) Every write goes through the WAL before it is applied in memory. The WAL is flushed before the write is acknowledged to the client. In-memory state is a cache of the WAL; on restart, the WAL is the source of truth. | Durability level | fsync timing | Process crash | Power loss | |---|---|---|---| | `async` (default) | Background flusher, ~10ms cadence | No loss | ≤10ms loss | | `sync` | fsync before ack | No loss | No loss | The durability level is set per-write via the `X-Relata-Durability` request header, or forced cluster-wide with `RELATA_WAL_SYNC=always` (default is `interval`). A bulk loader uses `async`; a payment record uses `sync`. Same table, same writer, different durability per row. The WAL is hash-chained: each WAL entry references the hash of the previous entry. A replay that produces a hash mismatch at any entry signals corruption or tampering. ## Parquet segments Each `tables//` directory holds Parquet data files. The default is one file per UTC day per type. | Property | Value | Why | |---|---|---| | Row group size | 128 MB | Balances scan throughput vs. zone-map granularity | | Compression | Zstd level 3 | Best speed/ratio for typed columns | | Sort key | `(valid_from, cluster_key)` | Co-locates related rows in one row group | | Bloom filters | Identity-typed columns | Skip row groups that cannot contain the join key | | Zone maps | `valid_from` column | Skip row groups outside the temporal predicate | | Page index | Sorted columns | Page-level skipping within a row group | `RELATA_FLUSH_SEGMENT_MAX_ROWS` (default `250000`) sets the max rows per flushed segment. A larger flush delta is split into `ceil(delta/N)` segments. `0` = unbounded, single segment per flush (legacy behavior). The **mutable tail** is a small WAL-spilled segment kept in RAM and on local NVMe. On query, it is merged with the Parquet main store via copy-on-write. This is what lets writes land without waiting for a Parquet compaction cycle. ## Three-tier cache | Tier | Technology | Latency | Scope | |---|---|---|---| | **L1** | foyer (in-process, NVMe-backed, `relata-cache`) | sub-ms | Single reader node | | **L2** | Consistent-hash ring across reader nodes | ~1 ms | N=2 hot replicas | | **L3** | S3 / R2 / GCS / MinIO | 20–100 ms | Durable, all data | - L1 admission uses S3-FIFO (scan-resistant — a full-table scan does not evict hot keys). - L2 fanout is bounded at N=2; reads beyond the hot working set go directly to L3. - Writes invalidate L1 and L2 on the owning shard synchronously, then propagate to replica shards asynchronously. ## Vector storage — three tiers The vector index has three tiers, each with different RAM/disk trade-offs: | Tier | Structure | Location | Role | |---|---|---|---| | Hot (RAM) | Custom HNSW (`vector.rs`) | RAM | Full graph + vectors resident; fastest recall | | Warm (DiskANN) | `PagedAnnIndex` — HNSW-backed with object-store segments (`vector_diskann.rs`) | Disk / object store | RAM-resident graph, disk-resident vectors | | Cold (IVF) | IVF bucket staging area | Object store | Overflow; spills when `RELATA_VECTOR_COLD_RESIDENT_MAX` is exceeded | `RELATA_VECTOR_COLD_RESIDENT_MAX` (default `100000`) is the soft cap on RAM-resident vectors in an IVF cold bucket's staging area before it spills to the `PagedAnnIndex`. Only active when an object store is configured. `RELATA_DISKANN_MAX_RESIDENT` (soft cap on RAM-resident vectors before the index warns to shard or restart; `0`/unset = unbounded). The HNSW implementation is custom — it is not a wrapper around an external library. Supported distance metrics: cosine (primary), L2, dot product. ## Per-column bloom filters Each Identity-typed column has a bloom filter per Parquet row group. The query planner consults these before opening a row group. A point lookup on a phone number or IBAN typically skips 90–99% of row groups before reading a byte of actual data. ## Summary index A per-type summary index maintains `COUNT` and `SUM` aggregates incrementally as rows are written. `COUNT(*)` and `SUM(col)` on an unfiltered type resolve in O(1) without scanning any Parquet data. ## Tamper-evident commit manifests Every commit produces a manifest JSON. Manifests form an append-only hash chain — each one references the SHA-256 of the previous manifest. ```json { "commit_id": "01HQ8X3F9...", "branch": "main", "prev_commit_hash": "sha256:9f86d081...", "files_added": [ "tables/Profile/2025-11-11.parquet", "audit/2025-11-11T00:00:01Z.ndjson" ], "manifest_root": "sha256:e3b0c442...", "writer_principal": "oidc:alice@example.com", "system_from": 1762828800000000000, "signatures": [ { "kid": "writer-key-1", "alg": "Ed25519", "sig": "base64:..." } ] } ``` - Verified offline by `relata doctor` or online via `GET /audit/count` (returns `chain_valid: true|false`). - A missing manifest file is itself a tamper signal — the chain references a hash that no longer resolves. - Signatures are optional but recommended on `cluster` profile writers. ## RAM budget and disk spill `RELATA_STORE_MAX_RAM_MB` sets the row-store RAM budget before spill-to-disk: - **`server` / `cluster`** — defaults to **1024 MB**. Small datasets never spill; the cap is byte-aware. - **`free`** — **unbounded**, for local dev. Explicit values override the profile default. Every in-memory structure has a paged counterpart for when the budget is exceeded: | Structure | Paged backend | |---|---| | Authoritative rows | Spill-to-disk mutable tail on local NVMe | | Full-text postings | `DiskIndexSource` | | Vector index | `PagedAnnIndex` + IVF cold tier | | Graph adjacency | `PagedCsrGraph` | | Identity index | Live-paged, incremental spill | ## Cold restart and lazy loading | Phase | Duration (10M rows, single node) | |---|---| | WAL + Parquet flush (graceful shutdown) | ~15 s | | Cold-load from Parquet (eager) | ~55 s | | Cold-load from manifest catalog only (lazy) | under 1 s | `RELATA_LAZY_RESTART=true` loads only the manifest catalog at startup — O(manifest), not O(rows). Segments hydrate on demand when first queried. This is the default on `server` and `cluster` profiles. `free` remains eager. `RELATA_HYDRATE_RECENT_SEGMENTS=N` (default `0`) hydrates the newest N segments into RAM at startup, trading startup time for warm-cache latency on the most recent data. ## See also - [Architecture Overview](/docs/architecture/overview) - [Data Model](/docs/architecture/data-model) - [Backup & Restore](/docs/guides/backup-restore) - [Environment Variables](/docs/reference/env-vars) ============================================================================== # Cognee vs RelataDB URL: https://relatadb.dev/docs/compare/cognee ============================================================================== # Cognee vs RelataDB > **TL;DR** – Cognee is an open-source **data-to-knowledge-graph pipeline** for LLMs (ingest → parse → graph + vectors, Pydantic-shaped datapoints). RelataDB is a **governed temporal knowledge database**. Both build knowledge structures for retrieval; they differ on whether the result must be **governed, bi-temporal, and queryable through your existing clients**. ## What each one is **Cognee** is an ETL-style framework: you feed it documents/databases, it runs a deterministic pipeline (the ECL — Extract, Cognify, Load), produces a knowledge graph plus vector embeddings defined as Pydantic `DataPoint`s, and lets you retrieve against it. Its center of gravity is **ingestion-to-graph**, framework-first. **RelataDB** is a database: you declare an ontology (`ObjectType`, `EventType`, `LinkType`, `ActionType`) and ingest rows; the engine standardizes identities (76 canonical kinds), links same-entity records into a graph deterministically, time-stamps every row twice, and notarizes every fact. Retrieval is SQL/Cypher/GQL/SPARQL/MCP, governed by cell-level ACL. ## Feature matrix | | **Cognee** | **RelataDB** | |---|---|---| | Center of gravity | Data-to-graph pipeline (framework) | Governed temporal database (engine) | | Schema | Pydantic `DataPoint` definitions | Ontology-declared types; planner/storage derive from them | | Graph construction | Pipeline stage extracts entities/relations | Automatic from validated identifiers + co-occurrence; you can also declare links | | Identity resolution | LLM/pipeline-driven | Deterministic checksum parsers (76 canonical kinds) | | Bi-temporal history | No | Yes — on every row (`AS OF` valid + `AS OF SYSTEM TIME`) | | Provenance / audit | Weak / none | Hash-chained, tamper-evident per fact | | Access control | App-enforced | Cell-level ACL in the scan predicate; per-tenant encryption | | Query languages | Python retrieval API | SQL, Cypher, GQL, SPARQL, MCP, plus 8 wire-protocol doors | | Talk to existing clients? | Cognee SDK | Postgres / S3 / Mongo / Redis / ClickHouse / Neo4j-Bolt / Flight | | Self-host | Yes | Yes (single binary) | ## When to pick Cognee - Your core problem is **"turn my documents/tables into a graph + vectors for RAG"** and you like the Pydantic-datapoint pipeline model. - You want a **framework** you embed in an existing app, not a database to operate. - The output doesn't need to be governed, bi-temporal, or court-grade reproducible. ## When to pick RelataDB - The knowledge must be a **defensible system of record** — auditable, reproducible, access-controlled, recoverable to any past state. - You want the graph to **form itself** from standardized identities, not be hand-built per pipeline run. - You want one engine behind the clients your stack already speaks, instead of a pipeline + Postgres + vector DB + graph DB. ## FAQ **Is RelataDB a Cognee replacement?** They overlap on "build a graph for retrieval." If you specifically want Cognee's Pydantic-pipeline developer model, Cognee is good at that. If you want the graph to be a governed, bi-temporal, multi-protocol database, RelataDB is the fit. **Does RelataDB do ETL?** Yes — `relata import --from postgres|neo4j|mongo` migrates existing data, and SmartIngest standardizes identifiers at ingest. But the result is a database, not a pipeline artifact. **Which is more "deterministic"?** Cognee markets determinism (the ECL pipeline is repeatable). RelataDB is deterministic at the **storage** level — same ingest always produces the same standardized, timestamped, hash-notarized rows, independent of any model. See also: [RelataDB vs the field](/docs/compare) and [SmartIngest](/docs/architecture/smartingest). ============================================================================== # RelataDB vs the field URL: https://relatadb.dev/docs/compare ============================================================================== # RelataDB vs the field If you are evaluating RelataDB against another AI-agent memory or knowledge tool, this is the shortlist. Each page below is an honest, feature-by-feature comparison with a clear "when to pick which" — no strawmen. All four alternatives below are good tools. RelataDB's wedge is specific: **governance, provenance, bi-temporal history, and deterministic identity** for workloads where a hallucinated or unexplainable memory is unacceptable (regulated, legal, financial, intelligence, enterprise). If you just need a fast drop-in memory for a chatbot, the lighter tools are often the right call. ## The comparisons | Compare | Best for search intent | |---|---| | **[Mem0 vs RelataDB](/docs/compare/mem0)** | "mem0 alternative", "mem0 vs relatadb", "governed mem0" | | **[Cognee vs RelataDB](/docs/compare/cognee)** | "cognee alternative", "cognee vs relatadb", "cognee vs mem0" | | **[Zep vs RelataDB](/docs/compare/zep)** | "zep alternative", "zep vs relatadb", "Graphiti vs relatadb" | | **[Letta (MemGPT) vs RelataDB](/docs/compare/letta)** | "letta alternative", "memgpt vs relatadb", "agent state vs memory" | ## The short version | | Mem0 / Cognee / Zep / Letta | **RelataDB** | |---|---|---| | Primary job | Give an LLM agent a memory layer | Governed temporal knowledge database | | Entity extraction | LLM-driven (lossy, run-to-run variance) | Deterministic checksum parsers (76 canonical kinds) | | History | Latest-wins (Zep has fact-level temporal validity) | Bi-temporal on **every** row — `AS OF` time-travel + `AS OF SYSTEM TIME` | | Provenance / audit | Weak or none | Tamper-evident, hash-chained per fact; court-grade replay | | Access control | Usually app-enforced | Cell-level ACL compiled into the scan predicate | | Talk to existing clients? | Their SDK only | Postgres/pgvector, S3, Mongo, Redis, ClickHouse, Neo4j/Bolt, Arrow Flight — keep your driver | | Topology | A layer over Postgres + a vector DB | One engine: relational + graph + vector + full-text | ## When to pick which - **Mem0** — you want the simplest path to "my chatbot remembers the user", hosted cloud, happy with LLM-extracted facts. - **Cognee** — your core need is data-to-knowledge-graph ETL for LLM retrieval, Pydantic-shaped datapoints. - **Zep / Graphiti** — you specifically want a temporal knowledge graph for facts that change over time, and like the Graphiti model. - **Letta (MemGPT)** — you want the agent-as-OS / memory-blocks model and self-hosted agent state. - **RelataDB** — the memories must be **defensible**: traceable, reproducible, access-controlled, and recoverable to any past state — and you want to keep your existing clients. See also: [Relata vs Others](/docs/concepts/relata-vs-others) (the broader map vs Postgres / Neo4j / Pinecone / lakehouse) and [Agent Memory](/docs/concepts/agent-memory). ============================================================================== # Letta (MemGPT) vs RelataDB URL: https://relatadb.dev/docs/compare/letta ============================================================================== # Letta (MemGPT) vs RelataDB > **TL;DR** – Letta (formerly MemGPT) gives an LLM agent an **operating-system-style memory hierarchy** (core / archival / recall blocks) and a self-hosted agent runtime. RelataDB is the **governed knowledge store** the agent reads and writes. They answer different questions: "how does the agent hold state?" vs "is the fact provable and governed?" ## What each one is **Letta** (the company/project behind **MemGPT**) treats the agent like an OS process: it has memory blocks it pages in and out of a context window, manages its own state, and runs as a self-hosted agent server. The emphasis is the **agent runtime** and a memory model inspired by operating-system virtual memory. **RelataDB** is not an agent runtime — it is the **database** the agent (any agent, including a Letta one) stores into. Its job is that the stored knowledge is standardized, bi-temporal, provenance-bearing, access-controlled, and recoverable to any past state. ## Feature matrix | | **Letta (MemGPT)** | **RelataDB** | |---|---|---| | Primary job | Agent runtime + memory hierarchy | Governed temporal knowledge database | | Memory model | OS-style blocks (core / archival / recall), paged into context | 10 cognitive verbs; rows are bi-temporal, provenance-bearing | | Is it an agent runtime? | Yes — runs the agent | No — storage/memory layer the agent calls | | Identity resolution | Not the focus | Deterministic checksum parsers (76 canonical kinds) | | Bi-temporal history | No | Yes — on every row | | Provenance / audit | Limited | Hash-chained, tamper-evident per fact | | Access control | App/runtime-enforced | Cell-level ACL in the scan predicate; per-tenant encryption | | Query languages | Letta SDK / API | SQL, Cypher, GQL, SPARQL, MCP | | Talk to existing clients? | Letta SDK | Postgres / S3 / Mongo / Redis / ClickHouse / Neo4j-Bolt / Flight | ## When to pick Letta - You want the **agent-as-OS** model — the agent manages its own memory blocks and runs as a server. - Your problem is **how the agent holds and pages state across long horizons**, not whether each fact is court-grade provable. ## When to pick RelataDB - The knowledge the agent reads/writes must be a **governed system of record** — auditable, reproducible, access-controlled, multi-tenant. - You already have an agent runtime (LangGraph, CrewAI, AutoGen, your own) and need the **memory/knowledge layer** underneath it. Letta is a strong choice for the agent runtime layer. RelataDB is a strong choice for the governed memory layer underneath. A Letta agent can persist its archival memory into RelataDB so that what it "remembers" is provable and recoverable. ## FAQ **Is RelataDB a Letta/MemGPT replacement?** No — they're different layers. Letta replaces "how does my agent hold state." RelataDB replaces "where does the governed, provable knowledge live." **MemGPT vs RelataDB for long-term memory?** MemGPT's contribution is the memory-hierarchy/paging model for long-running agents. RelataDB's contribution is that every stored memory is bi-temporal, identity-resolved, provenance-bearing, and access-controlled. The two compose. **Does RelataDB run agents?** No. Relata is the memory/knowledge layer; it integrates with LangChain, LlamaIndex, CrewAI, AutoGen/AG2, Pydantic-AI, smolagents, and LangGraph via adapters. See [Agent Memory](/docs/concepts/agent-memory). See also: [RelataDB vs the field](/docs/compare) and [Agent Memory](/docs/concepts/agent-memory). ============================================================================== # Mem0 vs RelataDB URL: https://relatadb.dev/docs/compare/mem0 ============================================================================== # Mem0 vs RelataDB > **TL;DR** — Mem0 is the simplest way to give a chatbot persistent, relevant memory. RelataDB is a governed temporal knowledge database. They overlap on "store + retrieve memories," and diverge on whether a memory needs to be **provable, access-controlled, and recoverable to any past state**. ## What each one is **Mem0** is an open-source (and hosted) **memory layer for LLM agents**. It extracts facts from conversations, scores them, stores them over a vector database (Qdrant / Chroma / Pinecone / pgvector), and retrieves the relevant ones at inference time. You add it to an existing agent with a few lines; the value is "the agent remembers the user." **RelataDB** is a **governed temporal knowledge database** — one Rust engine that holds relational rows, a graph, vectors, full-text search, and an audit chain. Agent memory is one surface (`remember · recall · recognize · justify · consolidate · forget · associate · episodes · resolve · summarise`); the others are governed storage, identity resolution, provenance, and bi-temporal history. ## Feature matrix | | **Mem0** | **RelataDB** | |---|---|---| | Primary job | Agent memory layer | Governed temporal knowledge database | | Memory verbs | add / search / get / update / delete | 10 cognitive verbs over MCP + HTTP | | How entities/facts are extracted | LLM-driven (model-dependent, run-to-run variance) | Deterministic checksum parsers (76 canonical identifier kinds); LLM extraction is optional and on top | | Provenance — can you prove where a memory came from? | Weak / none | Tamper-evident, hash-chained per memory + every write | | Time travel — "what did we know on Tuesday?" | No | Yes — bi-temporal `AS OF` (valid) + `AS OF SYSTEM TIME` (system) on every row | | Reproducible recall | "Depends on the model" | Court-grade replayable (same query → same result, forever) | | Access control | App-enforced | Cell-level ACL compiled into the scan predicate; per-tenant encryption | | Multi-tenancy | Usually single-tenant | Per-tenant isolation + cell-level ACL + tenant-scoped memory | | Storage model | A vector DB under the hood | Relational + graph + vector + full-text + audit in one engine | | Talk to existing clients? | Mem0 SDK | Postgres/pgvector, S3, Mongo, Redis, ClickHouse, Neo4j/Bolt, Arrow Flight — keep your driver | | Self-host | Yes | Yes (single binary, three profiles) | | Hosted cloud | Yes | License-based self-host | ## When to pick Mem0 - Your agent needs **memory**, fast, and the data is **low-stakes** (consumer chatbots, personalization, assistants where a wrong recollection is a minor annoyance). - You want a **hosted** memory service and don't want to operate a database. - "Good enough, LLM-extracted facts" is acceptable — you don't have to defend a memory in an audit. ## When to pick RelataDB - Memories must be **defensible** — regulated, legal, medical, financial, intelligence, enterprise knowledge work where "the model guessed" is not an acceptable provenance. - You need **time travel**: what did the agent know at the moment it made a decision? - You need **governance**: cell-level ACL, multi-tenant isolation, audit chain. - You are tired of bolting Postgres + Neo4j + a vector DB + a memory layer together and want one engine that speaks the wire protocols your stack already uses. Mem0 can sit in front of RelataDB for the chatbot ergonomics while RelataDB is the governed, provable store underneath. The common mistake is treating Mem0 as the system of record for sensitive facts. ## Migrating from Mem0 RelataDB's `Memory` client is the drop-in surface — same shape (add / search / forget), but every memory lands as a bi-temporal, provenance-bearing, ACL-checked row. The 10-verb cognitive surface plus `AS OF` / `WITH PROVENANCE` are the new capabilities you get for free. ```python from relata import Memory with Memory("http://localhost:9090", purpose="agent-notes") as m: mid = m.add("Alice prefers dark mode") for hit in m.search("ui preferences", top_k=5): print(hit["content"]) ``` See [Agent Memory](/docs/concepts/agent-memory) and the [Python SDK](/docs/sdks/python) quickstart. ## FAQ **Is RelataDB a Mem0 replacement?** For governed, auditable, multi-source memory — yes. For "give my consumer chatbot a personality in 5 minutes" — Mem0 is lighter and that's fine. **Does RelataDB use an LLM to extract memories?** Identity extraction is deterministic by default (checksum parsers, 76 canonical kinds). LLM-based extraction is available on top, but the system of record is the deterministic, replayable store — not the model's output. **Can I keep my existing client?** Yes — if you speak Postgres, Mongo, Redis, S3, ClickHouse, Neo4j/Bolt, or Arrow Flight, point it at RelataDB. Memory verbs are an additional MCP/HTTP surface on top. **Which is faster?** Mem0 is a thin layer over a vector DB, so for pure similarity recall it's hard to beat on simplicity. RelataDB's value isn't raw recall latency — it's that the recalled fact is correct, governed, and explainable. See also: [RelataDB vs the field](/docs/compare) and [Agent Memory](/docs/concepts/agent-memory). ============================================================================== # Zep vs RelataDB URL: https://relatadb.dev/docs/compare/zep ============================================================================== # Zep vs RelataDB > **TL;DR** – Zep (with **Graphiti**) is the closest analog to RelataDB in this list: both track **fact-level temporal validity** rather than latest-wins. They differ on **governance, provenance, deterministic identity, scale, and whether you can keep your existing clients**. ## What each one is **Zep** is a long-term memory service for AI assistants. Its **Graphiti** engine maintains a temporal knowledge graph: facts have validity intervals, are extracted from conversations, and can be retired/contradicted over time. It is the one alternative here that takes **time** seriously at the fact level. **RelataDB** is a governed temporal knowledge database. **Bi-temporality is on every row of every type** — not just facts, but identities, links, events, and audit entries carry `[valid_from, valid_to)` and `[system_from, system_to)`. Time travel (`AS OF`, `AS OF SYSTEM TIME`) is a first-class query mode across the whole store, not a fact-store feature. ## Feature matrix | | **Zep / Graphiti** | **RelataDB** | |---|---|---| | Temporal model | Fact-level validity intervals | Bi-temporal on **every** row (valid + system time) | | "What did we know on Tuesday?" | Partial (fact store) | Yes, across the entire store (`AS OF SYSTEM TIME`) | | Identity resolution | LLM/graph-extracted | Deterministic checksum parsers (76 canonical kinds) | | Provenance / audit chain | Limited | Hash-chained, tamper-evident per write; court-grade replay | | Access control | App-enforced | Cell-level ACL in the scan predicate; per-tenant encryption | | Scale target | Assistant-scale memory | 1B+ entities → 1T+ edges (cluster profile) | | Query languages | Zep SDK / Cypher-ish | SQL, Cypher, GQL, SPARQL, MCP | | Talk to existing clients? | Zep SDK | Postgres / S3 / Mongo / Redis / ClickHouse / Neo4j-Bolt / Flight | | Self-host | Yes | Yes (single binary, three profiles) | | Hosted cloud | Yes | License-based self-host | ## When to pick Zep - You want **assistant memory with fact-level temporality** and the Graphiti mental model fits your team. - You want a **hosted** memory service and don't need cell-level governance or a multi-protocol database. ## When to pick RelataDB - Temporality must apply to **everything** — identities, links, events, audit — not just extracted facts. - You need **governance**: cell-level ACL, multi-tenant isolation, a tamper-evident audit chain, reproducible recall. - You're past assistant-scale into **enterprise/intelligence** territory (1B+ entities, multi-region, multi-tenant). - You want to keep your existing Postgres/Mongo/Redis/Neo4j clients. Of the four tools compared here, Zep is the one whose philosophy (time-aware facts) is closest to RelataDB's. The wedge is governance + provenance + deterministic identity + the fact that bi-temporality is the storage model, not a feature of a fact extractor. ## FAQ **Is RelataDB a Zep replacement?** For governed, enterprise-scale, time-aware knowledge — yes. For "temporal memory for my assistant, hosted, done" — Zep is lighter. **What does "bi-temporal" add over Zep's fact validity?** Two time axes, not one. Valid time = when the fact was true in the world. System time = when the database recorded it. That distinction is what lets you answer "given what we knew on Tuesday, what should we have reported for Monday?" — the question auditors actually ask. **Graphiti vs RelataDB?** Graphiti is Zep's temporal knowledge graph engine. The comparison above is the same — Graphiti brings fact-level temporality; RelataDB brings bi-temporality + governance + provenance + deterministic identity + multi-protocol access. See also: [RelataDB vs the field](/docs/compare) and [Bi-Temporal Model](/docs/concepts/bitemporal). ============================================================================== # Compatibility & Doors — bring your existing client URL: https://relatadb.dev/docs/compatibility ============================================================================== # Compatibility & Doors — bring your existing client RelataDB is a **server** that speaks your existing database's wire protocol. Your **existing clients connect to Relata**; Relata does *not* connect to your existing databases. You keep your driver, your ORM, your GUI tool — you just repoint the host/port and use your bearer token as the password. One binary speaks **13 wire surfaces** from one governed store: **8 compatibility doors** (MongoDB, Postgres + pgvector, Redis, Neo4j HTTP, Bolt, ClickHouse HTTP, ClickHouse native, S3) plus **5 native protocols** (HTTP REST, gRPC, Arrow Flight, MCP, SPARQL). Write through any door, read back through any other — ACL, audit, and provenance apply uniformly. > **New here?** If you already run MongoDB / Postgres / Redis / Neo4j / ClickHouse / an S3 client, you can adopt Relata **without rewriting your app**. This page is the whole "how" — port table, connection strings, enable flags, and a 3-step quickstart per protocol. The deep wire-protocol reference lives at [Protocol Compatibility](/docs/reference/protocols). ## The one-line architecture ``` your existing client ──► RelataDB (speaks the wire protocol) ──► one governed bi-temporal store (Mongo / psql / (ACL + audit + provenance redis-cli / Neo4j / on every write/read, ClickHouse / boto3 / uniformly across doors) pyarrow Flight) ``` Three things Relata is **not**, and people often guess one of them: 1. **Not a sync from your existing DB.** Relata doesn't poll your Mongo/Postgres. Your clients speak to Relata instead. (For a one-time migration *out of* an existing DB into Relata, use [`relata import --from`](/docs/guides/connectors#migration-connectors-relata-import---from) — Postgres is live, Mongo/Neo4j/ClickHouse are honest stubs today.) 2. **Not an ETL engine.** Relata's signed binary has zero network dependencies. Polling/OAuth/vendor-SDK fetching lives in the external [`datagrep`](https://github.com/relatadb/datagrep) ETL tool that pushes governed rows in. See [Connectors & Extensions](/docs/guides/connectors). 3. **Not a query federation layer.** Relata doesn't forward your Mongo query to a hidden Mongo — it *is* the Mongo. Same for the other doors. ## Three adoption patterns — pick one | Pattern | What you do | When | |---|---|---| | **Drop-in (door)** | Repoint your existing client at Relata's port. Keep your code. | You have a working app and want governance/provenance/history without a rewrite. | | **Direct (SDK)** | Use the Relata SDK (Python / TypeScript / Go) for first-class SQL, graph, memory, and identity verbs. | Greenfield, or you want the full surface (MCP, Memory, hybrid search, AML ops). | | **Mixed** | Door for legacy writes (your Mongo app), SDK/SQL for new analytics reads. | Migration-in-place: existing app keeps writing via the door, new features read via SQL/SDK. | The **drop-in door** and the **SDK** read and write the *same* governed rows. Pick per-workflow, not per-project. ## The full port + credential table Every door shares **one credential**: your `RELATA_BEARER_TOKEN`. It's the password for every protocol below (for S3 SigV4, the secret key defaults to the same token unless you set `RELATA_S3_SECRET_KEY`). | Protocol | Enable flag | Port var (default) | Bind var (default) | Default state | |---|---|---|---|---| | **MongoDB wire** | `RELATA_MONGO_ENABLE` | `RELATA_MONGO_PORT` (`27017`) | `RELATA_MONGO_BIND` (`127.0.0.1`) | auto-enable on token | | **Postgres + pgvector** | *(token required)* | `RELATA_PG_PORT` (`5433`) | `RELATA_PG_BIND` (`127.0.0.1`) | fail-closed without token | | **Redis RESP** | `RELATA_REDIS_ENABLE` | `RELATA_REDIS_PORT` (`6379`) | `RELATA_REDIS_BIND` (`127.0.0.1`) | auto-enable on token | | **Neo4j HTTP (Cypher)** | `RELATA_NEO4J_ENABLE` | `RELATA_NEO4J_PORT` (`7474`) | `RELATA_NEO4J_BIND` (`127.0.0.1`) | auto-enable on token | | **Neo4j Bolt** | `RELATA_BOLT_ENABLE` | `RELATA_BOLT_PORT` (`7687`) | `RELATA_BOLT_BIND` (`127.0.0.1`) | auto-enable on token | | **ClickHouse HTTP** | `RELATA_CLICKHOUSE_ENABLE` | `RELATA_CLICKHOUSE_PORT` (`8123`) | `RELATA_CLICKHOUSE_BIND` (`127.0.0.1`) | auto-enable on token, **read-only** | | **ClickHouse native TCP** | `RELATA_CLICKHOUSE_NATIVE_ENABLE` | `RELATA_CH_NATIVE_PORT` (`9000`) | `RELATA_CH_NATIVE_BIND` (`127.0.0.1`) | auto-enable on token, **read-only** | | **S3 (boto3 / aws CLI / rclone / MinIO)** | `RELATA_S3_ENABLE` | `RELATA_S3_PORT` (`9191`) | `RELATA_S3_BIND` (`127.0.0.1`) | auto-enable on token | | Arrow Flight | `RELATA_FLIGHT_ENABLE` | `RELATA_FLIGHT_PORT` (`8815`) | `RELATA_FLIGHT_BIND` (`127.0.0.1`) | opt-in | | HTTP REST | *(always on)* | `RELATA_PORT` (`9090`) | `RELATA_HTTP_BIND` (profile-scoped) | always on | | gRPC | *(always on)* | `RELATA_GRPC_PORT` (`50051`) | `RELATA_GRPC_BIND` (profile-scoped) | always on | | MCP | *(always on)* | `/mcp` on HTTP | — | always on | | SPARQL | *(always on)* | `/sparql` on HTTP | — | always on | **Door enable rules (all profiles, no license gating):** - Off by default. Doors **auto-enable when `RELATA_BEARER_TOKEN` is set** (an unauthenticated port is never auto-exposed). - `RELATA__ENABLE=true` forces a door on; `=false` forces it off (overrides auto-enable). - pgwire is **fail-closed**: refuses to start without a token, period. - On `server`/`cluster`, `RELATA__ENABLE=true` without a token fails closed. - Every door honors `RELATA__BIND` as a plain override on **every** profile — set `0.0.0.0` to reach Relata from another container/host/pod (see [Deploying doors](/docs/deployment/protocol-doors)). No license needed. ## 3-step quickstarts (one per protocol) The pattern is identical every time: **(1)** set your bearer token, **(2)** let the door auto-enable (or set `RELATA__ENABLE=true`), **(3)** point your client at the port and authenticate with the token. ### MongoDB ```bash RELATA_BEARER_TOKEN=change-me relata serve # Mongo door auto-enables. Default port 27017. ``` ```javascript const { MongoClient } = require("mongodb"); const c = new MongoClient("mongodb://localhost:27017", { auth: { username: "relata", password: "change-me" }, // password = RELATA_BEARER_TOKEN }); const db = c.db("cases"); await db.collection("exhibits").insertOne({ _id: "ex1", body: "hello" }); console.log(await db.collection("exhibits").findOne({ _id: "ex1" })); ``` Read the same doc back over SQL: ```sql SELECT * FROM MongoDocument WHERE collection = 'exhibits'; ``` **Limits:** SCRAM-SHA-256 auth, `maxWireVersion` 17, no transactions / change streams / `$push` / `$pull` / `$unset`, nested equality via flattened columns. See [Protocol reference](/docs/reference/protocols#mongodb--any-mongo-wire-client). ### Postgres + pgvector (psql / psycopg2 / LangChain PGVector / DBeaver / TablePlus) ```bash RELATA_BEARER_TOKEN=change-me relata serve # pgwire auto-starts on 5433 (token required — fail-closed without one). ``` ```bash psql -h 127.0.0.1 -p 5433 -U relata relata # password = RELATA_BEARER_TOKEN ``` ```sql CREATE EXTENSION vector; CREATE TABLE docs (id text PRIMARY KEY, embedding vector(3)); INSERT INTO docs VALUES ('a', '[1,0,0]'), ('b', '[0.9,0.1,0]'); -- Cosine KNN — auto-routed to Relata's HNSW index SELECT id FROM docs ORDER BY embedding <=> '[0.9,0.1,0]' LIMIT 2; ``` `INSERT`/`UPDATE`/`DELETE` and ordinary `SELECT` work; GUI clients browse schema via the catalog intercept. KNN ops: `<=>` cosine (preferred, ANN-native), `<->` L2, `<#>` negative inner product (both metric-correct via over-fetch + re-rank). ### Redis (any RESP client) ```bash RELATA_BEARER_TOKEN=change-me relata serve ``` ```bash redis-cli -h 127.0.0.1 -p 6379 -a change-me SET foo bar redis-cli -h 127.0.0.1 -p 6379 -a change-me GET foo ``` Keys persist as governed `KvEntry` rows; read back over SQL (`SELECT key, value FROM KvEntry WHERE key = 'foo'`). Unsupported: `MULTI/EXEC`, `BLPOP`, scripting, cluster commands. Pub/Sub is in-memory only. ### Neo4j (HTTP Cypher or Bolt) ```bash RELATA_BEARER_TOKEN=change-me relata serve ``` ```bash # HTTP Cypher curl -X POST http://neo4j:change-me@127.0.0.1:7474/db/neo4j/tx/commit \ -H "Content-Type: application/json" \ -d '{"statements":[{"statement":"MATCH (n) RETURN n LIMIT 5"}]}' ``` ```python from neo4j import GraphDatabase driver = GraphDatabase.driver("bolt://127.0.0.1:7687", auth=("neo4j", "change-me")) with driver.session() as s: print(s.run("MATCH (n) RETURN n LIMIT 5").data()) ``` Cypher subset: relationship path patterns incl. bounded `-[r*1..5]->`, single-identifier `RETURN [AS alias]`, whitelisted property predicates. Typed labels `(n:Person)` and typed edges `[:KNOWS]` are parsed but ignored. ### ClickHouse (HTTP or native TCP — read-only) ```bash RELATA_BEARER_TOKEN=change-me relata serve ``` ```bash # HTTP curl -X POST http://127.0.0.1:8123/ \ -H "X-ClickHouse-Key: change-me" \ --data-binary "SELECT name FROM Person FORMAT JSONEachRow" ``` ```python from clickhouse_driver import Client ch = Client(host="127.0.0.1", port=9000, password="change-me") print(ch.execute("SELECT name FROM Person LIMIT 5")) ``` Read-only — governed `SELECT`s routed through the planner. Writes are not supported over this door. ### S3 (boto3 / aws CLI / rclone / MinIO client) ```bash RELATA_BEARER_TOKEN=change-me relata serve ``` ```python import boto3 from botocore.config import Config s3 = boto3.client( "s3", endpoint_url="http://127.0.0.1:9191", aws_access_key_id="change-me", aws_secret_access_key="unused", # SigV4 secret defaults to RELATA_BEARER_TOKEN config=Config(signature_version="s3v4", s3={"addressing_style": "path"}), ) s3.create_bucket(Bucket="cases") s3.put_object(Bucket="cases", Key="exhibit-1.txt", Body=b"hello world") print(s3.get_object(Bucket="cases", Key="exhibit-1.txt")["Body"].read()) ``` Supported: ListBuckets, Create/Head/DeleteBucket, ListObjectsV2, GetBucketLocation, Put/Get/Delete/HeadObject, multipart upload. ETag is SHA-256. Bodies ≥ `RELATA_S3_BLOB_THRESHOLD_MB` (default 4 MiB) spill to the content-addressed blob store. ### Arrow Flight (zero-copy columnar streaming) ```bash RELATA_FLIGHT_ENABLE=true RELATA_BEARER_TOKEN=change-me relata serve ``` ```python import pyarrow.flight as fl client = fl.connect("grpc://localhost:8815") reader = client.do_get(fl.FlightDescriptor.for_command( b"PURPOSE 'analytics' SELECT * FROM Person")) for batch in reader: print(batch.data.num_rows, "rows") ``` Arrow IPC — no JSON intermediate, no string serialisation. Use for high-throughput columnar reads. ## Cross-protocol consistency (the actual win) Write via S3, read over SQL. Write via Mongo, read over pgvector. Write via Redis, query the graph. **It's the same store** — ACL, org isolation, and the tamper-evident audit chain apply on every door, on every read and every write. You're not keeping two databases in sync; you have one database with thirteen front doors. ```sql -- Wrote via Mongo? Read here: SELECT * FROM MongoDocument WHERE collection = 'exhibits'; -- Wrote via Redis? Read here: SELECT key, value FROM KvEntry WHERE key = 'foo'; -- Wrote via S3? Read here: SELECT key, size, content_hash FROM S3Object WHERE bucket = 'cases'; ``` ## Which path should I pick? - **"I have a working Mongo/Postgres/Redis/Neo4j app and want governance + history + provenance."** → Drop-in door. Repoint the client; you're done. - **"I'm starting fresh."** → SDK (Python / TypeScript / Go) for the full surface — SQL + graph + memory + MCP + hybrid search + AML/intel operators. - **"I want to migrate data out of my existing DB into Relata once."** → [`relata import --from postgres`](/docs/guides/connectors#migration-connectors-relata-import---from) (live; Mongo/Neo4j/ClickHouse are documented stubs — use the door for ongoing traffic meanwhile). - **"I want Relata to ingest from Kafka / MISP / TAXII / sanctions feeds."** → Native ingest doors — see [Ingestion & SmartIngest](/docs/guides/ingestion). - **"I'm running a multi-node cluster — do I need `mongodb+srv://` / a seed list?"** → No. Relata is smart-server, dumb-client: point your client at one load balancer / Kubernetes `Service` in front of the cluster and the coordinator fans out internally. See [Connecting clients to a cluster](/docs/deployment/cluster#connecting-clients-to-a-cluster--one-address-not-a-seed-list). ## See also - [Protocol Compatibility (deep reference)](/docs/reference/protocols) — full per-protocol semantics, limits, and design notes - [Deploying Protocol Doors](/docs/deployment/protocol-doors) — bind vars, Docker `-p` publishing, Kubernetes `containerPort`, cross-host reachability - [Environment Variables](/docs/reference/env-vars#wire-protocol-ports) — the canonical door env-var table - [Connectors & Extensions](/docs/guides/connectors) — the ETL/extension framework (different from wire doors) and `relata import` - [Limits & Caveats](/docs/reference/limits) — per-protocol status and known gaps ============================================================================== # Agent Memory URL: https://relatadb.dev/docs/concepts/agent-memory ============================================================================== # Agent Memory Agents that rely on context windows for memory forget everything when the window closes. Agents that write to a plain vector store have no governance: no audit trail, no access control, no way to know who stored what or why, and no way to correct a wrong belief without losing its history. Relata is the governed memory layer: cognitive verbs that store, retrieve, and manage beliefs as bi-temporal, provenance-tracked rows — the same storage model as everything else in the database. Every memory item carries who wrote it, when it was believed to be true, what purpose was declared, and a confidence score. Old beliefs are not deleted — they are **superseded**, so the full belief history is always queryable. > **What lives where.** This page is the *concept* — why governed memory, what the model is, and how recall ranks results. For every verb, argument, response shape, and curl/JSON example, see the [Agent memory reference](/docs/reference/agent-memory); for the tool schemas, the [MCP tools reference](/docs/reference/mcp-tools). ## The verb model (conceptual) The surface is a small set of **cognitive verbs** rather than free-form CRUD. Conceptually they fall into three groups: - **Store & correct** — `remember` a belief, `consolidate` (supersede) a belief that changed, `forget` under a retention policy (never a hard delete — obeys legal holds). - **Retrieve & inspect** — `recall` by hybrid BM25 + vector search (optionally `AS OF` a past moment), `recognize` whether an identity is known, `justify` the full provenance chain behind a belief. - **Relate & summarise** — `associate` two items, `resolve` conflicting beliefs by policy, `summarise` a session/topic, and `episodes_in` to walk a session's narrative. Five canonical types back them: `MemoryItem` (one belief), `AgentSession`, `ToolCall`, `DecisionRecord`, and `Episode` — all ordinary bi-temporal rows. For the full verb/argument/type reference, see [Agent memory reference](/docs/reference/agent-memory). ## Why governed memory (vs. convenience-memory) Most agent-memory products (Mem0, RushDB, Zep) optimize for *convenience*: push JSON, get semantic recall, schema-free. Relata optimizes for *accountability* — the memory must answer **"what did the agent know, when, and why — and who was allowed to see it?"** | Concern | Relata answer | |---|---| | **Bi-temporal recall** | Every memory carries `valid_from/to` (when it was true) *and* `system_from/to` (when Relata learned it). Query "what did the agent believe at T?" exactly. | | **Tamper-evident audit** | Hash-chained commit manifests — any deletion or mutation leaves a forensic trail. | | **Provenance chain** | Every `MemoryItem` links back to the `ToolCall` and `AgentSession` that produced it; `justify` replays exactly how a decision was reached. | | **Governed access** | Cedar-inspired ABAC + PURPOSE restrict which agents can read which memories; multi-tenant isolation is enforced at the query-planner level. | | **Finite context window** | `recall` injects only a bounded, ranked slice per turn (`LIMIT N BUDGET T`) — unbounded memory, bounded prompts. | | **Hallucination / memory poisoning** | PROV-O per row + hash chain; no source = not a memory. Every result is `justify`-able. | ## How recall ranking works `recall` runs three steps: 1. **Hybrid retrieval** — BM25 over text fields, vector similarity over the stored embedding, fused via reciprocal-rank fusion (RRF). Results that score in both signals rank higher than results that score in only one. 2. **Temporal filter** — if `as_of` is supplied, only beliefs that were `valid` at that point in time are considered. This lets an agent reconstruct what it "knew" at a given past moment. 3. **Re-scoring** — the fused relevance is blended **additively** with recency and the forgetting curve (`0.70·relevance + 0.15·recency + 0.15·forget`), then multiplicatively gated by `confidence × class_weight`. The additive blend prevents a single near-zero lifecycle factor from collapsing the whole score — a high-confidence belief from yesterday outranks a low-confidence belief from a minute ago, but an old but highly-relevant memory is no longer dropped just because its recency is near zero. ## Unlimited memory, bounded prompts Relata gives an agent **unbounded memory with bounded prompts** by separating *what the agent knows* (unlimited) from *what it sees per turn* (bounded): - **Capacity is unbounded** — durable storage is object-store (S3 / self-hosted S3-compatible), not RAM-bound like a vector DB. You run out of bucket, not memory. - **The agent never loads it all** — each turn, `recall` returns only the small, relevant, ranked slice, capped by `LIMIT N BUDGET T`. A 10-year, billion-row memory and a 1 MB memory cost the *same* prompt budget. - **Cold vs hot** — cold history lives on object storage; active-case data is promoted into RAM/SSD via the tiered cache. ## Fast on constrained hardware (no GPU) - **No LLM on the hot path.** Ingest canonicalizes + validates declared identities (deterministic, cheap). Since v1.1 embeddings are caller-supplied — see [LLM & Embedding configuration](/docs/guides/llm-embedding) — so the ingest hot path is pure throughput. - **Early-pruned retrieval.** IdentityIndex bloom filters + graph pushdown + a BM25 shortlist mean the vector path scans a tiny candidate set. - **Single binary, embedded `free` profile** — one process alongside the agent's own loop; no separate vector server, Redis, and graph DB to operate. ## Agent-framework adapters Drop-in governed memory backends for every major Python agent framework. Each wraps the memory verbs, so purpose tracking and ACL stay on. None imports its framework at module load, so they are safe to install without the framework present. | Framework | Import path | Interface shape | |---|---|---| | LangChain | `relata_adapters.langchain.RelataMemory` | `BaseMemory` | | LlamaIndex | `relata_adapters.llamaindex.RelataMemory` | `BaseMemory` | | CrewAI | `relata_adapters.crewai.RelataStorage` | `Storage` | | AutoGen v0.2 | `relata_adapters.autogen.RelataMemory` | `Memory` (async) | | AG2 (v0.4+) | `relata_adapters.ag2.RelataAG2Memory` | `MemoryProtocol` | | Pydantic-AI | `relata_adapters.pydantic_ai.RelataMemoryBackend` | memory backend | | smolagents | `relata_adapters.smolagents.RelataTool` | tool callable | | LangGraph | `relata_langgraph.RelataCheckpointer` | checkpointer | | Auto-detect | `relata_adapters.registry.get_memory_adapter()` | picks installed framework | ```python from relata_adapters.langchain import RelataMemory mem = RelataMemory( base_url="http://localhost:9090", bearer_token=token, purpose="customer-support-agent", ) # chain = ConversationChain(llm=llm, memory=mem) ``` ## Querying memory as SQL Because memory items are ordinary rows, you can query them with SQL — including bi-temporal operators: ```sql -- All beliefs about "Alice" stored in the last 30 days, newest first SELECT content, confidence, valid_from FROM MemoryItem WITH PROVENANCE WHERE MATCH(content, 'Alice') AND system_from >= now() - INTERVAL '30 days' ORDER BY confidence DESC, system_from DESC LIMIT 20 -- What did the agent believe about Alice on 1 June? SELECT content, confidence FROM MemoryItem AS OF '2026-06-01T00:00:00Z' WHERE MATCH(content, 'Alice') ORDER BY confidence DESC ``` ## See also - [Agent memory reference](/docs/reference/agent-memory) — every verb, argument, response shape, and curl/JSON example - [MCP Tools Reference](/docs/reference/mcp-tools) — full tool schemas for all memory verbs - [Hybrid Search](/docs/concepts/hybrid-search) — `recall` runs on the same BM25 + vector pipeline - [Provenance](/docs/concepts/provenance) — `justify` returns the full lineage chain for any memory item - [Governance](/docs/concepts/governance) — ACL and PURPOSE apply to every cognitive verb - [Python SDK](/docs/sdks/python) — `Memory`, `A2AClient`, framework adapters ============================================================================== # AI in RelataDB URL: https://relatadb.dev/docs/concepts/ai-in-relatadb ============================================================================== # AI in RelataDB RelataDB is built both **for** AI and **with** AI — and those are two different things. This page separates them, shows the **SmartIngest pipeline** that turns raw data into governed knowledge, walks through **one complete RAG example**, and explains **why this beats the usual RAG stack**. > **The one-line distinction.** Agents and apps call **into** RelataDB to get governed memory, tools, and retrieval (*for AI*). RelataDB calls **out** to models to embed, search, and interpret (*with AI*). The trust path — identity, access control, provenance, audit — is **deterministic** and never depends on a model. ## The mental model: knowledge first, AI second Most "AI databases" bolt a vector index onto a pile of JSON and call it memory. RelataDB inverts that: **the knowledge is the product; the AI sits on top of it and underneath it, but never in the trust path.** ```text you preprocess RelataDB does this automatically agents/apps query (chunk, enrich, embed) with AI │ │ ▼ │ ┌─────────────┐ ingest door ┌───────────────────────────┐ recall / RAG / nl_query │ raw records │ ─────────────► │ SmartIngest (deterministic)│ ──────────────────────► │ docs, CDRs, │ │ • validate + canonicalize │ hybrid BM25 + HNSW │ posts, PDFs │ │ • identity detect + fuse │ identity-graph recall └─────────────┘ │ • provenance stamp │ provenance citations │ • bi-temporal rows │ governed (ACL + PURPOSE) │ • embeddings (lazy MV) │ └───────────────────────────┘ │ ▼ one governed knowledge graph ── the substrate for RAG & memory ``` The payoff: retrieval isn't just "find text that looks similar." Because SmartIngest canonicalizes identities and links records across sources, a query about *Alice* finds the **whole identity cluster** (her email, phone, accounts, co-occurrences) and every fact attached to it — not just chunks that mention the word "Alice." ## Built for AI — what agents and LLMs consume Every surface below runs through the same governed path as a human-issued query — ACL, cell masking, `PURPOSE`, tenant isolation, audit. **Governed AI**: the agent gets tools, not a back door. | Surface | What it is | Go deeper | |---|---|---| | **Agent memory** | 10 cognitive verbs (`remember · recall · recognize · justify · consolidate · forget · associate · episodes · resolve · summarise`) over MCP and `/memory/*`. Memories are bi-temporal, provenance-tracked rows — old beliefs are *superseded*, never deleted. | [Agent Memory](/docs/concepts/agent-memory) | | **MCP tools** | Governed tools an agent can call — query, search, graph, identity, and intelligence tools (`trace_crypto`, `beneficial_ownership`, `screen_sanctions`, `nl_query`, `similar_multimodal`, …). | [MCP Tools](/docs/reference/mcp-tools) | | **Framework adapters** | Drop-in backends for **8** frameworks: LangChain, LlamaIndex, CrewAI, AutoGen, AG2, Pydantic-AI, smolagents, LangGraph. | [Adapters](#wire-it-to-an-agent) below | | **Agent-to-agent (A2A)** | Typed `A2AClient` so agents coordinate through the governed knowledge plane (task lifecycle + checkpoints) instead of side channels. | [Python SDK](/docs/sdks/python) · [TS SDK](/docs/sdks/typescript) | | **Governed RAG** | Hybrid BM25 + vector retrieval, ACL-safe, with provenance citations on every chunk. | [Hybrid Search](/docs/concepts/hybrid-search) | | **Natural-language query** | `nl_query` — NL → governed SQL → execute. `interpret: true` adds an LLM summary. Falls back to a deterministic parser when no LLM is configured. | [LLM config](/docs/guides/llm-embedding) | ## Built with AI — what runs under the hood Each piece below is **model-agnostic and opt-in** — point it at the provider you trust. Nothing is hard-wired to a vendor. | Capability | What it does | Config | |---|---|---| | **Auto-embedding** | Computes `_emb_*` vectors for semantic search — caller-supplied or async via the media worker. **Never on the ingest hot path** (since v1.1). | `RELATA_EMBED_*` · [LLM & embeddings](/docs/guides/llm-embedding) | | **Hybrid ranking** | Fuses BM25 keyword scores with custom-HNSW vector similarity via reciprocal-rank fusion (RRF) in a single query. | [Hybrid Search](/docs/concepts/hybrid-search) | | **Media embeddings** | CLIP (image/video), ArcFace (face), CLAP (audio) — `_emb_image/_emb_face/_emb_audio/_emb_video`. | [Ingestion](/docs/guides/ingestion) | | **External scorers** | Bring-your-own model for NER / sentiment / stance / bias. RelataDB writes the scorer's output back as governed, provenance-stamped assertions. | [SmartIngest](/docs/architecture/smartingest) | | **LLM interpretation** | `nl_query` translation, incident clustering, anomaly detection, and detection-rule **tuning** (`/rules/:id/tuning`). | [Jobs & detection](/docs/guides/jobs-workflows) | > All model-touching config lives on one page: [LLM & Embedding Configuration](/docs/guides/llm-embedding) — the **LLM endpoint** (`RELATA_LLM_URL`) for NL query, and the **embedder sidecar** (`RELATA_EMBED_URL`, formerly `RELATA_ACCEL_ENDPOINT`) for vectors. ## The SmartIngest pipeline — preprocess once, retrieve forever SmartIngest is the deterministic engine that turns raw input into the governed substrate RAG and memory draw from. **You can preprocess (chunk, enrich, pre-embed) before you ingest; SmartIngest then does the rest, automatically, on every write.** ### What SmartIngest does (deterministic, checksum-gated) ```text raw text / cell value │ ▼ tokenize ──► per-token shape gate (regex) │ pass / fail ──► skip ▼ format + checksum validate ◄── IBAN mod-97, IMEI Luhn, │ pass / fail ──► skip Aadhaar Verhoeff, VIN… ▼ DetectionHit (CanonicalKind + Identity) │ ▼ IdentityIndex: (kind, bytes) → all observations │ ▼ fuse records that share a validated identifier → one cluster ``` This runs **lazily as materialized views** off the WAL, so it never blocks the write hot path and it's re-runnable — upgrade a detector and the MV refreshes, no source backfill. ### Why this supercharges RAG and agent memory | Without SmartIngest (typical vector store) | With SmartIngest (RelataDB) | |---|---| | `+44 7700…` and `07700…` are two unrelated strings | one canonical phone number — they join | | Retrieval finds *text that looks similar* | Retrieval finds *the entity and everything attached to it* across every source | | A fact's source is "trust me" | Every fact carries PROV-O provenance + a tamper-evident hash chain | | No notion of "what was true when" | Bi-temporal: replay exactly what the agent knew at any moment | > **The honest boundary:** RelataDB does *identifier* extraction, not general named-entity recognition. There is no in-tree transformer guessing names out of prose. For fuzzy NER, sentiment, or intent, register an **external scorer** — RelataDB writes its output back as governed, provenance-stamped typed assertions and fuses it into the same graph. ([SmartIngest deep dive](/docs/architecture/smartingest)) ### The end-to-end flow for an AI app ```text 1. PREPROCESS (optional, you) 2. INGEST (one door) 3. QUERY WITH AI (agent/LLM) chunk long docs client.ingest(...) recall / SEARCH HYBRID pre-compute _emb_text POST /ingest/document nl_query("…") enrich with a scorer PUT via S3 / Mongo / similar_multimodal(...) tag with purpose + class Redis / pgwire door → governed, cited, replayable │ │ ▲ └──────────────► SmartIngest ──────┘ │ canonicalize · detect · fuse · stamp │ embed (async MV) ── into the knowledge graph ┘ ``` ## Build RAG on RelataDB — one complete example A governed retrieval-augmented agent over an AML document corpus. Four steps; Python here, the same shapes exist in [TypeScript](/docs/sdks/typescript) and [Go](/docs/sdks/go). ### 0. Declare the ontology once You don't `CREATE TABLE` — you declare types. The schema shapes what SmartIngest validates and what the SDKs expose. ([Ontology & schema](/docs/concepts/ontology)) ```bash # ObjectType: IntelChunk — a retrievable text chunk with a vector + a source. curl -X POST http://localhost:9090/types \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "IntelChunk", "properties": [ {"name": "text", "type": "string", "indexed": "bm25"}, {"name": "_emb_text","type": "[1536]f32"}, {"name": "source", "type": "string"}, {"name": "mentions","type": "identity"}, {"name": "tags", "type": "[]string"} ] }' ``` `text` is BM25-indexed for keyword recall; `_emb_text` carries the vector (caller-supplied or populated by the embedder sidecar); `mentions` is an `Identity` column that SmartIngest will populate from identifiers in the text. Schema-as-code, versioned in git, evolvable without downtime. ### 1. Ingest — SmartIngest does the heavy lifting Drop documents in; they're auto-chunked, identity-detected, provenance-stamped, and embedded asynchronously. Identifiers in the text (IBANs, emails, phones) are canonicalized and linked into the identity graph. ```python # Bulk ingest chunks — or use POST /ingest/document with auto_chunk:true for whole PDFs client.ingest("IntelChunk", [ {"_pk": "c1", "text": "Alice wired $1.2M to Shell Co Ltd (IBAN GB29 NWBK 6016 1331 9268 19).", "source": "sar-2024-07.pdf", "tags": ["wire", "shell-co"]}, {"_pk": "c2", "text": "Shell Co Ltd is 100% owned by Atlas Holdings, BVI.", "source": "registry-bvi.json", "tags": ["ownership"]}, {"_pk": "c3", "text": "Alice's known account alice@corp.io; primary +1 415 555 0111.", "source": "kyc.csv", "tags": ["kyc"]}, ]) # SmartIngest: GB29…IBAN, alice@corp.io, +1 415… → validated, canonicalized, # fused into one identity cluster for "Alice" — across all three sources. ``` ### 2. Retrieve — hybrid + provenance, ACL-safe ```python # Simple path — the SDK helper hits = client.search("shell company laundering", "IntelChunk", limit=10, highlight=True) for h in hits.hits: print(h.score, h.fields["source"], h.highlights) # Full path — SQL with hybrid + provenance citations rows = client.query(""" SELECT id, text, source, score FROM IntelChunk WHERE SEARCH HYBRID('shell company laundering', top_k = 10) WITH PROVENANCE """) # Every returned chunk carries where it came from — no hallucinated citations. ``` ### 3. Ground the agent — memory + RAG in one plane Because `IntelChunk` rows and `MemoryItem` rows live in the **same** governed store, your agent recalls documents *and* its own notes through one governed surface. ```python from relata import Memory mem = Memory("http://localhost:9090", purpose="aml", bearer_token=TOKEN) # Agent remembers a working hypothesis… mem.add("Alice → Shell Co → Atlas Holdings looks like layering", confidence=0.8) # …and recalls both its notes and the corpus, ranked together. for m in mem.search("alice shell company", top_k=5): print(m["content"], "←", m.get("source", "agent-note")) ``` ```typescript // TypeScript — same verbs, same governance import { createClient } from "@zysec-ai/relata-sdk"; const relata = createClient("http://localhost:9090", { defaultPurpose: "aml", bearerToken: TOKEN }); await relata.remember("Alice → Shell Co → Atlas Holdings looks like layering"); const hits = await relata.recall("alice shell company", { topK: 5 }); ``` ### Same thing, over MCP — for Claude / Cursor / Cline ```json // Point your MCP client at Relata, then call tools directly: { "name": "recall", "arguments": { "q": "alice shell company", "top_k": 5, "purpose": "aml" } } { "name": "nl_query","arguments": { "query": "who owns Shell Co Ltd and how does Alice connect?", "purpose": "aml", "interpret": true } } ``` `nl_query` returns governed rows **plus** `generated_sql`, `model_id`, and `llm_used` (so you always know whether an LLM produced the SQL). `interpret: true` adds a natural-language summary with full model provenance. > **That's the whole loop:** declare once → ingest → retrieve with citations → ground the agent. No separate vector DB, no graph DB, no audit store, no ETL between them. ## Wire it to an agent Drop-in adapters wrap the memory verbs — purpose tracking and ACL stay on. None of the Python adapters imports its framework, so they're safe to install without it. | Framework | Install | Snippet | |---|---|---| | **LangChain** | `pip install relata-sdk` + langchain | `from relata_adapters.langchain import RelataMemory` — pass `RelataMemory(base_url=URL, purpose="agent")` to `ConversationChain(memory=mem)` | | **LlamaIndex** | `pip install relata-sdk` + llamaindex | `from relata_adapters.llamaindex import RelataMemory` | | **CrewAI** | `pip install relata-sdk` + crewai | `from relata_adapters.crewai import RelataStorage` | | **AutoGen / AG2** | `pip install relata-sdk` | `from relata_adapters.ag2 import RelataAG2Memory` | | **Pydantic-AI** | `pip install relata-sdk` | `from relata_adapters.pydantic_ai import RelataMemoryBackend` | | **LangGraph** | `pip install relata-sdk[langgraph]` | `from relata_langgraph import RelataCheckpointer` → `workflow.compile(checkpointer=cp)` | | **smolagents** | `pip install relata-sdk` | `from relata_adapters.smolagents import RelataTool` | ```python # LangChain example — governed long-term memory on any chain from langchain_openai import ChatOpenAI from langchain.chains import ConversationChain from relata_adapters.langchain import RelataMemory chain = ConversationChain( llm=ChatOpenAI(), memory=RelataMemory( base_url="http://localhost:9090", bearer_token=TOKEN, purpose="support-agent", session_id="cust-42", top_k=5, ), ) ``` LangGraph works the same way via `RelataCheckpointer` (governed checkpoint persistence over the A2A door) — see the [Agent Memory](/docs/concepts/agent-memory) adapters table. ## Why it's better than the usual RAG stack The typical stack is **Postgres + Pinecone + Neo4j + a memory store + an audit log**, glued together with ETL. RelataDB replaces all of it with one binary — and the differences show up exactly where RAG fails in production. | Dimension | Typical RAG stack | mem0 / cognee (memory SDK) | **RelataDB** | |---|---|---|---| | **Infra** | 4–6 databases + ETL pipelines | an SDK on *your* database | **one binary, one query plane** | | **Retrieval quality** | text similarity only | text similarity only | **hybrid + identity graph** — finds the entity, not just the string | | **Citations** | "trust the model" | none | **PROV-O provenance on every chunk** | | **Access control in retrieval** | post-filter (leaky) | none | **ACL compiled into the scan** — masked cells never reach the LLM | | **History** | latest-value-wins | current state only | **bi-temporal** — replay what the agent knew at *T* | | **Audit** | bolt-on log | none | **tamper-evident hash chain** | | **Identity across sources** | your ETL problem | your ETL problem | **SmartIngest fuses it automatically** | | **Governance for agents** | DIY | DIY | **every agent call is ACL'd, purpose-bound, audited** | The net: **where others give you retrieval, RelataDB gives you retrieval you can defend.** Same SDK ergonomics, none of the seams. ### The honest trade-off RelataDB optimizes for **accountability**; convenience-memory tools optimize for **fast onboarding**. If your RAG just needs "push JSON, get semantic hits" with no governance, a vector DB is simpler. If the answer must be **right, sourced, replayable, and access-controlled** — regulated, intel, LEA, FININT, court-grade, or enterprise — that's where RelataDB wins. ## The deterministic boundary — why you can trust it Models are great at *guessing*; they're bad at *proof*. So RelataDB keeps the proof path deterministic: - **Identity resolution is checksum-gated, not learned.** An IBAN's mod-97 must pass before records link. You trust the merge because you trust the math. ([SmartIngest](/docs/architecture/smartingest)) - **Access control is compiled into scans.** ACL decisions are bitmap predicates, not model outputs — identical on every call. - **Provenance and audit are tamper-evident.** Every row carries a PROV-O link; commits form a SHA-256 hash chain. No model is in that loop. **The AI is modular and replaceable; the guarantees are not.** Swap the embedder, the LLM, or the scorer without touching governance, history, or audit. ## Choose your posture Every model-touching piece is opt-in, so you pick where on the spectrum you run: | Posture | What's on | Trade-off | |---|---|---| | **Fully deterministic** | No LLM, no external embedder; query-side CPU embedder only | Maximal auditability; `nl_query` uses the deterministic fallback; vectors are caller-supplied | | **Bring-your-own models** | Your LLM + embedder sidecar (local or self-hosted) | Full AI surface, no vendor lock-in, sovereign | | **Managed models** | OpenAI-compatible provider | Fastest to stand up; model calls leave your perimeter unless proxied | Whatever you pick, the governed path does not change. ## See also - [Agent Memory](/docs/concepts/agent-memory) — the 10 cognitive verbs, recall ranking, and every adapter - [SmartIngest](/docs/architecture/smartingest) — the deterministic identity pipeline in depth - [Hybrid Search](/docs/concepts/hybrid-search) — BM25 + HNSW + RRF - [Governed RAG use case](/docs/use-cases/appdev-governed-rag) — a vertical walkthrough - [LLM & Embedding Configuration](/docs/guides/llm-embedding) — every model-touching env var - [Governance](/docs/concepts/governance) — why every AI call rides the same policy path - [MCP Tools](/docs/reference/mcp-tools) · [Agent Memory reference](/docs/reference/agent-memory) ============================================================================== # Bi-Temporal Model URL: https://relatadb.dev/docs/concepts/bitemporal ============================================================================== # Bi-Temporal Model Most databases track one thing when you update a row: that it changed. Relata tracks two: when the fact was true in the world, and when the database learned about it. Those are different questions, and collapsing them into one timestamp causes subtle, hard-to-audit bugs in any system that ingests late-arriving data, corrects historical records, or must reproduce a past decision exactly. Bi-temporality is not a layer on top of Relata — it is in the storage model, in the query planner, in the WAL, and in every Parquet snapshot. ## The four timestamps Every row in every type carries exactly four `i64` nanosecond UTC timestamps: | Timestamp | Axis | Meaning | |---|---|---| | `valid_from` | Valid time | When the fact became true in the real world (inclusive) | | `valid_to` | Valid time | When the fact stopped being true (exclusive) | | `system_from` | System time | When the database recorded this version (inclusive) | | `system_to` | System time | When a later write superseded this version (exclusive) | Both intervals are half-open: `[valid_from, valid_to)` — a row is visible at `valid_from` inclusive and invisible at `valid_to` exclusive. An open `valid_to` or `system_to` is represented as `i64::MAX`. ### A concrete example A bank processes a wire transfer on 1 June. The compliance team logs it on 3 June after a manual review. On 5 June an auditor corrects the amount. ``` Row 1 (original entry): valid_from = 2024-06-01 (when the transfer happened) valid_to = i64::MAX (still "true" until corrected) system_from = 2024-06-03 (when we logged it) system_to = 2024-06-05 (superseded when corrected) Row 2 (corrected amount): valid_from = 2024-06-01 (same real-world date) valid_to = i64::MAX system_from = 2024-06-05 (when the correction was recorded) system_to = i64::MAX (current truth) ``` With a single-timestamp system, row 2 overwrites row 1 — the original belief is gone. With bi-temporality, both rows coexist. You can ask either question at any time. ## Querying across time ### Valid-time travel — what was true? ```sql -- What was Alice's address on 1 January 2024? SELECT name, address FROM Person AS OF '2024-01-01T00:00:00Z' WHERE name = 'Alice' -- What transactions were active on a given date? SELECT * FROM Transaction AS OF '2024-06-01T00:00:00Z' WHERE amount > 10000 ``` `AS OF` filters to rows where `valid_from <= ts < valid_to`. `AS OF` answers "what was true at ts?" without touching `system_*`. It is the most common time-travel query — no `WHERE` predicates on timestamps needed. ### System-time travel — what did we believe? ```sql -- What did the database believe about Alice on 1 March 2024? SELECT name, address FROM Person AS OF SYSTEM TIME '2024-03-01T00:00:00Z' WHERE name = 'Alice' ``` `AS OF SYSTEM TIME` filters to rows where `system_from <= ts < system_to`. ### Combining both axes ```sql -- Given what we knew on 1 March, what should we have reported for 1 January? -- Use explicit predicates to combine both axes: SELECT name, address FROM Person WHERE valid_from <= 1704067200000000000 -- 2024-01-01 in ns AND valid_to > 1704067200000000000 AND system_from <= 1709251200000000000 -- 2024-03-01 in ns AND system_to > 1709251200000000000 ``` ### Seeing the full history of a row ```sql -- All versions of Alice's record, newest first SELECT name, address, valid_from, valid_to, system_from, system_to FROM Person WHERE name = 'Alice' ORDER BY system_from DESC ``` Without any `AS OF`, Relata returns the **current state** by default: the version whose `valid_to = i64::MAX AND system_to = i64::MAX`. ## Timestamp format The parser accepts two forms: - **ISO-8601 UTC**: `'2024-06-01T00:00:00Z'` or `'2024-06-01'` - **Raw nanoseconds**: `1717200000000000000` Explicit non-UTC offsets (e.g. `'2024-06-01T00:00:00+05:30'`) are **rejected** at parse time. Convert to UTC before inserting — Relata will not silently shift your timestamps. This keeps cross-region audit chains unambiguous. ## Writing bi-temporal data On a plain `INSERT`, Relata sets `system_from` to the current HLC timestamp and `valid_from` to the same value unless you override it. To record a fact that was true in the past: ```sql INSERT INTO Transaction (id, amount, valid_from, valid_to) VALUES ('tx-42', 9800.00, '2024-06-01T00:00:00Z', '9999-12-31T00:00:00Z') ``` To correct a historical record, insert the corrected row with the same `valid_from` — the store creates a new system-time version automatically. The old version is retained and queryable. ## Implementation notes - Row model and `i64` timestamp type: `relata-core` - In-memory bi-temporal store with per-type interior locking: `relata-storage` - WAL and Parquet snapshots persist all four timestamps unchanged - HLC (Hybrid Logical Clock) keeps `system_from` monotonically increasing across restarts ## What is not yet shipped SQL:2011 period predicates are **not yet parsed**: `FOR SYSTEM_TIME FROM … TO …`, `OVERLAPS`, `CONTAINS`, `PRECEDES`, and automatic period-splitting on `UPDATE`. Use explicit `valid_from`/`system_from` predicates or `AS OF` / `AS OF SYSTEM TIME` until they land. ## See also - [Provenance](/docs/concepts/provenance) — every write also gets a hash-chained manifest entry - [Governance](/docs/concepts/governance) — ACL and PURPOSE are recorded on the same system timeline - [SQL Reference](/docs/reference/sql) — full `AS OF` syntax ============================================================================== # Federation — cross-deployment query (roadmap) URL: https://relatadb.dev/docs/concepts/federation ============================================================================== # Federation — cross-deployment query (roadmap) > ## ⚠️ Status: design accepted, NOT yet shipped > > This is a **design document for a roadmap feature**, not a working capability. Federated cross-deployment query is an accepted design direction, but **no parser, planner, or wire support exists in the codebase yet** (`FEDERATED ACROSS (...)` is not parsed; `FederationAgreement` is not wired). We surface the design here so multi-agency / multi-region teams can plan around it. Do not write code against this page today — check the [release notes](https://github.com/relatadb) or open an issue before relying on any of it. ## The problem it solves One agency's lead is another agency's known entity. Today every cross-agency lookup is an email or a phone call, and the answer is "no idea, ask them." Federation is the unification multiplier: each department, agency, or region runs Relata on **its own data**, behind **its own ACL and jurisdiction policy**, and a single query can pull in what peers permit — with **zero data movement and zero sovereignty loss**. ## The design (when it ships) A query carrying `FEDERATED ACROSS (eu_relate, us_relate, apac_relate)` will be parsed and planned locally, then fanned out to each named peer deployment over Arrow Flight with mutual TLS + a signed query envelope. Each peer enforces its **own** ACL, jurisdiction-routing policy, purpose check, and compartment labels. Per-peer result batches are stitched at the originator, with the source deployment tagged on every returned row's provenance (`provenance.via_deployment`). The originator does **not** receive raw rows the peer refuses to release — it receives a typed **release receipt** (count, classes, redaction summary, deny reason). Identity operators (`LOOKUP_IDENTITY`, `RESOLVE_IDENTITY`, `IDENTITY_CLUSTER`) federate the same way: each peer returns its slice of the cluster with confidences, and the originator computes the union cluster with per-edge provenance tagged to the contributing deployment. ```sql -- Illustrative — does NOT parse today. Captured here so the shape is stable when it lands. PURPOSE 'cross-border-finance' SELECT id, name, risk_score FROM Person FEDERATED ACROSS (eu_relate, us_relate, apac_relate) WHERE LOOKUP_IDENTITY('+44 7700 900123'); ``` ## What makes it different from existing federation tools | | Postgres FDW / Trino federation | **Relata federation (planned)** | |---|---|---| | Authorization carried in the query | None — trusts the client | Purpose, compartment labels, jurisdiction routing — peer enforces its own | | Identity resolution across deployments | Manual joins | `LOOKUP_IDENTITY` / `RESOLVE_IDENTITY` federate natively | | Refusal semantics | Empty result (ambiguous) | Typed release receipt — distinguish "no match" from "refused for compartment reasons" | | Audit / provenance | None | Source deployment tagged on every row's PROV-O lineage | ## Constraints (per the design) - **Read-only in v0.1.** Cross-deployment writes happen via typed CDS export with dual control, not via federation. - **No federated joins below the object level.** A peer either returns object rows or refuses. Cross-deployment stitching happens via identity, not via raw column-level joins. - **Version compatibility.** Every peer must run a semver-minor-compatible Relata version; the planner refuses federation against incompatible peers. - **Peers may legitimately return zero rows for security reasons.** The originator cannot distinguish "no match" from "refused" beyond the typed release receipt. ## Alternatives Relata rejected (and why) - **Push everyone onto one shared deployment** — rejected: sovereignty, jurisdiction, and clearance compartments make a single shared instance impossible for multi-agency cooperation. - **Sync-via-export (each peer exports a snapshot daily)** — rejected: stale, expensive, and the snapshot itself is a CDS event every time, with no live identity resolution. - **Generic FDW-style federation** — rejected: FDWs don't carry typed purpose / compartment / routing-policy, so peer enforcement degenerates to "trust the client." ## When this matters to you - **Multi-agency law enforcement / intelligence** — each agency keeps its data, but a sanctioned identity or a phone number resolves across all cooperating peers with per-row provenance. - **Multi-region enterprise (bank, telco)** — data residency rules keep each region's customer data local; federation lets a global investigator query across regions without copying data. - **Federated KYC / AML** — one entity known to peer X under case Y surfaces as a pointer, without either side exposing its underlying records. ## Track the implementation - **No code yet.** If you need this capability, open an issue describing your topology so it informs the implementation sequencing. - **Dependencies:** MLS compartments, CDS sanitisation, jurisdiction routing, and purpose all need to be in place before federation can ship. ## See also - [Identity](/docs/concepts/identity) — the operators federation will carry peer-to-peer - [Governance](/docs/concepts/governance) — Cedar ACL, purpose, jurisdiction routing (the machinery each peer enforces) - [Provenance](/docs/concepts/provenance) — the per-row lineage federation extends with `via_deployment` - [Limits & Caveats](/docs/reference/limits) — what honestly ships today vs. roadmap ============================================================================== # Governance URL: https://relatadb.dev/docs/concepts/governance ============================================================================== # Governance Most databases let you add access control. Relata builds it into the query path so there is no way to bypass it: every read passes through a Cedar-inspired ABAC policy engine, every write enters a tamper-evident audit hash chain, and classified types are blocked at egress regardless of query success. You cannot "accidentally" skip it. The reason this matters: bolt-on access control fails at integration seams. When a new protocol door opens (pgwire, gRPC, S3, Arrow Flight), a bolt-on layer is easy to forget. In Relata the planner enforces policy before it hands results to any protocol handler — adding a new door does not create a new bypass. ## Policy evaluation model The policy engine lives in `relata-acl`. Its semantics are Relata's own — inspired by Cedar's attribute-based model but not using the open-source `cedar-policy` crate. | Property | Behaviour | |---|---| | Decision rule | **Deny-wins** — any matching deny overrides all allows | | Row filtering | Bitmap bitset; branch-predicted; ~1.0× raw-scan overhead on allow-all | | Conditional ACL | ~1.32× raw-scan p50 (realistic selective policies) | | Cell masking | `CellFilter::apply_to_row` — ~2.6× raw-scan p50 (allocates per-row); avoid on hot full-table scans | | Organisation isolation | Planner guard + `tenant_id` keying in `store/scan.rs` | | Multi-tenant | Every request carries `tenant=` → `X-Organization-Id`; no cross-tenant data ever crosses the planner | ### EXPLAIN POLICY To understand why a query was allowed or denied, prefix it with `EXPLAIN POLICY`: ```sql EXPLAIN POLICY SELECT name, ssn, salary FROM Employee WHERE department = 'Engineering' ``` The response is a decision tree showing which rules matched, in evaluation order, and what action each took (allow / deny / mask). This is the primary debugging tool for policy authors. ## PURPOSE tracking Every query may carry a `purpose` token. When present, it is recorded in the audit log against the principal, timestamp, and affected rows. When absent, the query still runs — PURPOSE is optional by design — but recorded as `purpose: null`. In `strict` mode, a missing or unregistered purpose is rejected before the query reaches the planner: ```bash # Strict: all queries must declare a registered purpose RELATA_PURPOSE_MODE=strict RELATA_PURPOSES=analytics,audit,compliance,product_research # Open: any purpose string is accepted (dev/test only) RELATA_PURPOSE_MODE=open ``` Purposes support hierarchical scoping with `:` as separator. A principal with permission for `analytics` automatically covers `analytics:external` and `analytics:internal`. ```sql -- Declaring purpose in SQL (pgwire / CLI) PURPOSE 'analytics' SELECT name, revenue FROM Account LIMIT 100 ``` ## Per-tenant encryption Each tenant has a Data Root Key (DRK) managed by the KMS integration. Rows are encrypted at rest under the tenant's DRK. Cross-tenant reads are impossible at the storage layer — the planner guard and the encryption boundary are independent controls that both have to fail for data to leak. When GDPR erasure runs, the DRK for the erased subject is destroyed, rendering all encrypted rows cryptographically unreadable without needing to locate and delete individual blocks. ## Egress filtering Four classified types are blocked at egress unconditionally — they never appear in query results, tool responses, or protocol-door outputs, regardless of what the query asked for or what ACL rules allow: | Type | What it represents | |---|---| | `SourceTrueIdentity` | HUMINT-protected true identity (SPECS §5.19) | | `SigintIntercept` | Signal intelligence intercept records | | `AccessScopedIntercept` | Restricted access-scoped data (SPECS §5.20) | | `LawfulInterceptRecord` | Lawful intercept records | Egress filtering runs after ACL evaluation, so it catches cases where a policy bug would otherwise have allowed the data through. ## GDPR Art. 17 erasure ```sql ERASE SUBJECT 'person-42' REASON 'gdpr-art17' CERTIFY; ``` What this does, in order: 1. Shreds all rows whose `entity_id` resolves to `person-42` via the `IdentityIndex`. 2. Deletes orphaned content-addressed blobs. 3. Destroys the per-subject Data Encryption Key via KMS (fail-closed — if the KMS call fails, erasure is aborted). 4. Writes a tombstone to the WAL and advances the audit hash chain. 5. Returns a signed Art. 17 receipt with a manifest hash. With no KMS configured, step 3 succeeds in `reason: "gdpr-art17"` mode — the rows are deleted but the cryptographic key destruction step is skipped. The receipt makes this explicit. The same operation is available as: - MCP tool: `erase_subject` - Python SDK: `IdentityClient.erase_subject(subject_id, reason="gdpr-art17")` ## Audit hash chain Every write — INSERT, UPDATE, erasure, schema change — is recorded in the append-only audit log with: principal, timestamp (HLC ns), purpose, cost units, and a SHA-256 hash that chains to the previous entry. Any retroactive modification breaks the chain. ```bash # Verify chain integrity relata doctor # Count entries and confirm chain validity curl http://localhost:9090/audit/count # → { "entries": 4821, "chain_valid": true } ``` ## Governance operators (Python SDK) ```python from relata import RelataClient, GovernanceClient, AuditClient with RelataClient(url, bearer_token=token, purpose="compliance") as client: gov = GovernanceClient.from_client(client) # Legal hold — suspends retention policies for a case gov.place_legal_hold(case_id="case-7", reason="litigation hold") # WORM retention — immutable for 7 years gov.set_worm_policy(object_type="AuditEvent", retention_days=2555) # Import Sigma detection rules gov.import_sigma(open("sigma/financial-fraud.yml").read()) # Break-glass emergency access (requires approver) gov.request_breakglass(reason="P0 incident", approver="ciso@example.com") # Data Subject Access Request gov.submit_dsar(subject_id="person-42", requester="dpo@example.com") with RelataClient(url, bearer_token=token, purpose="compliance_review") as client: audit = AuditClient.from_client(client) # Paginated audit log for page in audit.entries(filter={"purpose": "analytics", "since": "2026-01-01T00:00:00Z"}): print(page) # Signed receipt for a specific exhibit receipt = audit.signed_receipt(exhibit_id="exhibit-7") # PDF export for court submission pdf = audit.export_pdf(filter={"case_id": "case-7"}) ``` ## What is not yet shipped - gRPC cell masking is the highest-priority open audit finding — cell masking currently applies on the HTTP and SQL surfaces; gRPC responses can expose unmasked cell values to principals that should see redacted output. ## See also - [Provenance](/docs/concepts/provenance) — the hash chain that makes audit log entries tamper-evident - [Bi-Temporal Model](/docs/concepts/bitemporal) — governance decisions are themselves time-stamped on the system axis - [Identity Resolution](/docs/concepts/identity) — `ERASE SUBJECT` uses the identity graph to find all affected rows - [SQL Reference](/docs/reference/sql) — `EXPLAIN POLICY`, `ERASE SUBJECT`, `PURPOSE` syntax ============================================================================== # Hybrid Search URL: https://relatadb.dev/docs/concepts/hybrid-search ============================================================================== # Hybrid Search Vector-only search misses exact keyword matches. BM25-only search misses semantic paraphrases. Identity-only search misses documents that don't contain the identity value directly. Relata runs all three signals and fuses them with reciprocal-rank fusion (RRF) — a rank-based combiner that requires no score calibration between signals that live on completely different scales. The search engine is custom-built. Not Tantivy, not Lucene: integer posting lists with token interning, q-gram prefix/fuzzy search, reverse-trigram suffix search, hand-rolled Snowball-inspired stemmers (en/fr/de/es/pt/it/nl/sv/no/da/fi/hu/ro + Russian Cyrillic, Turkish, Arabic), subword tokenization (camelCase/digit splitting), stop words, synonyms, highlighting, and faceted search (full-match-set counts). The vector engine is a custom HNSW graph in memory, with a DiskANN warm tier for indexes that exceed RAM. Identity matching reuses the `IdentityIndex` built by SmartIngest. ## The three retrieval signals | Signal | Engine | What it finds that the others miss | |---|---|---| | BM25 full-text | Custom inverted index (integer posting lists) | Exact jargon, codes, account numbers typed verbatim | | Vector similarity | Custom HNSW + DiskANN warm tier | Semantic paraphrase, synonym matches, language variation | | Identity matching | `IdentityIndex` MV | The same entity referenced under a different surface form in a different source | Each signal independently produces a ranked list. RRF combines them: the final score for a result is the sum of `1 / (60 + rank_i)` across every signal that surfaced it, where `rank_i` is the 1-based position in that signal's list. Results that appear in multiple signals naturally bubble to the top. Results unique to one signal still appear — they are not discarded. ## HYBRID_SEARCH in SQL `HYBRID_SEARCH` is a top-level query form (like `LOOKUP_IDENTITY`), not a `SELECT` modifier. The grammar is `HYBRID_SEARCH FROM QUERY '' LIMIT `: ```sql -- Basic hybrid: BM25 + vector over a single type PURPOSE 'investigation' HYBRID_SEARCH FROM Document QUERY 'terror finance' LIMIT 25 -- Governed-RAG alias (same pipeline, RAG-shaped surface) PURPOSE 'investigation' RAG_RETRIEVE FROM Document QUERY 'terror finance' LIMIT 25 ``` `HYBRID_SEARCH` runs BM25 over the row's indexed text fields and cosine similarity over the row's stored embedding. The two ranked lists are fused via RRF before results are returned. Optional trailing modifiers: - `RERANK` — re-score the top-K via a sidecar cross-encoder - `METRIC ` — override the vector distance metric - `WEIGHTS ` — per-query fusion weights for the graph, BM25, and vector channels ## MATCH operator `MATCH` is the pure-BM25 predicate form. Use it when you want keyword filtering without the vector overhead, or when you need one of the specialised modes: ```sql -- Default: token-level BM25 posting list lookup SELECT * FROM Document WHERE MATCH(title, 'financial fraud') -- Phrase: words in this exact order, adjacent positions SELECT * FROM Document WHERE MATCH(body, 'money laundering', PHRASE) -- Fuzzy: edit-distance expansion — catches 'recieve', 'finacial', etc. SELECT * FROM Post WHERE MATCH(text, 'recieve', FUZZY) -- Stemmed: stem-reduced token match (covers 'laundering', 'laundered', 'launder') SELECT * FROM Document WHERE MATCH(body, 'launder', STEMMED) -- Suffix: reverse-trigram index — find strings ending with a pattern SELECT * FROM Account WHERE MATCH(account_number, '4242', SUFFIX) ``` | Mode | Index used | Best for | |---|---|---| | Default | BM25 integer posting list | Standard keyword search | | `PHRASE` | Positional index | Exact phrase matching | | `FUZZY` | Q-gram + edit-distance expansion | Typo tolerance | | `STEMMED` | stemmed posting list | Morphological variants | | `SUFFIX` | Reverse-trigram index | Suffix matching (e.g. card last 4) | ## The /search endpoint The universal HTTP search API exposes the full three-signal pipeline with typeahead support, faceting, and hit highlighting. The Python SDK's `SearchBuilder` is the recommended entry point: ```python from relata import RelataClient, SearchBuilder with RelataClient(url, bearer_token=token, purpose="investigation") as client: results = client.search( SearchBuilder("money laundering correspondent banking") .types(["Transaction", "Document", "Alert"]) .limit(25) .facet("source") .facet("risk_tier") .highlight(True) ) for hit in results.hits: print(f"{hit.score:.3f} [{hit.type}] {hit.highlight or hit.id}") # Facet counts for facet, counts in results.facets.items(): print(facet, counts) ```
HTTP equivalent ```bash curl -X POST http://127.0.0.1:9090/search \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \ -d '{ "query": "money laundering correspondent banking", "types": ["Transaction", "Document", "Alert"], "limit": 25, "facets": ["source", "risk_tier"], "highlight": true }' ``` Response envelope: `{ "hits": [...], "facets": {...}, "took_ms": 12 }`. Each hit carries `id`, `type`, `score`, and `highlight` (the matched snippet with `` tags).
## BM25 engine internals The engine was built to avoid two Tantivy/Lucene constraints: floating-point score normalization overhead and the inability to do integer-keyed prefix/suffix matching efficiently. | Feature | How it works | |---|---| | Posting lists | Integer RowId postings — tokens are interned to 32-bit IDs; posting lists are `Vec` | | BM25 params | k1 = 1.2, b = 0.75 (standard Okapi defaults) | | Prefix search | Q-gram inverted index (bigrams + trigrams) | | Fuzzy search | Q-gram overlap + edit-distance expansion on candidates | | Suffix search | Reverse-trigram index (strings stored reversed, prefix-searched) | | Stemming | Hand-rolled Snowball-inspired suffix strippers (en/fr/de/es/pt/it/nl/sv/no/da/fi/hu/ro + Russian Cyrillic, Turkish, Arabic) | | Stop words | Per-language lists, applied at index time and query time | | Synonyms | Configurable per-tenant synonym maps; applied at query time | | Highlighting | Match-position tracking; returns snippet with `` tags | | Faceted search | Per-facet posting list aggregation with count rollup | | Custom ranking | Boost functions configurable per type and per field | ## Vector search Vectors are stored in a custom HNSW graph (`crates/relata-storage/src/vector.rs`). The primary distance metric is cosine similarity. For indexes that exceed available RAM, a DiskANN warm tier (`vector_diskann.rs`) pages segments to object-store-backed `PagedAnnIndex` buckets — cold vectors are re-loaded on demand without a full index rebuild. The IVF cold tier (`RELATA_VECTOR_COLD_RESIDENT_MAX`, default 100,000 vectors) stages incoming vectors in paged buckets before spilling them to the object store. This means large write batches do not stall while the HNSW graph grows. ## ACL-aware vector search ACL filtering on vector results uses an adaptive strategy: - **Broad principal** (allowed rows > 25% of index): post-filter — score all candidates, then apply the ACL bitmap to discard denied results. - **Narrow principal** (allowed rows ≤ 25% of index): pre-filter — score only the allowed slots. This prevents the recall cliff that naive post-filtering causes when most of the index is off-limits. The 25% threshold is tuned so that wide-access principals (analysts with most-row access) keep the fast path, while compartment-restricted principals get correct recall even on selective ACLs. ## Search presets `RELATA_SEARCH_PRESET` controls the BM25 fuzzy expansion aggressiveness. It applies uniformly to `MATCH`, `HYBRID_SEARCH`, and `/search` — no per-query override needed. | Preset | Behaviour | Use when | |---|---|---| | `strict` | Minimal fuzzy expansion; exact matches dominate | High-precision queries over structured data | | `balanced` | Moderate expansion (default) | General investigation and discovery | | `lenient` | Aggressive expansion; maximises recall | Broad exploration over noisy or user-generated text | Change it at runtime without restarting the server: ```bash RELATA_SEARCH_PRESET=lenient cargo run -p relata-cli -- serve ``` ## Query result cache Repeated identical search queries (same query string, same types, same principal) are served from the result cache (`relata-query::result_cache`) without re-running the pipeline. Cache-aside reads use `WITH CACHE` in SQL: ```sql PURPOSE 'fraud' HYBRID_SEARCH FROM Document QUERY 'fraud indicators' LIMIT 25 WITH CACHE ``` Optional `WITH CACHE` knobs: `TTL `, `STALENESS `, and `BYPASS` (skip the cache for fresh results): ```sql PURPOSE 'fraud' HYBRID_SEARCH FROM Document QUERY 'fraud indicators' LIMIT 25 WITH CACHE BYPASS ``` The cache is invalidated on any write to the types covered by the query. ## Agent memory recall The `recall` cognitive verb runs the same three-signal pipeline, then applies a fourth re-scoring pass: an additive relevance/recency/forgetting blend gated by `confidence × class_weight`. There is no separate search engine for agent memory — memory items are rows in the same store. See [Agent Memory](/docs/concepts/agent-memory). ## See also - [Identity Resolution](/docs/concepts/identity) — the third retrieval signal; how `IdentityIndex` is built - [Agent Memory](/docs/concepts/agent-memory) — `recall` runs on the same pipeline - [Governance](/docs/concepts/governance) — ACL-aware pre-filtering for narrow principals - [Query Engine](/docs/architecture/query-engine) — how the planner lowers `HYBRID_SEARCH` and `MATCH` - [SQL Reference](/docs/reference/sql) — full operator syntax and `WITH CACHE` options ============================================================================== # Identity Resolution URL: https://relatadb.dev/docs/concepts/identity ============================================================================== # Identity Resolution The same person appears in your data under a phone number in one system, an email address in another, and a government ID in a third. Joining those records manually is error-prone, slow, and breaks the moment a new source is added. Relata solves this at the storage layer: every ingested value is checked against a catalogue of 76 canonical identity types, linked into an `IdentityIndex` materialised view, and made queryable with SQL operators that treat entity resolution as a first-class operation — not a post-processing step. ## The `Identity` datatype An `Identity` wraps two things: a `CanonicalKind` (one of 76 enum variants) and a deterministic binary encoding of the validated value. The binary encoding ensures that `+1-800-555-0100` and `18005550100` resolve to the same bytes — no normalisation code on your side. The 76 shipped kinds span: | Domain | Examples | |---|---| | Contact | Email, phone (E.164), MSISDN | | Financial | IBAN, LEI, PAN (India), Luhn (cards / IMEI), GSTIN, BTC address | | Transport | MMSI (maritime), IMO, ICAO24 (aircraft), ICAO airport code, NORAD, VIN (vehicles), licence plate | | Device | IMEI | | Network | IPv4, IPv6, MAC address | | Crypto | SHA-256 digest | | Payment | UPI handle, mobile-money rails (M-Pesa, MTN, Airtel, GCash, …), GCC/MENA rails (Sadad, KNET, CliQ, …) | | Social | TikTok, Facebook, Instagram, LinkedIn, Snapchat, Telegram | | ICS/OT | Modbus unit ID, OPC-UA node, DNP3 address, IEC 61850, Siemens S7 | | Trade / regulatory | ECCN, CAS RN, frequency band, call sign, FARA registration | | Other | DateTime, GeoPoint, perceptual / video / audio fingerprints, embedding field descriptor | The enum is `#[non_exhaustive]` — new kinds are added without breaking existing code. The canonical list is the source of truth: `crates/relata-canonical/src/lib.rs`. > A few kinds (GSTIN, BTC address, FARA) ship validators but no SmartIngest detection gate — they round-trip through the type system but are not auto-detected from free text. See `docs/src/end-users/limits.md` for the honest per-kind status. ## How identity links are built — SmartIngest SmartIngest (`relata-detect`) runs at write time, not query time. When a row is ingested, it scans text fields in two phases, per token: 1. **Eager gate** — cheap pattern check (length, prefix, character class). Rejects obvious non-matches immediately. 2. **Lazy validate** — full canonical parser with checksum verification (Luhn, mod-97, Base58Check, …). Only runs when the eager gate passes. Both phases run synchronously on the write path and produce `Identity` rows. The heavier work — turning those identifiers into graph structure (`IdentityIndex` / `IdentityLink` MVs) — is handed off to a non-blocking enrichment queue (`relata-jobs::enrichment_queue`) drained in batches by a background task. The result is that by the time a query asks `RESOLVE_IDENTITY(email)`, the link is already there — no join-time detection. ### Controlling which packs load SmartIngest is divided into detector packs. Three are on by default; the rest are opt-in because they add CPU cost and some produce false positives on general text: ```bash # Default (on at startup) RELATA_DETECT_PACKS=network,contact,crypto # Add financial + payment detection RELATA_DETECT_PACKS=network,contact,crypto,financial,payment # All packs RELATA_DETECT_PACKS=all # Disable all auto-detection RELATA_DETECT_PACKS=none ``` | Pack | Detects | |---|---| | `network` | IPv4, IPv6, MAC | | `contact` | Phone (E.164), email | | `crypto` | SHA-256 digests | | `financial` | IBAN, LEI, PAN (India), Luhn (cards / IMEI) | | `payment` | UPI, mobile-money rails (M-Pesa, MTN, Airtel, …), GCC/MENA rails (Sadad, KNET, CliQ, …) | | `social` | TikTok, Facebook, Instagram, LinkedIn, Snapchat, Telegram | | `transport` | ICAO24, MMSI, ICAO airport, IMO, NORAD | | `device` | IMEI, VIN | | `ics` | Modbus, OPC UA, DNP3, S7, IEC 61850 | > `relata detect ""` at the CLI always runs all packs regardless of `RELATA_DETECT_PACKS`. That env var controls only the HTTP ingest and per-row write paths. ## SQL operators ### Lookup by value ```sql -- Is this phone number known to the system? SELECT * FROM LOOKUP_IDENTITY('+919876543210') -- Returns: kind, canonical_value, linked_entity_ids[] ``` ### Resolve an identity to its canonical form or cluster ```sql -- Canonical form: what is the authoritative surface for this email? SELECT * FROM RESOLVE_IDENTITY('alice@example.com') -- Cluster: every identity value linked to the same entity SELECT * FROM RESOLVE_IDENTITY('alice@example.com', MODE => 'cluster') -- Column projection: resolve inline in a query SELECT name, RESOLVE_IDENTITY(email) AS canonical_email FROM Person WHERE country = 'IN' ``` ### Resolution modes | Mode | What it returns | |---|---| | `cluster` (default) | Every identity value the entity is linked to | | `canonical` | The single authoritative surface form (passthrough of the raw value) | | `fuse` | Runs the registered `EnrichmentRule` chain — returns a descriptive error if no rules are registered, no silent fallback | ### Graph traversal between identities Once entities are linked through shared identities, you can walk the resulting graph: ```sql -- All paths between two entities, up to 4 hops SELECT * FROM PATHS_BETWEEN('person-123', 'org-456', max_hops => 4) -- Identity verdict — do two identifiers resolve to the same person? SELECT * FROM SAME_IDENTITY('person-123', 'org-456') -- Full cluster for a given entity SELECT * FROM IDENTITY_CLUSTER('person-123') ``` `PATHS_BETWEEN` uses the graph engine's bidirectional BFS / DFS traversal for path enumeration (`traversal.rs:388`). Each returned path includes the intermediate identity values and the types they were found in. (PLL distance labeling is wired into `GRAPH_SSSP` (`algo => 'pll'`) and `GRAPH_DIJKSTRA` as a reachability pre-check — not into `PATHS_BETWEEN`.) ### A worked example — linking a phone to a transaction ```sql -- Find all transactions associated with a phone number, -- even if the phone wasn't stored directly on the Transaction row SELECT t.id, t.amount, t.valid_from FROM Transaction t JOIN IDENTITY_CLUSTER(RESOLVE_IDENTITY('+919876543210')) ic ON t.actor_id = ANY(ic.entity_ids) ORDER BY t.valid_from DESC LIMIT 20 ``` ## Batch detection For pipelines that ingest text in bulk, the enrichment queue processes detection in chunks of `RELATA_DETECT_BATCH_SIZE` (default `256`) rows, reusing a single hit buffer per chunk. Results are identical to per-row detection — the batch size only affects throughput. ## Python SDK ```python from relata import RelataClient, IdentityClient with RelataClient(url, bearer_token=token, purpose="identity-match") as client: id_client = IdentityClient.from_client(client) # Look up an entity by identity value result = id_client.lookup("+919876543210") # Get the full cluster cluster = id_client.cluster("alice@example.com") # GDPR erasure — shreds all rows linked to this subject receipt = id_client.erase_subject("person-42", certify="governed-tombstone") ``` ## Active learning — tune the resolver from human feedback Identity resolution isn't a black box you have to accept. When `RESOLVE_IDENTITY` gets a pair wrong (fuses two distinct people, or refuses to merge the same person), you can submit a human judgement and the resolver takes a correction step on its feature weights — turning "the resolver got this wrong" into something the system **learns from**. Labels persist across restarts in `${RELATA_DATA_DIR}/active_learning.json`. ### Label a pair (match / no-match) ```bash curl -X POST http://127.0.0.1:9090/identity/label \ -H 'Authorization: Bearer ' -H 'Content-Type: application/json' \ -d '{"left": "+44 7700 900123", "right": "07700 900123", "is_match": true}' ``` ```python # Python SDK client.identity_client.label("+44 7700 900123", "07700 900123", is_match=True) client.identity_client.label("alice@x.com", "bob@y.com", is_match=False) # hard negative ``` ### Ask for the pairs the resolver is least sure about The uncertainty endpoint surfaces the candidate pairs whose labels would most improve the resolver — active-learning style. Hand-label these first for the highest return on effort: ```bash curl 'http://127.0.0.1:9090/identity/uncertainty?candidates=id-a|id-b,id-c|id-d&k=5' \ -H 'Authorization: Bearer ' ``` ```python # The resolver's top "I'm not sure — please tell me" pairs uncertain = client.identity_client.record_uncertainty(...) ``` ### Tips & takeaways - **Hard negatives matter as much as matches.** Label genuine distinct-entity pairs (`is_match: false`) — they prevent over-merging, which is the harder failure to detect later. - **Batch-label from a review UI.** Run a queue of `uncertainty` candidates past an analyst, push the judgements back via `label`. A few hundred labels typically moves resolution quality noticeably. - **Labels are governed.** Each label is audit-logged with the principal and purpose — defensible "who taught the resolver what" history. - **Not a replacement for canonical types.** Active learning corrects *ambiguous* matches (fuzzy phone, partial name). It does not override deterministic canonical-type fusions (same verified IBAN = same entity, always). ## What is not yet shipped - Detection gates for GSTIN, FARA, and a handful of government ID types are not wired into SmartIngest — values of these kinds validate correctly but are not auto-detected from free text. - The `fuse` resolution mode requires at least one enrichment rule registered via `POST /ontology/enrichment-rules` before it is useful. ## See also - [Governance](/docs/concepts/governance) — cell masking can redact identity values for users without the right ACL - [Hybrid Search](/docs/concepts/hybrid-search) — identity matching is the third retrieval signal in hybrid queries - [SQL Reference](/docs/reference/sql) — full operator signatures - [Limits](/docs/reference/limits) — per-kind validator and detection gate status ============================================================================== # Concepts URL: https://relatadb.dev/docs/concepts ============================================================================== # Concepts The ideas that make RelataDB different from a regular database. Each page below covers *why* a capability exists, the model behind it, and how it shows up in queries — without reproducing the full API reference. ## In this section - [AI in RelataDB](/docs/concepts/ai-in-relatadb) — the agent-native surface: memory, governed tools, and multi-agent coordination - [Ontology & Schema](/docs/concepts/ontology) — declare types, not tables; how `ObjectType` / `EventType` / `LinkType` drive storage and the planner - [Bi-Temporal](/docs/concepts/bitemporal) — every row carries `valid` and `system` time; rewind the database with `AS OF` - [Identity](/docs/concepts/identity) — deterministic canonical-identifier resolution; how the graph forms itself - [Governance](/docs/concepts/governance) — cell-level ACL and `PURPOSE` compiled into the scan predicate - [Provenance](/docs/concepts/provenance) — tamper-evident, hash-chained lineage on every fact - [Agent Memory](/docs/concepts/agent-memory) — the cognitive-memory model: remember · recall · justify · consolidate · forget - [Hybrid Search](/docs/concepts/hybrid-search) — BM25 + HNSW vector + identity fusion, ranked in one query - [Federation](/docs/concepts/federation) — the roadmap for federated queries across RelataDB instances - [Relata vs Others](/docs/concepts/relata-vs-others) — where RelataDB fits against Postgres, Neo4j, Pinecone, and agent-memory tools (and when not to use it) See also: [Architecture](/docs/architecture) for how these concepts are implemented, and the [Reference](/docs/reference) for the exact verbs, arguments, and response shapes. ============================================================================== # Ontology & Schema URL: https://relatadb.dev/docs/concepts/ontology ============================================================================== # Ontology & Schema RelataDB uses a schema-as-code ontology model. Types are declared as code, versioned via git branches, and enforced at write time. ## Types and properties Each type has a set of typed properties: ```bash curl -X POST http://localhost:9090/types \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "Person", "properties": [ {"name": "id", "type": "string", "required": true}, {"name": "name", "type": "string"}, {"name": "email", "type": "email"}, {"name": "birth_date", "type": "datetime"} ] }' ``` ### Supported property types | Type | Validation | |---|---| | `string`, `int`, `float`, `bool` | Basic | | `email`, `phone`, `iban`, `mmsi`, `vin`, `imei` | 76 canonical validators | | `datetime` | ISO 8601 → i64 ns UTC | | `uuid` | 128-bit UUID | | `[]string`, `[]int`, `[]float` | Arrays | | `[N]f32`, `[N]f16`, `[N]i8` | Vector types | ## Computed columns Define computed properties that derive their value from other fields: ```sql ALTER TYPE Person ADD COMPUTED full_name = CONCAT(first_name, ' ', last_name); ``` ## State-machine constraints Types can declare state machines that constrain valid transitions: ```json { "name": "Case", "state_machine": { "field": "status", "transitions": [ {"from": "open", "to": "investigating"}, {"from": "investigating", "to": "closed"}, {"from": "closed", "to": "reopened"} ] } } ``` An invalid transition (e.g. `open` → `closed`) is rejected at write time. ## Schema branches (git-branched ontology) Schema changes can be developed on a branch without affecting production. (Schema branches fork the **ontology only**; to fork an entire data namespace — every type and every row, in constant time — use `BRANCH ... FROM ...` / `POST /v1/namespaces/{name}/branch`. See [Branching & Namespaces](/docs/reference/branching) for the distinction.) ```bash # Create a schema branch curl -X POST http://localhost:9090/schema/branches/dev-schema \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \ -d '{"from": "main"}' # Make changes on the branch... # Merge when ready # Delete if discarded curl -X DELETE http://localhost:9090/schema/branches/dev-schema \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" ``` ## Schema evolution Add or drop properties without downtime: ```bash curl -X PATCH http://localhost:9090/types/Person/schema \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \ -H "Content-Type: application/json" \ -d '{"add": [{"name": "department", "type": "string"}]}' ``` Existing rows get `null` for new properties. Rows are schema-flexible post-creation. ## See also - [Data model](/docs/architecture/data-model) — bi-temporal rows - [Identity](/docs/concepts/identity) — canonical type validators - [SQL reference](/docs/reference/sql) — ALTER TYPE, CREATE TYPE ============================================================================== # Provenance URL: https://relatadb.dev/docs/concepts/provenance ============================================================================== # Provenance Chain of custody is not something you add to a database after a finding. By the time a compliance team asks "who wrote this row, from what source, and has it been tampered with?", the answer either exists in the storage layer or it does not exist at all. Relata records provenance on every write: who, what source, what purpose, and a hash that chains to the previous commit. The result is a tamper-evident ledger where any row can be traced to its origin, and any logged exhibit can be re-derived byte-identically. ## WITH PROVENANCE A trailing SQL modifier that attaches a lineage object to every returned row: ```sql -- Basic provenance on a type query SELECT id, amount, currency FROM Transaction LIMIT 50 WITH PROVENANCE -- Combine with temporal travel — what did we know on 1 March, with full lineage? SELECT id, amount, currency FROM Transaction AS OF SYSTEM TIME '2024-03-01T00:00:00Z' LIMIT 50 WITH PROVENANCE ``` `WITH PROVENANCE` must come after `LIMIT`. It attaches a per-row provenance object (alongside the row data) carrying: | Field | Meaning | |---|---| | `source` | The source system that produced the row (e.g. `kafka:topic-payments`, `http-ingest`, `csv-upload`) | | `method` | Collection method (e.g. `bulk-ingest`, `realtime-stream`, `derived-update`) | | `confidence` | The assertion's confidence in `[0.0, 1.0]` | | `recorded_at` | System-time when the assertion was recorded (i64 ns UTC) | | `derived_from` | Hex `ProvenanceRef` of the source assertion this row derives from, or `null` for flat genesis attribution | This metadata is not stored redundantly per row — it is derived from the assertion chain at query time. The principal and purpose for each write live in the parallel audit entry (see [Governance](/docs/concepts/governance)). ## Commit manifests Every write produces a commit manifest entry containing: - A monotonically-increasing sequence number - The commit's system-time (`committed_at`, i64 ns UTC) - The row count in the batch - A `batch_fingerprint` — SHA-256 over the row-level provenance refs concatenated in insertion order (order-dependent, so re-ordering a batch breaks the chain) - A `prev_ref` to the previous manifest entry (genesis for the first) The principal and purpose for each write are recorded in the parallel audit entry (`AuditEntry`), which carries its own principal, purpose, cost, and result-fingerprint fields and is itself hash-chained. Together the manifest chain and the audit chain give you "who wrote what, when, under which purpose, and from what source." The chaining means that any retroactive modification — to any row, at any point in history — changes the hash of that manifest, which cascades forward to invalidate every subsequent manifest hash. The chain can be verified at any time: ```bash # Full chain verification + node health check relata doctor # Quick count + validity flag curl http://localhost:9090/audit/count # → { "entries": 4821, "chain_valid": true } ``` ## EXPLAIN_REPLAY `EXPLAIN_REPLAY` re-derives a specific logged exhibit's seal byte-identically. This is the audit replay path used in legal and regulatory contexts: given an exhibit ID and a sequence number, Relata reconstructs the exact bytes that were sealed, so an independent auditor can confirm the conclusion without trusting the database's current state. ```sql -- Re-derive exhibit-7, sequence point 5 EXPLAIN_REPLAY('exhibit-7', SEQ => 5) ``` The replay path: 1. Reads the manifest chain from the anchor point to SEQ 5. 2. Re-derives each referenced blob by hash. 3. Confirms the chain is intact end-to-end. 4. Returns the reconstructed exhibit seal and a `chain_valid: true` confirmation. If any blob is missing or any hash does not match, replay returns an explicit failure with the first broken link identified. ## Content-addressed blobs Large payloads (media, documents, binary fields) are stored out-of-row in a content-addressed blob store (SHA-256). The same bytes stored twice are deduplicated automatically — the `prov_hash` points to the same blob. The blob store wires into three ingest paths: - **S3 door**: objects at or above `RELATA_S3_BLOB_THRESHOLD_MB` (default 4 MiB) are stored out-of-row - **Media ingest**: `ingest_media` MCP tool accepts base64 payloads; the blob is stored and the hash recorded in the manifest - **IVF cold tier**: large vector indexes spill to the object-store-backed `PagedAnnIndex`, with each segment referenced by hash ## Provenance on agent memory When an agent stores a memory item with `remember`, the resulting `MemoryItem` row carries the same provenance fields as any `WITH PROVENANCE` request: - `source`: the write path that produced it (e.g. `mcp:remember`, `http:/memory/remember`) - `method`: the collection method - `recorded_at`: the system-time stamp (i64 ns UTC) - `derived_from`: the upstream assertion the memory was derived from (or `null` for a fresh genesis write) The principal and purpose are recorded in the parallel audit entry, not on the row itself. The `justify` cognitive verb retrieves the complete provenance + audit trail for any memory item — useful when an agent needs to explain how it reached a conclusion: ```bash # Via MCP { "method": "tools/call", "params": { "name": "justify", "arguments": { "id": "", "purpose": "audit" } } } # Via HTTP curl -X POST http://localhost:9090/memory/justify \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \ -d '{"id":"","purpose":"audit"}' ``` The response includes the source rows that the memory item was derived from, the chain of cognitive operations that produced it, and the manifest hashes covering each step. ## Python SDK — AuditClient ```python from relata import RelataClient, AuditClient with RelataClient(url, bearer_token=token, purpose="compliance_review") as client: audit = AuditClient.from_client(client) # Paginated audit log, filtered by principal and time window for page in audit.entries(filter={ "principal": "api-user-finance", "since": "2026-01-01T00:00:00Z", "until": "2026-06-30T23:59:59Z", }): for entry in page: print(entry.timestamp, entry.purpose, entry.prov_hash) # Signed receipt for an exhibit (court-ready) receipt = audit.signed_receipt(exhibit_id="exhibit-7") print(receipt.chain_valid, receipt.seal_hash) # PDF export for regulatory submission pdf_bytes = audit.export_pdf(filter={"case_id": "case-42"}) open("audit-case-42.pdf", "wb").write(pdf_bytes) ``` ## See also - [Governance](/docs/concepts/governance) — the ABAC policy layer that sits in front of every write - [Bi-Temporal Model](/docs/concepts/bitemporal) — provenance rows are themselves bi-temporal; `system_from` records when each manifest entry was committed - [Agent Memory](/docs/concepts/agent-memory) — `justify` returns the provenance chain for any memory item - [SQL Reference](/docs/reference/sql) — `WITH PROVENANCE`, `EXPLAIN_REPLAY` syntax ============================================================================== # Relata vs Others URL: https://relatadb.dev/docs/concepts/relata-vs-others ============================================================================== # Relata vs Others The fastest way to understand an unfamiliar tool is to map it to familiar ones. RelataDB is a **database** — you ingest rows and query them. What makes it unusual is *what it does for you automatically, inside the database*: it standardizes identities, connects them into a graph, time-stamps everything twice, notarizes every fact, and enforces cell-level access rules on read. Tools you know each do **one** of those. RelataDB does them **together, by default**. ## Side-by-side | Capability | Postgres / MySQL | Graph DB (Neo4j) | Vector DB (Pinecone) | Agent memory (Mem0/Zep) | **RelataDB** | |---|:-:|:-:|:-:|:-:|:-:| | Store & query rows/objects | ✅ | ⚠️ | ⚠️ | ⚠️ | ✅ | | Auto-validate & standardize identifiers (76 kinds) | ❌ | ❌ | ❌ | ❌ | ✅ | | Connect same-entity across sources into a graph | ❌ | manual | ❌ | LLM-guess | ✅ deterministic | | Two timelines on every fact (time travel) | ❌ | ❌ | ❌ | ❌ | ✅ | | Tamper-evident provenance on every fact | ❌ | ❌ | ❌ | ❌ | ✅ | | Cell-level access control in the query path | ❌ | partial | ❌ | ❌ | ✅ | | Vector / semantic search | pgvector | plugin | ✅ | ✅ | ✅ | | Multi-protocol doors (pgwire/S3/Redis/Mongo/…) | pgwire | ❌ | ❌ | ❌ | ✅ (8+) | The pattern: every other tool leaves **identity standardization, history, provenance, and governance as your problem**. RelataDB builds them into the engine so your team spends time on *analysis*, not *plumbing*. ## vs a regular database (Postgres) A regular database stores what you put in, exactly as you put it in. If `+44 7…` and `07…` land in two tables, they're two unrelated strings forever — *you* build the cleaning, matching, and dedup. History usually means "latest value wins" — the old value is gone unless you built a slowly-changing-dimension rig yourself. Provenance means a `created_at` column you hope people fill in. RelataDB does all of that as the default behavior of the write and read paths. You don't build it; you configure it. > **Use Postgres when** your data is clean, low-stakes, and answers don't need to be defensible. > **Use RelataDB when** the data is messy, sensitive, and the answer must be right and provable. ## vs a graph database (Neo4j) A graph database is excellent at storing nodes and edges and traversing them — once *you* decide what the nodes and edges are. It won't tell you that two rows are the same person. RelataDB builds the graph **for you** out of standardized identities: any two records sharing a validated identifier are automatically linked, and co-occurring identities create inferred edges. You query a graph that formed itself. (You can even query RelataDB *through* Cypher and Bolt — it speaks graph-db dialects natively while standardizing identities underneath.) ## vs a vector database (Pinecone / pgvector) A vector database finds *semantically similar* rows — "documents close in meaning." It has no concept of *identity*, *correctness*, or *history*. It's a similarity index. RelataDB has vector/hybrid search built in (TurboVec + HNSW + DiskANN), but similarity is **one signal among many**. The primary signal is *exact validated identity* — far stronger than "sort of close" when you need to prove two records are the same entity. ## vs agent-memory tools (Mem0 / Zep / Cognee) This is the comparison most AI builders care about. These tools give an LLM agent a memory: store facts, retrieve relevant ones later. **Where they overlap:** all store "memories" and retrieve by relevance. **Where RelataDB differs (the wedge):** | | Mem0 / Zep / Cognee | **RelataDB** | |---|---|---| | How entities are extracted | **LLM-driven** (lossy, hallucination-prone, different every run) | **Deterministic checksum parsers** (zero hallucination, byte-identical every run) | | Can it prove where a memory came from? | Weak / none | Tamper-evident provenance per memory | | Can it recall "what I knew on Tuesday"? | No | Yes — bi-temporal `AS OF` | | Multi-tenant isolation | Usually single-tenant | Per-tenant encryption + cell-level ACL | | Reproducible? | "Depends on the model" | Court-grade replayable | **Rule of thumb:** *if hallucinated or unexplainable memory is tolerable, those tools are simpler. If it isn't — regulated, legal, medical, financial AI — RelataDB is the safe choice.* See [Agent Memory](/docs/concepts/agent-memory) for the full surface. ## vs a data warehouse / lakehouse (Snowflake / Databricks) Those excel at large-scale analytics over **already-structured** data. They assume cleaning, identity resolution, and governance happen *upstream* in your ETL. RelataDB does that work **in the engine**. They're often complementary — RelataDB as the governed identity/resolution layer, the warehouse as the analytics surface. ## When NOT to use RelataDB - Your data is clean, public, and low-stakes → a regular database is simpler and cheaper. - You only need semantic search over documents → a vector DB is lighter. - You want an AI agent *runtime* (the agent itself) → RelataDB is the memory layer, not the agent. - You have no governance, provenance, or history requirements → you'd pay for features you won't use. RelataDB earns its keep **precisely when** standardization, history, proof, and governance are non-negotiable. ============================================================================== # Cluster Setup URL: https://relatadb.dev/docs/deployment/cluster ============================================================================== # Cluster Setup `RELATA_PROFILE=cluster` is **alpha** (see [Deployment](/docs/deployment) — the same caveat applies here: petabyte / very-high-cardinality sharding still needs more work). This page walks through what it actually takes to get multiple cluster-profile nodes talking to each other **outside of Kubernetes** — local dev, bare metal, or just understanding what the [Helm chart](/docs/deployment/kubernetes) is doing under the hood. Every requirement below was verified against a real 3-node boot (not just read off source) as of this writing. Cluster is under active development — re-check against `crates/relata-cli/src/serve.rs` before relying on exact error strings in a script. ## Topology & roles A cluster node advertises a `CLUSTER_ROLE`: `coordinator` | `reader` | `writer` | `indexer` (default `coordinator`; an unrecognized value fails startup). | Role | Intent | |---|---| | `coordinator` | Query planning + request routing entry point | | `reader` | Read-heavy query execution | | `writer` | Write ingest + WAL | | `indexer` | Registered as a dedicated index-builder node in the cluster topology for fan-out routing purposes | Important nuance: `CLUSTER_ROLE` is a **routing/registry hint**, not a hard access gate. Every node — regardless of role — still exposes the full HTTP surface (`/query`, `/ingest`, `/health`, etc.). The role tells the cluster registry how to characterize a node for cross-node routing decisions (e.g. `nodes_of_role(Reader)`); it does not, by itself, make a `reader` node reject writes or a `writer` node reject reads. Don't confuse `CLUSTER_ROLE` with the separate `RELATA_ROLE` env var (`query` | `indexer` | `both`, default `both`) — that one controls where deferred secondary/vector index-*maintenance* work is applied, and is independent of cluster fan-out routing entirely. If you only set one, set `CLUSTER_ROLE`. Data is distributed across nodes by consistent hashing (`RELATA_CLUSTER_SHARDS`, default 8 shards) — every node must derive the **same** partition key, which is what `RELATA_CLUSTER_SEED` (below) is for. ## Connecting clients to a cluster — one address, not a seed list Relata is **smart-server, dumb-client** (the opposite of MongoDB's architecture). The cluster does fan-out **internally** — you point any client at **one stable address** in front of the cluster, the coordinator routes to peers over gRPC (`QueryShard`), and results are merged. Your client never learns about the other nodes. **You do NOT use `mongodb+srv://`** — that's a MongoDB Atlas DNS-SRV seed-discovery mechanism, and Relata doesn't implement it (or replica-set `hello`/`isMaster` handshake). You don't need it: standard drivers connect to a single host just fine, and a load balancer gives you HA/failover without driver-side seed discovery. ``` ┌────────────────────────┐ your client ──► LB / VIP / K8s Service ──► coordinator node └────────────────────────┘ │ └─► gRPC fan-out to peers (internal) ``` | Front-end | Works for | Notes | |---|---|---| | **Kubernetes `LoadBalancer` or headless `Service`** | all protocols | The Helm chart does this for you — one stable DNS name, kube-proxy load-balances across pods. | | **AWS NLB / GCP TCP LB / Azure LB** (L4 TCP) | Mongo, pg, redis, Bolt, ClickHouse, S3 | L4 — forward bytes, don't terminate TLS. | | **HAProxy (`mode tcp`) / nginx stream** | all | Health-check `GET /health/ready` on the HTTP port (9090), drop dead nodes. | | **DNS round-robin** | cheap HA | No health checks — last resort. | ```javascript // MongoDB — any official driver, single host. password = RELATA_BEARER_TOKEN const c = new MongoClient("mongodb://relata-vip:27017", { auth: { username: "relata", password: "" }, }); ``` ```bash # Postgres / pgvector / Redis / Neo4j / ClickHouse / S3 — same pattern, one VIP psql -h relata-vip -p 5433 -U relata # password = token redis-cli -h relata-vip -p 6379 -a cypher-shell -a bolt://relata-vip:7687 -u neo4j -p ``` **Failover:** the LB health-checks `GET /health/ready` (HTTP 9090) and drops any node returning non-200. Every node can serve reads; writes funnel through the planner regardless of which node received them. No driver change, no `+srv`, no replica-set name. > **One constraint:** the compat doors default to the **same port on every node**. Running multiple nodes on one host causes bind collisions (non-fatal WARN, that node loses the door). One node per host/container/pod is the supported topology — which is exactly what an LB in front assumes. See [Deploying Protocol Doors](/docs/deployment/protocol-doors) and [Compatibility & Doors](/docs/compatibility). ## Required env vars Every cluster-profile node needs all of the following. Names and defaults are current as of this writing (`crates/relata-cli/src/serve.rs`, `serve/cluster.rs`, `serve/admin_listener.rs`, `relata-cluster/src/scatter_gather.rs`). | # | Var | Same on every node? | Why | |---|---|---|---| | 1 | `RELATA_PROFILE=cluster` | yes | Selects the profile. | | 2 | `RELATA_CLUSTER_SEED=` | **yes** | Seeds the partition-key hash. Without it (or the `RELATA_PARTITION_KEY_K0`/`K1` pair), boot FATALs: `cluster profile requires a stable partition key`. A per-process random key means every node derives a different shard ring — silent data scatter. | | 3 | `RELATA_KMS_LOCAL_DEV=true` | yes (dev only) | Cluster (like `server`) defaults at-rest encryption **ON**, and refuses the committed dev-secret KMS fallback unless this flag is set. Confirmed by direct test: omitting it FATALs at startup with `RELATA_ENCRYPTION_AT_REST set but at-rest encryption init failed — refusing to start` / `RELATA_KMS_KEY_ARN is required in the 'cluster' profile`, **even with a purely local (no S3) data directory** — any on-disk persistence backend triggers the check, not just a remote object store. **Never use this flag in production** — set a real `RELATA_KMS_KEY_ARN` instead. | | 4 | `RELATA_PUBLIC_URL=http://:` | **no — unique per node** | This node's externally-reachable base URL, used to announce itself for cluster gossip. Without it, boot FATALs: `RELATA_PUBLIC_URL is unset but RELATA_PROFILE=cluster — this node cannot announce itself for cluster gossip`. Must match this node's own HTTP host:port. | | 5 | `NODE_ID=` | **no — unique per node** | Plain `NODE_ID`, **not** `RELATA_NODE_ID` (that's a different, unrelated var — it overrides the persistent deployment UUID shown in the startup banner). Setting both is harmless but only `NODE_ID` drives cluster identity. The `node-0`/`node-1` placeholder default is explicitly rejected: boot FATALs with `cluster profile requires a unique NODE_ID env var`. | | 6 | `CLUSTER_ROLE=coordinator\|reader\|writer\|indexer` | no | This node's role (see above). | | 7 | `CLUSTER_PEERS=http://host1:port1,http://host2:port2` | no (list differs per node) | Comma-separated **HTTP** URLs of the other nodes. The Helm chart's StatefulSet template actually generates one identical peer list containing *every* replica (including self) — the runtime tolerates a self-referential entry fine, so excluding self by hand (as in the example below) is the simpler/cleaner but not strictly required. | | 8 | `CLUSTER_AUTH_TOKEN=` | **yes** | Shared secret gating inter-node calls (`/internal/cluster/replicate`, `/internal/cluster/join`, `/internal/cluster/snapshot`) and the gRPC `QueryShard` fan-out RPC. Distinct from `RELATA_BEARER_TOKEN` (client-facing). See [gotchas](#cluster_auth_token-fails-silently-not-loudly) below — this is **not** a boot-time FATAL. | | 9 | `RELATA_HTTP_BIND=:` | no — unique per node on one host | **Combined** `host:port` string. See the [bind-var inconsistency gotcha](#three-different-bind-var-conventions) below. | | 10 | `RELATA_PORT=` | no | Used internally for cross-shard query routing; keep it equal to `RELATA_HTTP_BIND`'s port. | | 11 | `RELATA_GRPC_BIND=` (host only) + `RELATA_GRPC_PORT=` | **`RELATA_GRPC_PORT` should be the SAME across every node** | See the [gRPC scatter-gather gotcha](#relata_grpc_port-must-match-across-nodes-not-be-unique) below — this is the one most likely to bite you and it contradicts the naive "give every door a unique port" instinct. | | 12 | `RELATA_ADMIN_BIND=:` | no — unique per node on one host | Combined `host:port`. Must resolve to loopback (`127.0.0.0/8`/`::1`) — Zero-Trust control plane — a non-loopback value FATALs. | | 13 | `RELATA_PG_PORT=` | no — unique per node on one host | Defaults to `5433` for every node; a collision is a non-fatal WARN + the pgwire door disables itself on that node. | | 14 | `RELATA_BEARER_TOKEN=` | yes (typically) | The normal client-facing auth token, same as any other profile — distinct from `CLUSTER_AUTH_TOKEN`. | ## A tested local 3-node example This exact recipe was booted end-to-end while writing this page: all three nodes reported `200` on `/health`, and a write via `/ingest` on the writer replicated to the other two nodes over HTTP with **no peer-replication errors** in any node's log. ```bash BIN=/path/to/relata # or `cargo run -p relata-cli --release -- serve` COMMON=( RELATA_PROFILE=cluster RELATA_CLUSTER_SEED=my-local-cluster-seed # any string — identical on all 3 RELATA_KMS_LOCAL_DEV=true # local/dev ONLY — never in production CLUSTER_AUTH_TOKEN=shared-cluster-secret # identical on all 3 RELATA_BEARER_TOKEN=my-client-token # identical on all 3 (client-facing) ) # node0 — coordinator env "${COMMON[@]}" \ NODE_ID=node0 CLUSTER_ROLE=coordinator \ CLUSTER_PEERS=http://127.0.0.1:29081,http://127.0.0.1:29082 \ RELATA_PUBLIC_URL=http://127.0.0.1:29080 \ RELATA_HTTP_BIND=127.0.0.1:29080 RELATA_PORT=29080 \ RELATA_GRPC_BIND=127.0.0.1 RELATA_GRPC_PORT=29180 \ RELATA_ADMIN_BIND=127.0.0.1:29280 RELATA_PG_PORT=29380 \ RELATA_DATA_DIR=/tmp/relata-cluster/node0 \ "$BIN" serve & # node1 — reader env "${COMMON[@]}" \ NODE_ID=node1 CLUSTER_ROLE=reader \ CLUSTER_PEERS=http://127.0.0.1:29080,http://127.0.0.1:29082 \ RELATA_PUBLIC_URL=http://127.0.0.1:29081 \ RELATA_HTTP_BIND=127.0.0.1:29081 RELATA_PORT=29081 \ RELATA_GRPC_BIND=127.0.0.1 RELATA_GRPC_PORT=29181 \ RELATA_ADMIN_BIND=127.0.0.1:29281 RELATA_PG_PORT=29381 \ RELATA_DATA_DIR=/tmp/relata-cluster/node1 \ "$BIN" serve & # node2 — writer env "${COMMON[@]}" \ NODE_ID=node2 CLUSTER_ROLE=writer \ CLUSTER_PEERS=http://127.0.0.1:29080,http://127.0.0.1:29081 \ RELATA_PUBLIC_URL=http://127.0.0.1:29082 \ RELATA_HTTP_BIND=127.0.0.1:29082 RELATA_PORT=29082 \ RELATA_GRPC_BIND=127.0.0.1 RELATA_GRPC_PORT=29182 \ RELATA_ADMIN_BIND=127.0.0.1:29282 RELATA_PG_PORT=29382 \ RELATA_DATA_DIR=/tmp/relata-cluster/node2 \ "$BIN" serve & wait ``` Verify boot and write path: ```bash curl http://127.0.0.1:29080/health # -> 200, all three ports curl http://127.0.0.1:29081/health curl http://127.0.0.1:29082/health # Writes go through /ingest, NOT /query — /query is read-only (a plain SQL # INSERT against /query returns 400: "/query is read-only and does not # accept INSERT; use POST /ingest?object_type=&purpose=

"). curl -X POST "http://127.0.0.1:29082/ingest?object_type=Person" \ -H "Authorization: Bearer my-client-token" -H "Content-Type: application/json" \ -d '{"id": "p-1", "name": "Example Person"}' # -> 200 {"rows_ingested":1,...} with no peer-replication warnings in any log ``` This is enough to prove the topology boots cleanly and that HTTP-level peer replication (`governed_upsert`) works with no `CLUSTER_AUTH_TOKEN`/connectivity errors. **It is not, by itself, enough for cross-node query fan-out (`SELECT`, including a plain primary-key lookup) to work** — see the gRPC gotcha immediately below, which is a separate, deeper issue than peer replication. ## Gotchas ### `RELATA_GRPC_PORT` must match across nodes, not be unique This is the one that costs the most time, and it's the opposite of the instinct that "every door needs a unique port to avoid a bind collision." Cross-node query fan-out (`relata-cluster/src/scatter_gather.rs`) does **not** read each peer's real gRPC address. It rewrites the peer's **HTTP** URL from `CLUSTER_PEERS` by keeping the peer's host and swapping in **this node's own `RELATA_GRPC_PORT`** (`rewrite_to_grpc_port()`). This is correct and required in the normal production topology — Kubernetes/the Helm chart's `StatefulSet` gives every replica the **same** container port for gRPC (`.Values.service.ports.grpc`, `RELATA_GRPC_PORT` unset → default `50051` everywhere) and differentiates nodes purely by pod DNS name/IP. In that world, "peer host + my own gRPC port" always resolves correctly. On a single machine, giving every node a *unique* `RELATA_GRPC_PORT` (to dodge the `127.0.0.1:50051` bind collision — this is exactly what the required-vars table above tells you to do to get the process to boot) breaks that assumption: node0's rewrite of a request meant for node1 targets `127.0.0.1:` (its own gRPC port, not node1's real one). The symptom is **not** a boot failure — every node reports healthy — it's every cross-node query returning: ```json {"status":503,"title":"Service Unavailable","detail":"cluster fan-out failed: scatter-gather: all 2 peer dispatches failed"} ``` This was reproduced directly against the recipe above: `/health` is `200` on all three nodes and `/ingest` replicates cleanly, but a `SELECT ... WHERE id = '...'` against any node returns the `503` above — because data is hash-partitioned across the 3 shards/nodes, so even a single-row primary-key lookup usually isn't fully answerable from local state alone and needs a working peer gRPC connection. The logs show `QueryShard peer connect failed` targeting the wrong (self) gRPC port. **Fix**: give every node the **same** `RELATA_GRPC_PORT`, and instead vary the *host* each node binds/is reached on: - **Real hosts / separate containers / Kubernetes** (recommended, matches the Helm chart) — each node is a distinct IP or DNS name; leave `RELATA_GRPC_PORT` at the default (or any value) as long as it's identical everywhere. - **Single-machine simulation** — give each node its own loopback alias IP with a shared gRPC port: `sudo ifconfig lo0 alias 127.0.0.2 up` (macOS repeated per extra address needed; harmless and local-only) or use distinct `127.x.x.x` addresses directly on Linux (the whole `127.0.0.0/8` block routes to loopback there without extra config). Bind each node's `RELATA_GRPC_BIND` to its own alias and reference that same alias (not `127.0.0.1`) in `CLUSTER_PEERS` and `RELATA_PUBLIC_URL`, keeping `RELATA_GRPC_PORT` identical across all three. This variant was not independently re-verified in a sandbox without root — it follows directly from `rewrite_to_grpc_port()`'s logic and the same host-per-node pattern the Helm chart uses, but confirm it in your own environment. - **Docker Compose** — each service gets its own container IP on the compose network; publish distinct host ports for `HTTP`/`admin`/`pgwire` as needed, but leave the container-internal gRPC port identical across services (mirrors Kubernetes exactly). If you only need to prove nodes boot and replicate writes over HTTP (as in the tested recipe above), unique gRPC ports per node are fine — just know that `/query` fan-out won't work until the gRPC ports line up. ### Three different bind-var conventions | Var | Shape | Example | |---|---|---| | `RELATA_HTTP_BIND` | **combined** `host:port` | `127.0.0.1:29080` | | `RELATA_ADMIN_BIND` | **combined** `host:port`, must be loopback | `127.0.0.1:29280` | | `RELATA_GRPC_BIND` | **host only** — port is the separate `RELATA_GRPC_PORT` | `RELATA_GRPC_BIND=127.0.0.1` + `RELATA_GRPC_PORT=29180` | Passing `RELATA_GRPC_BIND=127.0.0.1:29180` (treating it like the other two) does not FATAL cleanly — it gets concatenated with `RELATA_GRPC_PORT` into something like `127.0.0.1:29180:50051`, and the process dies with a raw `Error: failed to bind : invalid socket address` rather than a helpful message pointing at the actual mistake. ### `CLUSTER_AUTH_TOKEN` fails silently, not loudly Unlike `RELATA_CLUSTER_SEED`, `RELATA_PUBLIC_URL`, and `NODE_ID` — all of which FATAL the process at startup if wrong — an unset `CLUSTER_AUTH_TOKEN` on a `cluster`-profile node does **not** stop that node from starting. It boots, reports `200` on `/health`, and looks completely healthy. Every peer that tries to replicate a write to it or run a `QueryShard` RPC against it gets rejected with a `503`: ``` "cluster internal endpoints require RELATA_CLUSTER_AUTH_TOKEN to be configured" ``` Note the error text itself says `RELATA_CLUSTER_AUTH_TOKEN` — that env var doesn't exist; the real var the code reads is `CLUSTER_AUTH_TOKEN` (no `RELATA_` prefix). Don't go looking for a var named `RELATA_CLUSTER_AUTH_TOKEN` — it's a naming slip in the error string, not a second real var. Because this fails per-request rather than at boot, it's easy to bring up a cluster, watch every node's `/health` return `200`, declare victory, and only discover the misconfiguration later when writes stop propagating or fan-out queries start 503ing. Double-check `CLUSTER_AUTH_TOKEN` is identical (not just "set") on every node before trusting a green `/health`. If the tokens are set on both sides but simply **mismatched** (not empty), the failure mode is different again: a `401 Unauthorized`, not a `503` — the request falls through to a bearer-token check instead. ### Every other protocol door defaults to the same port on every node `RELATA_PG_PORT` (pgwire), and the S3/ClickHouse/Neo4j/Redis/MongoDB/Bolt compat doors, all default to a fixed port regardless of `NODE_ID`. Running 3 nodes on one machine means 2 of them lose those doors to a non-fatal `bind failed — disabled` WARN unless you give each node distinct values (`RELATA_PG_PORT` above) or explicitly disable the doors you don't need per-node. This doesn't break the core cluster (HTTP + gRPC fan-out), but it's worth knowing the WARNs in the log are expected and not something to chase. ### At-rest encryption's FATAL trigger is broader than "using a remote store" `RELATA_KMS_LOCAL_DEV=true` (or a real `RELATA_KMS_KEY_ARN`) is required the moment cluster profile has *any* on-disk persistence — a plain local `RELATA_DATA_DIR` is enough to trigger the KMS FATAL at startup, not just an S3/remote object store. If you see `RELATA_KMS_KEY_ARN is required in the 'cluster' profile; refusing to fall back to the committed dev secret`, this is why. ## Helm / Kubernetes vs. this guide The [Helm chart](/docs/deployment/kubernetes) (`infra/helm/relata` in the main repo) already encodes a correct version of most of this for you: - The `StatefulSet` gives every replica the same container ports (so the gRPC-port gotcha above never comes up — pods are differentiated by DNS name, not port). - `CLUSTER_PEERS` is generated automatically from the headless-service DNS names for all replicas. - `CLUSTER_AUTH_TOKEN` defaults to the same secret as `RELATA_BEARER_TOKEN` when not set explicitly, so you can't accidentally leave it unset on `cluster.enabled: true`. - `RELATA_GRPC_PLAINTEXT_OK` / `RELATA_GRPC_TLS_*` are wired for you depending on `cluster.grpcPlaintext` / `tls.enabled`. Use this page to understand the mechanics, debug a Helm-based cluster that isn't fan-out-ing correctly, or run cluster profile somewhere that isn't Kubernetes at all (bare metal, systemd units on separate hosts, Docker Compose). For a real deployment, prefer the Helm chart. ## See also - [Deployment](/docs/deployment) — the three profiles and when to use each - [Kubernetes Deployment](/docs/deployment/kubernetes) — the Helm-chart production path - [Environment Variables](/docs/reference/env-vars) — full reference - [Limits & Caveats](/docs/reference/limits) — capacity & scaling caveats for cluster profile ============================================================================== # Kubernetes Deployment URL: https://relatadb.dev/docs/deployment/kubernetes ============================================================================== # Kubernetes Deployment RelataDB ships as a single binary, making it straightforward to deploy on Kubernetes. ## Helm chart ```bash helm repo add relata https://charts.relatadb.dev helm install relata relata/relata-db \ --set profile=server \ --set tenancyMode=multi \ --set persistence.enabled=true \ --set persistence.size=100Gi ``` ## Basic deployment ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: relata-db spec: replicas: 1 selector: matchLabels: app: relata-db template: metadata: labels: app: relata-db spec: containers: - name: relata-db image: ghcr.io/relatadb/relata:latest ports: - containerPort: 9090 name: http - containerPort: 5433 name: pgwire - containerPort: 50051 name: grpc env: - name: RELATA_PROFILE value: server - name: RELATA_BEARER_TOKEN valueFrom: secretKeyRef: name: relata-secrets key: admin-token - name: RELATA_TENANCY_MODE value: multi - name: AWS_ENDPOINT_URL value: https://s3.amazonaws.com - name: AWS_S3_BUCKET value: my-bucket volumeMounts: - name: data mountPath: /data readinessProbe: httpGet: path: /health/ready port: 9090 livenessProbe: httpGet: path: /health/live port: 9090 volumes: - name: data persistentVolumeClaim: claimName: relata-data ``` ## Persistent storage For production, use a `PersistentVolumeClaim` or configure object storage (S3/GCS/Azure Blob): ```bash AWS_ENDPOINT_URL=https://s3.amazonaws.com AWS_S3_BUCKET=my-bucket ``` Credentials go via `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` (see [Environment Variables](/docs/reference/env-vars#storage)). For a native GCS/Azure backend instead of the S3-compatible path, use `RELATA_OBJECT_STORE` (`gcs://bucket/prefix` or `azure://container/prefix`). When object storage is configured, the PVC is used only for WAL segments and local cache — the source of truth is the object store. This enables stateless compute nodes. ## Cluster mode For horizontal scaling, deploy multiple replicas with `RELATA_PROFILE=cluster`: ```yaml spec: replicas: 3 template: spec: containers: - name: relata-db env: - name: RELATA_PROFILE value: cluster - name: NODE_ID valueFrom: fieldRef: fieldPath: metadata.name ``` All nodes share the same object store. Consistent hashing distributes segment ownership. Query fan-out merges results across nodes. A single `Deployment`/`StatefulSet` template gives every replica the same `CLUSTER_ROLE` (defaults to `coordinator` if unset). To run dedicated `reader`/`writer`/`indexer` nodes instead, split them into separate specs — see [Cluster Setup](/docs/deployment/cluster) for the role breakdown and a worked multi-role example. ## Health checks | Probe | Endpoint | Purpose | |---|---|---| | Liveness | `GET /health/live` | Process is alive | | Readiness | `GET /health/ready` | Ready to serve (indexes loaded) | | Startup | `GET /health/ready` | Wait for boot before traffic | ## Protocol ports | Port | Protocol | Use | |---|---|---| | 9090 | HTTP | REST API, health, metrics | | 5433 | PostgreSQL wire | psql, psycopg2, pgvector | | 50051 | gRPC | gRPC / Arrow Flight door | | 7687 | Bolt | Neo4j drivers, Cypher | | 27017 | Mongo wire | MongoDB drivers | ## Monitoring ```yaml env: - name: RELATA_METRICS_PUBLIC value: "true" ``` Scrape `/metrics` for Prometheus metrics. See [Observability](/docs/guides/observability) for the full metrics reference. ## See also - [Deployment](/docs/deployment) — profiles and configuration - [Cluster Setup](/docs/deployment/cluster) — the non-Kubernetes mechanics: every env var cluster profile needs and the gotchas behind a working (or silently broken) fan-out - [Scaling](/docs/guides/scaling) — cluster sizing and capacity planning - [Configuration](/docs/guides/configuration) — environment variables ============================================================================== # Deploying Protocol Doors URL: https://relatadb.dev/docs/deployment/protocol-doors ============================================================================== # Deploying Protocol Doors The compatibility doors (see [Compatibility & Doors](/docs/compatibility)) default to `127.0.0.1` so an unauthenticated port is never accidentally exposed. To let another container, host, or pod reach a door, you do three things: **(1)** enable the door, **(2)** override its bind address, **(3)** publish the port. This page is the whole story for Docker, Kubernetes, and bare-metal clusters. > Every door honors `RELATA__BIND` as a **plain override on every profile** — no license needed. The HTTP and gRPC listeners bind `0.0.0.0` automatically on the `server` and `cluster` profiles; the 8 compat doors default to loopback on every profile and opt up only when you say so. ## The 60-second recipe Pick the doors you actually use and copy the pattern. The bearer token is the password for every protocol. ```bash # Enable Mongo + Postgres + Redis, bind all to 0.0.0.0 so other hosts/containers can reach them RELATA_BEARER_TOKEN=change-me \ RELATA_MONGO_ENABLE=true RELATA_MONGO_BIND=0.0.0.0 \ RELATA_PG_ENABLE=true RELATA_PG_BIND=0.0.0.0 \ RELATA_REDIS_ENABLE=true RELATA_REDIS_BIND=0.0.0.0 \ relata serve ``` If `RELATA_BEARER_TOKEN` is set you can drop the explicit `_ENABLE=true` lines — every door auto-enables on a token. Keep the `_BIND=0.0.0.0` overrides; they're what makes the door reachable off-loopback. (pgwire is the one exception — it auto-starts the moment a token is present, no enable flag needed, and it's fail-closed without a token.) ## Docker — publish every door you enable `RELATA__BIND=0.0.0.0` makes the door listen on all interfaces *inside the container*; you still need `-p` to publish the port to the host. A door enabled but not `-p`-published is unreachable from outside the container. ```bash docker run -d \ -p 9090:9090 `# HTTP REST (always on)` \ -p 5433:5433 `# Postgres / pgvector` \ -p 27017:27017 `# MongoDB wire` \ -p 6379:6379 `# Redis RESP` \ -p 7474:7474 `# Neo4j HTTP Cypher` \ -p 7687:7687 `# Neo4j Bolt` \ -p 8123:8123 `# ClickHouse HTTP` \ -p 9000:9000 `# ClickHouse native TCP` \ -p 9191:9191 `# S3-compatible` \ -p 50051:50051 `# gRPC` \ -p 8815:8815 `# Arrow Flight` \ -e RELATA_PROFILE=server \ -e RELATA_BEARER_TOKEN=change-me \ -e RELATA_MONGO_BIND=0.0.0.0 \ -e RELATA_PG_BIND=0.0.0.0 \ -e RELATA_REDIS_BIND=0.0.0.0 \ -e RELATA_S3_BIND=0.0.0.0 \ -e RELATA_FLIGHT_ENABLE=true \ -e RELATA_FLIGHT_BIND=0.0.0.0 \ -v "$PWD/relata-data:/data/relata" \ --name relata ghcr.io/relatadb/relata:2.0.0 ``` Only publish the doors you actually use — every published port is attack surface. To bind to a specific interface instead of all interfaces, use the host IP (e.g. `RELATA_MONGO_BIND=10.0.0.5` or `-p 10.0.0.5:27017:27017`). ## Kubernetes — declare `containerPort` + the env pair A door enabled without a matching `containerPort` is silent: it binds inside the pod but no `Service` routes to it. Declare every door you use. ```yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: relata-db spec: serviceName: relata-db replicas: 1 template: spec: containers: - name: relata-db image: ghcr.io/relatadb/relata:2.0.0 ports: - { containerPort: 9090, name: http } - { containerPort: 5433, name: pgwire } - { containerPort: 27017, name: mongo } - { containerPort: 6379, name: redis } - { containerPort: 7474, name: neo4j-http } - { containerPort: 7687, name: bolt } - { containerPort: 8123, name: clickhouse-http } - { containerPort: 9000, name: clickhouse-native } - { containerPort: 9191, name: s3 } - { containerPort: 50051, name: grpc } - { containerPort: 8815, name: flight } env: - { name: RELATA_PROFILE, value: server } - { name: RELATA_BEARER_TOKEN, valueFrom: { secretKeyRef: { name: relata-secrets, key: admin-token } } } # Bind the doors you expose to 0.0.0.0 (default is loopback). - { name: RELATA_MONGO_BIND, value: "0.0.0.0" } - { name: RELATA_PG_BIND, value: "0.0.0.0" } - { name: RELATA_REDIS_BIND, value: "0.0.0.0" } - { name: RELATA_S3_BIND, value: "0.0.0.0" } - { name: RELATA_FLIGHT_ENABLE, value: "true" } - { name: RELATA_FLIGHT_BIND, value: "0.0.0.0" } # ...add one BIND per door you publish... ``` Then expose each door through a `Service`. A single `ClusterIP` for internal callers, or a `LoadBalancer`/`Ingress` per protocol for external clients (most compat protocols aren't HTTP, so `Ingress` usually isn't the right fit — prefer `LoadBalancer` or `NodePort` for Mongo/Postgres/Redis/Bolt/ClickHouse/S3). ```yaml apiVersion: v1 kind: Service metadata: name: relata-mongo spec: selector: { app: relata-db } type: LoadBalancer ports: - { port: 27017, targetPort: 27017, name: mongo } ``` Repeat per protocol. For a full worked example (PVC, readiness/liveness, multi-tenant mode), see [Kubernetes Deployment](/docs/deployment/kubernetes). ## Full port reference (all 11 networked surfaces) | Door | Enable flag | Port | Bind var | Default bind | |---|---|---|---|---| | HTTP REST | always on | `9090` | `RELATA_HTTP_BIND` | `0.0.0.0` on server/cluster, `127.0.0.1` on free | | gRPC | always on | `50051` | `RELATA_GRPC_BIND` | `0.0.0.0` on server/cluster, `127.0.0.1` on free | | Postgres + pgvector | token required | `5433` | `RELATA_PG_BIND` | `127.0.0.1` | | MongoDB | `RELATA_MONGO_ENABLE` | `27017` | `RELATA_MONGO_BIND` | `127.0.0.1` | | Redis | `RELATA_REDIS_ENABLE` | `6379` | `RELATA_REDIS_BIND` | `127.0.0.1` | | Neo4j HTTP | `RELATA_NEO4J_ENABLE` | `7474` | `RELATA_NEO4J_BIND` | `127.0.0.1` | | Bolt | `RELATA_BOLT_ENABLE` | `7687` | `RELATA_BOLT_BIND` | `127.0.0.1` | | ClickHouse HTTP | `RELATA_CLICKHOUSE_ENABLE` | `8123` | `RELATA_CLICKHOUSE_BIND` | `127.0.0.1` | | ClickHouse native | `RELATA_CLICKHOUSE_NATIVE_ENABLE` | `9000` | `RELATA_CH_NATIVE_BIND` | `127.0.0.1` | | S3-compatible | `RELATA_S3_ENABLE` | `9191` | `RELATA_S3_BIND` | `127.0.0.1` | | Arrow Flight | `RELATA_FLIGHT_ENABLE` | `8815` | `RELATA_FLIGHT_BIND` | `127.0.0.1` | MCP and SPARQL ride on the HTTP listener (`/mcp`, `/sparql`) — no separate port. ## Cluster mode — one door port per node In a multi-node cluster (see [Cluster Setup](/docs/deployment/cluster)), every node defaults to the same door ports. If you run three nodes on one host for testing, two of them will silently WARN-fail the door binds and lose ⅔ of your door capacity. Either: - Run one node per host (production), or - Give each node a distinct `RELATA__PORT` in test, or - Disable the doors on `reader`/`indexer` nodes (`RELATA_MONGO_ENABLE=false`, etc.) and route door traffic only to `coordinator`/`writer` nodes. Doors are stateless front-ends over the same governed store — any node can serve any door; writes funnel through the planner regardless of which node received them. ## Security checklist before production - [ ] `RELATA_BEARER_TOKEN` is a strong random value (e.g. `openssl rand -hex 32`), not `change-me`. - [ ] Only the doors you use have `RELATA__BIND=0.0.0.0`; the rest stay on loopback. - [ ] Only the doors you use are `-p` published / have a `Service`. - [ ] TLS is terminated either by Relata (`RELATA_TLS_CERT`/`RELATA_TLS_KEY`, `RELATA_PG_TLS_CERT`/`RELATA_PG_TLS_KEY`, `RELATA_GRPC_TLS_CERT`/`_KEY`) or by a reverse proxy / sidecar in front (`RELATA_PLAINTEXT_OK=true` only if you terminate TLS upstream). - [ ] For S3 in production, set `RELATA_S3_SECRET_KEY` to a dedicated SigV4 secret (defaults to the bearer token otherwise) and leave `RELATA_S3_ALLOW_PLAINTEXT` unset. - [ ] `RELATA_TENANCY_MODE=multi` is set only if you actually want per-tenant isolation (it's the one real cluster-only gate). ## See also - [Compatibility & Doors](/docs/compatibility) — connection strings and 3-step quickstarts per protocol - [Protocol Compatibility (reference)](/docs/reference/protocols) — per-protocol semantics and limits - [Kubernetes Deployment](/docs/deployment/kubernetes) — full Helm/StatefulSet walkthrough - [Cluster Setup](/docs/deployment/cluster) — multi-node mechanics and gotchas - [Environment Variables](/docs/reference/env-vars#wire-protocol-ports) — canonical env-var reference ============================================================================== # Deployment URL: https://relatadb.dev/docs/deployment ============================================================================== # Deployment Relata ships three deployment profiles from one binary. The SDK works identically across all three; only the runtime characteristics differ. | Profile | Use case | Default caps | Start command | |---|---|---|---| | `free` | Embedded / single-process / dev / CI | Unbounded (small datasets stay in RAM) | `RELATA_PROFILE=free relata serve` | | `server` | Single-node production | RAM walls default-on at 1024 MB | `RELATA_PROFILE=server relata serve` | | `cluster` | Multi-node distributed (**alpha**) | Same as server, plus cluster coordination | `RELATA_PROFILE=cluster relata serve` | `free` is the default (`lite` was a legacy alias and is now rejected outright — use `free`). ## Protocol doors — bring your existing client RelataDB speaks **13 wire protocols** from one binary: 8 compatibility doors (MongoDB, Postgres + pgvector, Redis, Neo4j HTTP/Bolt, ClickHouse HTTP/TCP, S3) plus 5 native (HTTP, gRPC, Arrow Flight, MCP, SPARQL). **Your existing clients connect to Relata** — you don't rewrite your app, you repoint host/port and use `RELATA_BEARER_TOKEN` as the password. See [Compatibility & Doors](/docs/compatibility) for the full story. Doors default to `127.0.0.1` so an unauthenticated port is never auto-exposed, and they auto-enable when `RELATA_BEARER_TOKEN` is set. To let another container/host/pod reach a door, override its bind and publish the port: ```bash # Expose Mongo + Postgres + Redis off-loopback, on any profile (no license needed) RELATA_BEARER_TOKEN= \ RELATA_MONGO_BIND=0.0.0.0 \ RELATA_PG_BIND=0.0.0.0 \ RELATA_REDIS_BIND=0.0.0.0 \ RELATA_PROFILE=server relata serve ``` For the full Docker `-p` / Kubernetes `containerPort` wiring, TLS, and the security checklist, see **[Deploying Protocol Doors](/docs/deployment/protocol-doors)**. ## Profile defaults that bite The `server` and `cluster` profiles enable the **disk-first walls** by default so you don't OOM in production: - `RELATA_STORE_MAX_RAM_MB` → **1024 MB** on server/cluster (unbounded on free) - Graph `RELATA_GRAPH_RAM_BUDGET_MB` → 1024 MB - Identity `RELATA_IDENTITY_RAM_BUDGET_MB` → 1024 MB - DiskANN `RELATA_DISKANN_MAX_RESIDENT` → 1,000,000 resident/bucket An explicit env var always overrides the profile default. The budgets are generous — small/medium deployments stay fully resident (no behaviour change). ## Free — local dev ```bash relata serve ``` Storage defaults to local disk at `./data/relata/objects` (persistent, no config needed). Override with `RELATA_LOCAL_DATA_DIR` for a custom path, or `AWS_ENDPOINT_URL` for S3/MinIO. Without `RELATA_BEARER_TOKEN` the server runs in dev mode (no auth, pgwire disabled). The warning is expected: ``` WARNING: RELATA_BEARER_TOKEN not set — running in unauthenticated mode (dev only) ``` LLM calls and telemetry are opt-in — leave `RELATA_LLM_URL`/`RELATA_LLM_API_KEY` and `RELATA_OTLP_ENDPOINT` unset for an air-gapped node. For demo / load-testing, disable rate limits directly: ```bash RELATA_RATE_LIMIT_RPS=99999 RELATA_RATE_LIMIT_AUTH_FAIL_RPS=99999 relata serve ``` ## Server — single-node production ```bash RELATA_PROFILE=server \ RELATA_BEARER_TOKEN= \ RELATA_PORT=9090 \ relata serve ``` Set `RELATA_PLAINTEXT_OK=true` only if you terminate TLS at a reverse proxy / sidecar. ### Object-store persistence Object-store persistence is always on — Relata defaults to local disk at `RELATA_DATA_DIR/objects`. `RELATA_LOCAL_DATA_DIR` overrides the path; `AWS_ENDPOINT_URL` selects S3 / MinIO / R2 / GCS / Azure Blob. WAL + Parquet snapshots persist across restarts. ### Cold-restart RTO (measured at 10 M rows) | Phase | Time | |---|---| | WAL + Parquet flush (shutdown) | ~15 s | | Cold-load from Parquet (restart, no warm cache) | ~55 s | Single-node RTO ≈ 1 minute at 10M-row scale. Paged backends + WAL replay are the production path for faster restarts on larger datasets. For faster cold-load: ```bash RELATA_LAZY_RESTART=true \ RELATA_HYDRATE_RECENT_SEGMENTS=5 \ relata serve ``` Lazy restart loads the manifest **catalog only** (O(manifest), not O(rows)); the newest N segments hydrate into RAM at startup and the rest hydrate on demand. ## Cluster — multi-node (alpha) ```bash RELATA_PROFILE=cluster \ NODE_ID=node-1 \ CLUSTER_ROLE=coordinator \ CLUSTER_PEERS=http://node-2:9090,http://node-3:9090 \ relata serve ``` (Plain `NODE_ID` / `CLUSTER_ROLE` / `CLUSTER_PEERS` — not `RELATA_NODE_ID`/`RELATA_ROLE`/`RELATA_PEERS`, which are different, unrelated vars. A real cluster node also needs `RELATA_CLUSTER_SEED`, `RELATA_PUBLIC_URL`, `CLUSTER_AUTH_TOKEN`, and — for local/dev without a real KMS key — `RELATA_KMS_LOCAL_DEV=true`; see [Cluster Setup](/docs/deployment/cluster) for the complete, tested list and a copy-pasteable 3-node example.) | Role | Responsibility | |---|---| | `coordinator` | Query planning, request routing | | `reader` | Read-only query execution | | `writer` | Write ingest + WAL | | `indexer` | Background indexing (FTS, vectors, identity) | Cluster is **alpha** — petabyte / 200B-subscriber cardinality still needs sharding / cluster fan-out (epic #797). ## Observability | Variable | Default | Effect | |---|---|---| | `RELATA_LOG_FORMAT` | `pretty` | `json` for production log shippers. | | `RELATA_LOG_LEVEL` | `info` | Log level. | | `RELATA_OTLP_ENDPOINT` | — | OTLP/HTTP traces endpoint. Unset = OpenTelemetry fully disabled. | | `RELATA_OTLP_SAMPLE_RATIO` | `0.01` | Parent-based TraceID-ratio sampler. | | `RELATA_METRICS_PUBLIC` | — | Serve `/metrics` without a bearer token (Prometheus behind network-layer auth). | ## Graceful shutdown The server handles SIGTERM gracefully: 1. Stops accepting new requests. 2. Drains in-flight queries (configurable drain timeout). 3. Flushes the WAL + Parquet snapshot. 4. Closes the HTTP/gRPC listeners. Cold-restart RTO at 10M rows is ~1 minute (see above). ## See also - [Deploying Protocol Doors](/docs/deployment/protocol-doors) — bind vars, Docker `-p` publishing, Kubernetes `containerPort`, cross-host reachability - [Compatibility & Doors](/docs/compatibility) — connection strings and 3-step quickstarts per protocol - [Cluster Setup](/docs/deployment/cluster) — full multi-node walkthrough, a tested local 3-node recipe, and the non-obvious gotchas - [Kubernetes Deployment](/docs/deployment/kubernetes) — the Helm-chart production path - [Environment Variables](/docs/reference/env-vars) — full reference - [Limits & Caveats](/docs/reference/limits) — capacity & scaling ============================================================================== # Build an anomaly monitoring engine URL: https://relatadb.dev/docs/guides/anomaly-monitoring ============================================================================== # Build an anomaly monitoring engine Most "anomaly detection" projects fail at the plumbing, not the math: joining the stream to the historical baseline, getting the alert to the right investigator with provenance, and reconstructing later *why* the system fired. RelataDB already owns that plumbing — bi-temporal storage, streaming windows, commit-time detection, governed alerts, and a tamper-evident audit chain — so you build the **detection logic**, not the pipeline. This page walks the anomaly primitives Relata ships, then designs a worked **Twitter / social monitoring engine** end-to-end, and closes with a **statistical-bias / distribution-shift detector** (the same machinery, pointed at skew instead of spikes). > Every capability below is verified against source. The one honesty flag: `AnomalyRateJob` is a **library type, not a turnkey scheduled job** — Relata's `JobRegistry` has no scheduler consumer for it yet, so you trigger it from your own cron/scheduler. See the maturity table. ## The anomaly primitives | Primitive | What it does | Where | |---|---|---| | **Streaming windows** — `TUMBLE`, `HOP`, `SESSION` | Time-bucketed aggregation in SQL for volume/rate baselines | `streaming_ops.rs`, SQL planner | | **Detection rules** | A SQL `WHERE` over any type, firing at **commit time** (not on a poll) into governed `Alert` rows | [Detection Rules](/docs/guides/detection-rules) | | **`AnomalyRateJob`** | Compares an incident-count in a rolling window vs an EWMA-smoothed historical baseline; emits `AnomalyAlert` at a configurable σ threshold (default 2σ) | `crates/relata-intelligence/src/anomaly.rs` | | **Pattern library** | `PatternTemplate` matching with `Contradiction` detection (two facts that can't both be true) | `crates/relata-intelligence/src/pattern_library.rs` | | **Threat-proximity scoring** | `SanctionsProximityJob` — score entities by graph distance to known-bad | `sanctions_proximity.rs` | | **Incident clustering** | Group related signals into one incident | `crates/relata-intelligence/src/cluster.rs` | | **LLM interpretation** | Natural-language summary + `nl_query` for "what just happened?" | `crates/relata-intelligence/src/llm.rs` | ## Worked design — a Twitter / social monitoring engine Goal: ingest social posts (and engagement signals), surface spikes (volume, sentiment, coordinated behavior), fuse identities across handles, detect threat patterns, and present an investigator with a defensible, time-travelable case. Relata does the storage / identity / detection / audit; you write the per-feed fetcher. ### Architecture ``` fetcher(s) ──► /ingest (HTTP/Kafka/OTLP) ──► governed store (your code) │ │ SmartIngest canonicalizes │ │ handles → identity graph ▼ ▼ ┌────────────── streaming SQL windows ──────────────┐ │ TUMBLE / HOP / SESSION over Post/event rows │ └──────────────────────┬───────────────────────────┘ │ ┌──────────────────────▼───────────────────────────┐ │ detection rules (commit-driven) + AnomalyRateJob │ │ + pattern library (contradictions) │ └──────────────────────┬───────────────────────────┘ ▼ governed Alert rows ──► webhook (SOAR) + /alerts/stream (SSE) + audit chain │ ▼ investigator: PATHS_BETWEEN, MCP investigate_entity, EXPLAIN_REPLAY ``` ### Step 1 — ingest + canonicalize Your fetcher pulls from the social API and pushes governed rows through any door (HTTP `/ingest/bulk`, Kafka, or the Mongo door if your existing pipeline already writes Mongo docs): ```bash curl -X POST 'http://127.0.0.1:9090/ingest/bulk?object_type=SocialPost&purpose=osint' \ -H 'Authorization: Bearer ' -H 'Content-Type: application/json' \ -d '{"rows":[ {"_pk":"tw-1","author_handle":"@suspect_42","body":"...","posted_at_ns":1735490000000000000, "like_count":3,"retweet_count":1,"lang":"en"}, {"_pk":"tw-2","author_handle":"@suspect_42","body":"...","posted_at_ns":1735490060000000000, "like_count":4120,"retweet_count":980,"lang":"en"} ]}' ``` Enable the **social detector pack** so SmartIngest canonicalizes handles into the `IdentityIndex`: ```bash RELATA_DETECT_PACKS=network,contact,social relata serve ``` > **Honest scope on Twitter specifically:** Relata ships canonical types for **Facebook, Instagram, LinkedIn, Snapchat, Telegram, TikTok, and UPI** identifiers, plus a generic `social` pack. There is **no dedicated `TwitterHandle` canonical type** today — for an X/Twitter pipeline, either (a) treat `@handle` as a generic social identifier resolved by the `social` pack, or (b) add a one-file canonical type in `relata-canonical` (it's a typed validator pattern — see `tiktok_handle.rs` as the template). The rest of the engine is identical. ### Step 2 — fuse identities across handles and sources The same operator running a Telegram channel, a TikTok account, and a Bitcoin address is auto-linked: ```sql PURPOSE 'osint' -- Resolve everything known about this identity across all sources SELECT * FROM RESOLVE_IDENTITY('@suspect_42', MODE => 'cluster'); -- Is the @handle the same entity as a wallet we already track? SELECT SAME_IDENTITY('SocialHandle:suspect_42', 'Wallet:0xabc...') AS same; -- Who is this account connected to, within 5 hops? SELECT * FROM PATHS_BETWEEN('SocialHandle:suspect_42', 'SocialHandle:target', 5); ``` Identity fusion is **deterministic** (no LLM guessing) — see [Identity](/docs/concepts/identity). ### Step 3 — detect spikes with streaming windows Use `TUMBLE` / `HOP` / `SESSION` to compute rolling baselines in SQL, then alert on deviation: ```sql PURPOSE 'osint' -- Hourly post volume per author, tumbling window SELECT author_handle, tumble_start(posted_at_ns, 3600*1000*000000) AS hour_start, COUNT(*) AS posts, SUM(like_count) AS total_likes FROM SocialPost WHERE posted_at_ns > now() - INTERVAL '7' days GROUP BY author_handle, hour_start ORDER BY total_likes DESC; ``` ```sql -- Compare this hour to the trailing 30-day baseline for the same author WITH baseline AS ( SELECT author_handle, AVG(post_count) AS mean_posts, STDDEV(post_count) AS sigma_posts FROM ( SELECT author_handle, tumble_start(posted_at_ns, 3600*1000*000000) AS h, COUNT(*) AS post_count FROM SocialPost WHERE posted_at_ns > now() - INTERVAL '30' days GROUP BY author_handle, h ) GROUP BY author_handle ) SELECT b.author_handle, mean_posts, sigma_posts, (mean_posts + 3 * sigma_posts) AS spike_threshold FROM baseline b WHERE sigma_posts > 0; ``` ### Step 4 — wire a commit-time detection rule A rule fires the moment a matching row lands — no poller lag: ```bash curl -X POST http://127.0.0.1:9090/rules \ -H 'Authorization: Bearer ' -H 'Content-Type: application/json' \ -d '{ "name": "viral-spike-coordinated-amplification", "target_type": "SocialPost", "condition_sql": "retweet_count > 500 AND like_count > 2000 AND lang = '\''en'\''", "severity": "medium", "mitre_technique": "T1585", "purpose": "osint" }' ``` ### Step 5 — the `AnomalyRateJob` (2σ incident rate) For "did the overall incident rate jump?" — the library job compares the current rolling window's incident count against an EWMA-smoothed historical baseline and emits an `AnomalyAlert` past a configurable σ threshold. ```rust // crates/relata-intelligence/src/anomaly.rs (paraphrased) pub struct AnomalyRateJobConfig { pub baseline_windows: usize, // e.g. 24 = last 24h pub sigma_threshold: f64, // default 2.0 pub ewma_decay: f64, // 0.0–1.0 — how fast baseline forgets } ``` > **Honest wiring note:** `JobKind::AnomalyRateDetect` is **library-only today** — Relata's `JobRegistry` has no scheduler consumer for it, so the binary will not auto-run it on a cron. Wire it from your own scheduler (systemd timer, Kubernetes `CronJob`, `tokio::spawn` loop) that calls the job against the running store. This is the one piece of the anomaly story that isn't turnkey out of the box. ### Step 6 — alert routing + investigation ```python # Register your SOAR / Slack / PagerDuty webhook client.register_webhook("https://soar.example.com/relata", event_types=["alert.high", "alert.medium"]) # Stream alerts to a live dashboard for ev in client.streaming_client.alerts(): print(ev["severity"], ev["rule_name"], ev["target_id"]) ``` ```python # Investigate via MCP — natural-language + graph mcp.call_tool("investigate_entity", {"entity_type": "SocialHandle", "entity_id": "suspect_42", "purpose": "osint"}) mcp.call_tool("find_threats", {"entity_type": "SocialPost", "purpose": "osint"}) ``` Every alert is a bi-temporal `Alert` row — backtest detection quality with `SELECT ... FROM Alert AS OF ''` before going live. See [Detection Rules](/docs/guides/detection-rules). ## Building a "bias" / distribution-shift detector The same machinery, pointed at **skew** instead of spikes. A bias detector asks: "has the *distribution* of a signal drifted from its baseline?" — e.g., a sentiment classifier whose positive/negative ratio suddenly shifts, a recommendation pipeline whose demographic skew crosses a threshold, or a feed whose language mix diverges. ```sql PURPOSE 'ml-observability' -- Track the sentiment-class distribution per day; flag days where the -- positive share deviates more than 3σ from the trailing 30-day mean. WITH daily_dist AS ( SELECT tumble_start(classified_at_ns, 86400*1000*000000) AS day, sentiment, COUNT(*) AS c FROM ClassifiedPost WHERE classified_at_ns > now() - INTERVAL '60' days GROUP BY day, sentiment ), positive_share AS ( SELECT day, 1.0 * SUM(CASE WHEN sentiment='positive' THEN c ELSE 0 END) / NULLIF(SUM(c), 0) AS pos_share FROM daily_dist GROUP BY day ), baseline AS ( SELECT AVG(pos_share) AS mu, STDDEV(pos_share) AS sigma FROM positive_share WHERE day < tumble_start(now(), 86400*1000*000000) - INTERVAL '1' days ) SELECT p.day, p.pos_share, b.mu, b.sigma, ABS(p.pos_share - b.mu) / NULLIF(b.sigma, 0) AS z_score FROM positive_share p CROSS JOIN baseline b WHERE ABS(p.pos_share - b.mu) > 3 * b.sigma; ``` Turn that into a governed `Alert` via a detection rule or a scheduled job, and you have a **bias monitor** with the same audit chain, provenance, and time-travel as every other Relata signal — defensible to a regulator or an ML-governance review. ## Maturity table (honest) | Capability | Status | |---|---| | Streaming windows (TUMBLE/HOP/SESSION) | ✅ Shipped — SQL + `streaming_ops` | | Commit-driven detection rules + Sigma import | ✅ Shipped — [Detection Rules](/docs/guides/detection-rules) | | Bi-temporal `Alert` rows + webhook + `delivery_status` | ✅ Shipped | | `AnomalyRateJob` (2σ, EWMA) | 🟡 Library type — wire your own scheduler | | Pattern library (`PatternTemplate`, `Contradiction`) | ✅ Shipped in `relata-intelligence` | | Threat-proximity / sanctions-proximity scoring | ✅ Shipped | | Incident clustering | ✅ Shipped | | LLM interpretation + `nl_query` | ✅ Shipped (needs `RELATA_LLM_URL`) | | Canonical social types | ✅ Facebook/Instagram/LinkedIn/Snapchat/Telegram/TikTok/UPI; 🟡 no dedicated Twitter handle (use `social` pack or add one) | | MCP investigation verbs | ✅ `investigate_entity`, `find_threats`, `search_video_frames`, `face_match` | ## Tips & takeaways - **Pick the right primitive per signal.** Single-row pattern → detection rule. Rate/volume spike → streaming-window SQL or `AnomalyRateJob`. Distribution drift → windowed-stats SQL. Multi-signal correlation → pattern library + graph traversal. - **Backtest before going live.** `SELECT FROM Alert AS OF ''` against a new rule tells you what it *would* have fired — tune σ and thresholds on history, not customers. - **Wire `AnomalyRateJob` from Kubernetes `CronJob` or a `tokio` loop** — it won't self-schedule today. - **Add a custom canonical type per social platform** (one file, see `tiktok_handle.rs`) if you need checksum-level validation beyond the generic `social` pack. - **Pair detection with governance.** A bias/anomaly alert that drives an automated action should go through `PURPOSE` + Cedar — so the action is as auditable as the detection. - **Use the LLM interpretation for triage, not decision.** `nl_query` and the LLM summary explain a spike in prose for the on-call analyst; the governed `Alert` row is the authoritative record. ## See also - [Detection Rules](/docs/guides/detection-rules) — the commit-driven rule engine - [Jobs, Workflows & Detection](/docs/guides/jobs-workflows) — the typed-Job engine - [Streaming & SSE](/docs/reference/search) — `WATCH` subscriptions + `/alerts/stream` - [OSINT — Identity Fusion](/docs/use-cases/osint-identity-fusion) — cross-source handle resolution - [Graph Analytics](/docs/reference/graph-analytics) — `PATHS_BETWEEN`, community detection, link prediction - [For Security Teams](/docs/use-cases/for-security-teams) — the SIEM-replacement framing ============================================================================== # Backup & Restore URL: https://relatadb.dev/docs/guides/backup-restore ============================================================================== # Backup & Restore RelataDB's durability model is object-store-native. The bucket is the backup. Snapshots are open-format Parquet and Arrow IPC files readable by DuckDB, Spark, or Athena even without RelataDB running. There is no proprietary backup format. ## How durability works 1. Every write appends to the WAL before the ack is returned. 2. On graceful shutdown, the WAL is flushed and a Parquet snapshot is written to the object store. 3. On restart, the node replays the manifest catalog (lazy restart) or all segments (eager restart). 4. Hash-chained commit manifests make any tampering detectable. A `SIGKILL` skips the Parquet snapshot — the WAL is still intact and replayed on next boot. This takes longer than a clean restart but data is not lost. ## Wire the object store Without an object-store endpoint, RelataDB writes to `RELATA_DATA_DIR/objects` on local disk. That is fine for dev but not for production. Configure S3-compatible object storage: ```bash export AWS_ENDPOINT_URL=http://minio:9000 # or real S3: https://s3.amazonaws.com export AWS_ACCESS_KEY_ID=your-access-key export AWS_SECRET_ACCESS_KEY=your-secret-key export AWS_REGION=us-east-1 # required for real S3 export RELATA_S3_BUCKET=relata-backups ``` Verify the config before starting the server: ```bash relata check # Prints: object_store: ok wal: ok chain: valid ... ``` For GCS or Azure Blob, the same `AWS_ENDPOINT_URL` pattern works via the object-store compatibility layer: ```bash # GCS via S3 compatibility AWS_ENDPOINT_URL=https://storage.googleapis.com \ AWS_ACCESS_KEY_ID=GOOGXXXXXXXXXXXXXXXX \ AWS_SECRET_ACCESS_KEY=xxxxxx \ RELATA_S3_BUCKET=my-gcs-bucket \ relata serve # Azure Blob via azurite (local) or production endpoint AWS_ENDPOINT_URL=http://azurite:10000/devstoreaccount1 \ AWS_ACCESS_KEY_ID=devstoreaccount1 \ AWS_SECRET_ACCESS_KEY=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw== \ RELATA_S3_BUCKET=relata \ relata serve ``` ## MinIO validation recipe Run this end-to-end before any production deployment to confirm object-store wiring is correct. ```bash # 1. Start MinIO docker run -d --name minio -p 9000:9000 -p 9001:9001 \ -e MINIO_ROOT_USER=minio \ -e MINIO_ROOT_PASSWORD=minio123 \ minio/minio server /data --console-address ":9001" # 2. Wait for MinIO to be healthy until curl -sf http://localhost:9000/minio/health/live; do sleep 1; done # 3. Set object-store env export AWS_ENDPOINT_URL=http://localhost:9000 export AWS_ACCESS_KEY_ID=minio export AWS_SECRET_ACCESS_KEY=minio123 export RELATA_S3_BUCKET=relata-test # 4. Verify config relata check # 5. Start server and ingest data RELATA_PROFILE=server RELATA_BEARER_TOKEN=test relata serve & sleep 3 curl -X POST http://localhost:9090/ingest \ -H "Authorization: Bearer test" \ -H "Content-Type: application/json" \ -d '{"object_type":"Person","data":[{"name":"Ada"},{"name":"Alan"}]}' relata query "SELECT * FROM Person LIMIT 5" # 6. Graceful shutdown — flushes WAL + writes Parquet snapshot kill -TERM $(pgrep -f "relata serve") sleep 5 # 7. Restart — should reload from object store RELATA_PROFILE=server RELATA_BEARER_TOKEN=test relata serve & sleep 3 # If this returns rows, durability is working relata query "SELECT * FROM Person LIMIT 5" # 8. Backup and restore relata backup relata reset reset reset # triple "reset" is a safety guard relata restore ~/.relata/store-*.json relata query "SELECT * FROM Person LIMIT 5" # Cleanup pkill -f "relata serve" docker rm -f minio ``` Step 7 is the critical check. Zero rows after restart means the object-store wiring is wrong — fix it before deploying. ## MinIO docker-compose For a persistent local MinIO setup alongside RelataDB: ```yaml services: relata: image: ghcr.io/relatadb/relata:latest restart: unless-stopped ports: - "9090:9090" environment: RELATA_PROFILE: server RELATA_BEARER_TOKEN: "change-me" RELATA_LAZY_RESTART: "true" RELATA_LOG_FORMAT: json AWS_ENDPOINT_URL: "http://minio:9000" AWS_ACCESS_KEY_ID: minio AWS_SECRET_ACCESS_KEY: minio123 RELATA_S3_BUCKET: relata-data depends_on: minio: condition: service_healthy minio: image: minio/minio:latest command: server /data --console-address ":9001" environment: MINIO_ROOT_USER: minio MINIO_ROOT_PASSWORD: minio123 ports: - "9000:9000" - "9001:9001" volumes: - minio-data:/data healthcheck: test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] interval: 5s timeout: 3s retries: 5 volumes: minio-data: ``` ## Graceful shutdown flow `SIGTERM` triggers an ordered shutdown: 1. Stop accepting new requests. 2. Drain in-flight queries (bounded timeout). 3. Flush the WAL. 4. Write a final Parquet snapshot to the object store. 5. Close listeners and release the `relata.lock` singleton. ```bash # Graceful shutdown kill -TERM $(pgrep -f "relata serve") # Force shutdown (WAL intact, Parquet snapshot skipped) kill -9 $(pgrep -f "relata serve") ``` Always use `SIGTERM` in production. Kubernetes `terminationGracePeriodSeconds` should be at least `30` to allow the WAL flush to complete. ## Per-write durability levels Control durability per-write with the `X-Relata-Durability` header: ```bash # Default: async — WAL durable, fsync deferred ~10ms curl -X POST http://localhost:9090/ingest \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \ -H "X-Relata-Durability: async" \ -H "Content-Type: application/json" \ -d '{"object_type":"Person","data":[{"name":"Ada"}]}' # Sync: fsync before ack — RPO = 0 curl -X POST http://localhost:9090/ingest \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \ -H "X-Relata-Durability: sync" \ -H "Content-Type: application/json" \ -d '{"object_type":"Person","data":[{"name":"Alan"}]}' ``` | Header value | fsync timing | Power-loss RPO | |---|---|---| | `async` (default) / `batch` / `interval` | Background flusher (~10 ms) | ≤ 10 ms of acked writes | | `sync` / `always` / `rpo0` | Before ack | 0 (no data loss) | Use `sync` for financial or compliance writes where you need RPO = 0. Use `async` (default) for bulk ingestion. ## PITR via object-store versioning Enable versioning on your S3 bucket to get point-in-time recovery for free — each Parquet segment flush writes a new object version. To restore to a point in time: ```bash # List object versions (AWS CLI example) aws s3api list-object-versions \ --bucket relata-backups \ --prefix relata/ \ --query 'Versions[?LastModified>=`2026-07-13T00:00:00`]' # Download a specific version aws s3api get-object \ --bucket relata-backups \ --key relata/manifest.json \ --version-id "xxxxxx" \ manifest-pitr.json ``` Then restore with the downloaded manifest: ```bash relata reset reset reset relata restore ./manifest-pitr.json relata check ``` ## Cold-restart RTO | Scenario | Time at 10 M rows | |---|---| | WAL + Parquet flush on SIGTERM | ~15 s | | Cold restart (eager, `RELATA_LAZY_RESTART=false`) | ~55 s | | Cold restart (lazy, `RELATA_LAZY_RESTART=true`) | O(manifest) — seconds | | Single-node RTO with lazy restart | <10 s to ready | `RELATA_LAZY_RESTART=true` is the default on `server` and `cluster`. It loads only the manifest catalog on startup — row data hydrates on demand. Set `RELATA_HYDRATE_RECENT_SEGMENTS=5` to pre-warm the 5 most recent segments for hot-path queries while staying mostly lazy. ## Singleton enforcement `relata serve` acquires an exclusive `flock` on `data_dir/relata.lock`. A second process mounting the same data directory fails immediately with a clear error rather than silently corrupting data. In Kubernetes, use a `StatefulSet` with `accessMode: ReadWriteOnce` to prevent two pods from racing on the same `PersistentVolumeClaim`. ## See also - [Configuration](/docs/guides/configuration) — storage and cold-load env vars - [Scaling](/docs/guides/scaling) — lazy restart, RAM walls, paged backends - [Observability](/docs/guides/observability) — WAL health metrics, audit chain ============================================================================== # Configuration URL: https://relatadb.dev/docs/guides/configuration ============================================================================== # Configuration RelataDB is configured through environment variables, with an optional `relata.toml` config file for persisted settings (search order: `--config ` flag, `RELATA_CONFIG` env var, `./relata.toml`, then `~/.relata/config.toml`). A bare `relata serve` starts in dev mode with safe defaults. Production requires three things: a profile, a bearer token, and (optionally) an object-store endpoint. > **Strict parsing.** Every `RELATA_*` value is validated at startup. > A malformed value (e.g. `RELATA_PORT=abc` or `RELATA_WAL_SYNC=true`) causes a > **FATAL startup error** — Relata refuses to boot rather than silently using a > wrong default. Removed/renamed vars (e.g. `RELATA_ORG_MODE`) also FATAL. > > **Operator tools:** > - `relata config --print-template` — emit a commented `.env` template of every `RELATA_*` var. > - `relata config --validate` — check your config before boot. > - `relata config --migrate` — auto-rewrite old var names to the current surface (run when upgrading from 1.x). > - `relata doctor` — 12 pre-flight checks (config, ports, KMS, disk). ## Pick a profile first `RELATA_PROFILE` is the master switch. Set it before anything else. ```bash # Local dev — no auth required, unbounded RAM, eager restart RELATA_PROFILE=free relata serve # Single-node production — 1 GB row-store RAM wall, disk-first walls on, requires bearer token RELATA_PROFILE=server RELATA_BEARER_TOKEN= relata serve # Multi-node (alpha) — same as server plus coordination RELATA_PROFILE=cluster RELATA_BEARER_TOKEN= relata serve ``` `lite` was a legacy alias for `free` and is now rejected outright — startup fails if `RELATA_PROFILE=lite` is set; use `free`. The `server` and `cluster` profiles refuse to start without `RELATA_BEARER_TOKEN` set. | Profile | RAM wall | Lazy restart | Disk-first walls | |---|---|---|---| | `free` | unbounded | off | off | | `server` | 1024 MB | on | on | | `cluster` | 1024 MB | on | on | ## Domain profile `RELATA_DOMAIN_PROFILE` loads the matching ontology overlay and default detector packs. ```bash RELATA_DOMAIN_PROFILE=finint relata serve ``` | Value | Use case | Default detectors | |---|---|---| | `enterprise` (default) | General business | `network,contact,crypto` | | `lea` | Law enforcement | network + identity | | `finint` | Financial intelligence | network + contact + crypto | | `security` | Security operations | network + device + crypto | | `custom` | Bring-your-own ontology | whatever you wire | ## Environment variable reference ### Profile and deployment | Variable | Default | Description | |---|---|---| | `RELATA_PROFILE` | `free` | Deployment shape: `free` \| `server` \| `cluster`. `lite` is removed — rejected at startup. | | `RELATA_DOMAIN_PROFILE` | `enterprise` | Ontology overlay: `enterprise` \| `lea` \| `finint` \| `security` \| `custom`. | | `RELATA_PORT` | `9090` | HTTP/gRPC listen port. | ### Auth | Variable | Default | Description | |---|---|---| | `RELATA_BEARER_TOKEN` | — | Unset = dev mode, no auth, pgwire disabled. Required on `server`/`cluster`. | | `RELATA_ADMIN_TOKEN` | — | Secondary token for `/admin/*` operations. | | `RELATA_AUTH_MODE` | — | `oidc` \| `oidc-verify` \| `saml` \| `mtls` for federated auth. | | `RELATA_PLAINTEXT_OK` | — | `true` to allow plain HTTP on `server`/`cluster` (not recommended). | ### Storage | Variable | Default | Description | |---|---|---| | `RELATA_DATA_DIR` | `./data/relata` | Root for WAL state and local object store. | | `RELATA_LOCAL_DATA_DIR` | — | Explicit local object-store path (dedicated volume). | | `AWS_ENDPOINT_URL` | — | S3-compatible endpoint. When set, overrides local disk. Also set `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `RELATA_S3_BUCKET`. | | `RELATA_IN_MEMORY` | — | `true` = in-memory only, data lost on restart. | | `RELATA_WAL_SYNC` | `interval` | Process-global WAL fsync mode: `always` (fsync on every flush, RPO≈0) \| `interval` (default — batched fsync every ~10 ms, RPO≈10 ms) \| `off` (page-cache only). **Any other value (including `true`/`1`) is a FATAL startup error.** | ### RAM and cache limits | Variable | Default (server/cluster) | Description | |---|---|---| | `RELATA_STORE_MAX_RAM_MB` | `1024` | Row-store RAM budget before spill to disk. Unbounded on `free`. | | `RELATA_DISKANN_MAX_RESIDENT` | `0` (unbounded) | Soft cap on RAM-resident vectors. | | `RELATA_MV_MAX_ROWS` | `1000000` | Max rows in an incremental materialized-view cache. `0` = unbounded. | | `RELATA_VECTOR_COLD_RESIDENT_MAX` | `100000` | Max staging vectors in an IVF cold bucket before spill. | | `RELATA_TOMBSTONE_CACHE_MAX_ROWS` | `1024` | Per-type tombstone cache size. Lower saves RAM; raises on-disk re-reads. | | `RELATA_DECODED_SEGMENT_CACHE_MAX` | `64` | Max decoded disk-segment entries in RAM. | ### Cold-start and restart | Variable | Default | Description | |---|---|---| | `RELATA_LAZY_RESTART` | `false` (`true` on server/cluster) | `true` loads manifest catalog only — O(manifest) not O(rows). | | `RELATA_HYDRATE_RECENT_SEGMENTS` | `0` | With lazy restart, pre-warm the newest N segments into RAM. `0` = fully lazy. | | `RELATA_FLUSH_SEGMENT_MAX_ROWS` | `250000` | Max rows per flushed Parquet segment. Larger deltas split into `ceil(delta/N)` segments. | ### Search and embedding | Variable | Default | Description | |---|---|---| | `RELATA_SEARCH_PRESET` | `balanced` | BM25/FTS fuzzy expansion: `strict` \| `balanced` \| `lenient`. | | `RELATA_EMBED_BATCH_SIZE` | `32` | Texts per `embed()` call per drain cycle. Higher = fewer round-trips, more first-result latency. | | `RELATA_EMBED_CONCURRENCY` | `4` | Concurrent drain-worker tasks. | | `RELATA_EMBED_QUEUE_MAX` | `100000` | Hard cap on embedding backlog. | | `RELATA_EMBED_TIMEOUT_MS` | `30000` | Timeout for embedder sidecar HTTP calls. | | `RELATA_EMBED_CIRCUIT_COOLDOWN_MS` | `60000` | After 5 consecutive errors, circuit opens; `/health/ready` returns `503`. | | `RELATA_DETECT_BATCH_SIZE` | `256` | Chunk size for batched identity detection. | ### SmartIngest detector packs ```bash # Default RELATA_DETECT_PACKS=network,contact,crypto # Add financial and payment RELATA_DETECT_PACKS=network,contact,crypto,financial,payment # All packs RELATA_DETECT_PACKS=all # Disable detection RELATA_DETECT_PACKS=none ``` Available opt-in additions: `financial`, `payment`, `social`, `transport`, `device`, `ics`. ### Observability | Variable | Default | Description | |---|---|---| | `RELATA_LOG_FORMAT` | `pretty` | `pretty` for terminals, `json` for log shippers. | | `RELATA_LOG_LEVEL` | `info` | `trace` \| `debug` \| `info` \| `warn` \| `error`. | | `RELATA_OTLP_ENDPOINT` | — | OTLP/HTTP trace exporter, e.g. `http://otel-collector:4318/v1/traces`. Unset = OTel fully disabled. | | `RELATA_OTLP_SAMPLE_RATIO` | `0.01` | TraceID-ratio sampler. `1.0` = sample everything. | | `RELATA_METRICS_PUBLIC` | — | `true` serves `/metrics` without a bearer token (for Prometheus scrapers behind mTLS/NetworkPolicy). | ### Rate limits | Variable | Default | Description | |---|---|---| | `RELATA_RATE_LIMIT_RPS` | profile-dependent | Per-principal requests/sec cap. | | `RELATA_RATE_LIMIT_AUTH_FAIL_RPS` | profile-dependent | Auth-failure rate cap. `0` is treated as `1`. Use `99999` to disable. | To disable rate limits for load tests or air-gapped demos: ```bash RELATA_RATE_LIMIT_RPS=99999 RELATA_RATE_LIMIT_AUTH_FAIL_RPS=99999 relata serve ``` ### Purpose enforcement | Variable | Default | Description | |---|---|---| | `RELATA_PURPOSE_MODE` | `open` | `strict` = only registered purposes accepted; `open` = any non-empty string. | | `RELATA_PURPOSES` | — | Comma-separated registered purpose IDs, e.g. `analytics,audit,compliance`. | ## Minimal production docker-compose Copy this, replace the token and bucket values, and run `docker compose up -d`. ```yaml services: relata: image: ghcr.io/relatadb/relata:latest restart: unless-stopped ports: - "9090:9090" environment: RELATA_PROFILE: server RELATA_BEARER_TOKEN: "change-me-use-openssl-rand-hex-32" RELATA_PORT: "9090" RELATA_DOMAIN_PROFILE: enterprise RELATA_PURPOSE_MODE: strict RELATA_PURPOSES: analytics,audit,compliance RELATA_STORE_MAX_RAM_MB: "2048" RELATA_LAZY_RESTART: "true" RELATA_HYDRATE_RECENT_SEGMENTS: "5" RELATA_LOG_FORMAT: json RELATA_LOG_LEVEL: info RELATA_OTLP_ENDPOINT: "http://otel-collector:4318/v1/traces" RELATA_METRICS_PUBLIC: "true" AWS_ENDPOINT_URL: "http://minio:9000" AWS_ACCESS_KEY_ID: minio AWS_SECRET_ACCESS_KEY: minio123 RELATA_S3_BUCKET: relata-data volumes: - relata-data:/var/lib/relata healthcheck: test: ["CMD", "curl", "-f", "http://localhost:9090/health"] interval: 10s timeout: 5s retries: 3 depends_on: minio: condition: service_healthy minio: image: minio/minio:latest command: server /data --console-address ":9001" environment: MINIO_ROOT_USER: minio MINIO_ROOT_PASSWORD: minio123 ports: - "9000:9000" - "9001:9001" volumes: - minio-data:/data healthcheck: test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] interval: 10s timeout: 5s retries: 3 volumes: relata-data: minio-data: ``` Verify the node is healthy after startup: ```bash curl http://localhost:9090/health curl http://localhost:9090/health/ready curl http://localhost:9090/version ``` ## Singleton enforcement `relata serve` acquires an exclusive `flock` on `data_dir/relata.lock`. If two pods share the same PVC, the second one fails with a clear error rather than silently splitting writes. This is especially important in Kubernetes rolling updates — use `RollingUpdate` strategy with `maxUnavailable: 1` or switch to a `StatefulSet`. ## See also - [Auth & Security](/docs/guides/security) — bearer tokens, OIDC, mTLS, ACL - [Backup & Restore](/docs/guides/backup-restore) — WAL, Parquet, MinIO setup - [Scaling](/docs/guides/scaling) — RAM walls, paged backends, lazy restart - [Observability](/docs/guides/observability) — logs, metrics, traces, health probes ============================================================================== # Connectors & Extensions URL: https://relatadb.dev/docs/guides/connectors ============================================================================== # Connectors & Extensions Relata does **not** fetch from external sources inside the database process. Per the **ETL Boundary** policy, this repository ships **canonical type contracts + ingest trait impls + schema validators only** — zero network dependencies in the signed binary. The actual fetching (polling, OAuth, vendor SDKs, webhooks) lives in **[datagrep](https://github.com/relatadb/datagrep)**, the external ETL tool that consumes these contracts, fetches from each source, and pushes governed rows into Relata. This keeps the binary SLSA-clean and credentials out of the DB process. Relata's role at the boundary: **map incoming data onto the identity model and build the governed graph** — canonicalize → detect identities (SmartIngest) → build the graph → attach provenance (PROV-O) on every row. ## The extension framework Six extension kinds share one framework — one Manifest, one signing chain, one principal + ACL + audit contract: | Kind | Role | |---|---| | **Connector** | Ingest from an external source | | **Detector** | SmartIngest identity detection (two-phase) | | **Enricher** | Augment rows from registered enrichment tables / rules | | **Scorer** | Analytics scorers (social-media, behavioural — 13 `ScorerOp`s) | | **Job** | Scheduled / event-triggered maintenance + detection jobs | | **Report** | Typed, signed analytical reports | Three deployment modes: | Mode | Runtime | Notes | |---|---|---| | **builtin** | Rust crate (`relata-connector-`) | Compiled in; enabled via `--enable-connector=` | | **WASM** | Wasmtime Component Model + WASI 0.2 | Language-agnostic guest; typed WIT contract; capability-sandboxed; fuel/memory-capped (≤ 2× native overhead p50; ≤ 20% ingest throughput hit with 3 detectors) | | **external** | gRPC / HTTP | Sidecar process | ## The Connector trait A typed `Connector` trait in `relata-core::connectors` enforces the bi-temporal + provenance contract at the connector boundary, preventing untagged rows: ```rust pub trait Connector { fn name(&self) -> &'static str; fn schema(&self) -> HashMap; // field → canonical-type label fn ingest(&self, batch: ConnectorBatch) -> Result, ConnectorError>; } ``` `ConnectorBatch` carries `object_type`, a `BiTemporalRange`, a `ProvenanceRef`, and `Vec` — so connectors never need to know the object-store API. `ConnectorError` has three variants: `InvalidSchema`, `IngestFailed`, `Io`. The reference impls live in `relata-connector-stub` (`NoopConnector` for tests, `DbtConnector` which parses dbt NDJSON model export). ### dbt adapter The dbt path parses a pre-exported dbt NDJSON model and maps JSON values to `CanonicalValue` (`DbtConnector::parse_ndjson`), making it testable without a running dbt instance. Full `dbt run` integration ships in the Python `sdks/python/dbt_relata/` package. ## Migration connectors: `relata import --from` Migrate an existing database straight into the governed identity fabric — no CSV export step. Every row lands through `POST /ingest/bulk` → `governed_upsert_many`, the same write path every protocol door uses, so SmartIngest identity detection, registered ingest pipelines, ACL, tenant-ownership, and audit logging all apply. ```bash # Postgres is a real, wired connector relata import --from postgres \ --dsn "postgresql://user:pass@localhost:5432/appdb" \ --table users --type Person \ --dry-run # prints ≤5 mapped rows, opens no write transaction ``` | Source | Status | |---|---| | `postgres` | **Live** — server-side cursor, streaming `FETCH` pages, type-faithful JSON (numeric/decimal kept as exact text), single-column PK → `_pk` | | `csv` / `ndjson` | **Live** — `relata import --from csv --file ` | | `neo4j`, `mongo`, `clickhouse` | **Honest stubs** — no driver wired; each prints the CSV/NDJSON export workaround and exits non-zero (never a silent no-op). Use the documented workaround today. | Key v1 limitations: `--type` must name an already-registered governed type (no DDL); TLS is not wired (`NoTls`); quoted/mixed-case identifiers rejected; no relationship/FK import yet (one table → one governed type per invocation). ## Connector catalog ~120 catalogued type contracts across telco, social, news, email, documents, threat-intel, cyber telemetry, cloud audit, financial/AML, blockchain, identity/KYC, geospatial, and streams. **Three-tier status** per entry: - ✅ **live** — registered + fetches (`noop`, `dbt`; MISP/TAXII/STIX bundle import via `POST /import?format=stix`) - 🟡 **registered stub** — `?connector=` resolves; returns a clear `not yet implemented` (never a silent no-op); fetch lives in datagrep - 🔲 **catalog spec** — canonical type contract documented for datagrep; not yet registered Highlights that are **live in this repo** (not through datagrep): | Source | Door | Notes | |---|---|---| | CSV / NDJSON / JSON array | `POST /ingest` | Auto-detected by first byte | | Kafka | `KafkaIngestAdapter` | Pure-Rust wire-protocol client (no `rdkafka`/`unsafe`) — see [Ingestion](/docs/guides/ingestion) | | CDR | `POST /ingest/cdr` | Typed fast path for call-detail records — see [Ingestion](/docs/guides/ingestion) | | STIX bundle | `POST /import?format=stix` | Threat-intel object import | | MISP | `relata misp-pull` | Pulls via MISP restSearch API | | TAXII 2.1 | `relata taxii-poll` | Polls a collection's STIX objects | | Sigma rule import | `relata import-sigma` | Detection-rule import | | Sanctions pull | `relata pull-ioc` | OFAC/UN/EU/OFSI/MEA/RBI/OpenSanctions | The full catalog (telco IPDR/tower-dump/SMS/EDR, OSINT 13, FHIR R4, AIS stream, FININT 40, cloud audit, cyber telemetry, etc.) is documented as canonical type contracts in the source repo's connector catalog. For sources not in the catalog, author a custom connector as a `relata-connector-` crate implementing `Connector`. ## Ontology packs (domain packs) Packs bundle a domain's ontology types, detectors, jobs, reports, and detection rules into a signed, versioned unit. The repo ships ~20 domain packs + ~25 jurisdiction packs; the portal's [use-cases](/docs/use-cases/appdev-governed-rag) map to these: | Pack | Domain | |---|---| | `finint` | Financial intelligence / AML (sanctions, PEP, wires, crypto) | | `cyber` | Cyber threat intel, C2/beacon detection | | `counter_terror` | Counter-terrorism pattern detection | | `counter_intel` | Counter-intelligence | | `lea` | Law-enforcement telco (CDR, IPDR, tower dump) | | `maritime` | AIS, dark-fleet detection | | `narcotics` | Narcotics supply-chain patterns | | `border` | Border-crossing analysis | | `defense` | Defense / military intelligence | | `geopolitics`, `gcc_mena`, `india`, `health`, `aml` | Regional / sectoral | | `jurisdiction-` (25) | Per-jurisdiction legal type sets | `relata-pack-stub` demonstrates the pack layout for authoring new packs. ## Field-mapping convention (all connectors) Every connector follows the same canonicalization recipe: 1. **Parse** the source field to its native type. 2. **Validate** against the canonical type's checksum/format/registry (76 canonical types — email, IBAN, MMSI, VIN, IMEI…). 3. **Canonicalize** to the binary representation (uint32 for IPv4, E.164 uint64 for phone, S2 cell for geo…). 4. **Attach provenance** — `(source_connector_id, file_or_batch_id, record_offset, observed_at, recorded_at)`. 5. **Index in `IdentityIndex`** with `observed_in = (object_type, object_id, property_path)`. 6. **Quarantine** on validation failure (strict reject / permissive auto-create / auto-map suggest). ## See also - [Ingestion & SmartIngest](/docs/guides/ingestion) — Kafka, CDR, ingest pipelines, `relata import` - [Identity](/docs/concepts/identity) — canonical types and entity resolution - [Jobs, Workflows & Detection](/docs/guides/jobs-workflows) — the Job/Report extension kinds - [SQL Reference](/docs/reference/sql) — `DETECT_IDENTITIES`, `RESOLVE_IDENTITY`, domain TVFs ============================================================================== # Detection rules — ingest → detect → investigate in one binary URL: https://relatadb.dev/docs/guides/detection-rules ============================================================================== # Detection rules — ingest → detect → investigate in one binary Relata collapses the SIEM + sidecar + database triangle into one engine. A detection rule is a SQL `WHERE` against any governed type; it fires **within the request cycle on commit** (not on a 30-second poll), emits governed `Alert` rows with full PROV-O provenance, and pushes to per-tenant webhooks with queryable delivery status. Import Sigma's 10 000+ community detection rules as-is. The result: ingest → detect → investigate is one closed loop, and alerts are bi-temporal — "would this rule have fired last Tuesday?" is an `AS OF` query. ## Create a rule ```bash curl -X POST http://127.0.0.1:9090/rules \ -H 'Authorization: Bearer ' -H 'Content-Type: application/json' \ -d '{ "name": "suspicious-dns-exfil", "target_type": "DnsEvent", "condition_sql": "query_length > 100 AND rdata_type = '\''TXT'\'' AND query LIKE '\''%.xyz'\''", "severity": "high", "mitre_technique": "T1048", "purpose": "security" }' ``` ```python # Python SDK client.governance_client.create_rule({ "name": "suspicious-dns-exfil", "target_type": "DnsEvent", "condition_sql": "query_length > 100 AND rdata_type = 'TXT'", "severity": "high", "mitre_technique": "T1048", }, purpose="security") ``` From then on, every governed `DnsEvent` ingest that matches the condition produces an `Alert` row — in the **same request**, before the ingest returns. No poller, no lag. ## Commit-driven firing (the differentiator) Most rule engines run on a cron/poll loop — ingest a row now, the alert lands 30 seconds later (or never, if the poller is down). Relata fires rules on the **commit bus** (the `GraphChangeEvent` stream, in `crates/relata-cli/src/serve/rule_eval.rs`) — the candidate check runs as part of the write transaction, so: - **Latency is sub-request.** By the time `POST /ingest` returns `200`, any alerts it triggered are already in the audit log. - **No missed events.** If the server is up enough to accept the write, it's up enough to fire the rule. A down poller can't silently drop detections. - **Bitmap-indexed candidate filter.** Each rule compiles to a bitmap predicate over the target type, so the commit-time check is a cheap set-membership test against just-committed rows — not a full table scan per rule. ## Import Sigma rules (10000+ community detections, as-is) Sigma is the open standard for generic detection rules — the community publishes thousands for known attacker TTPs. Relata imports them natively: ```bash relata import-sigma rules/suspicious-powershell.yaml # or via HTTP curl -X POST http://127.0.0.1:9090/rules/sigma \ -H 'Authorization: Bearer ' -H 'Content-Type: application/x-yaml' \ --data-binary @rules/suspicious-powershell.yaml ``` ```python client.governance_client.import_sigma(open("rules/suspicious-powershell.yaml").read(), purpose="security") ``` Each Sigma rule's logsource → detection → fields map onto your registered governed types and the SQL `WHERE` condition. ## Bi-temporal alerts — backtest and "what would have fired?" `Alert` rows are bi-temporal just like every other governed row. Run the rule as-of a past window to backtest detection quality without re-ingesting: ```sql PURPOSE 'security' SELECT alert_id, rule_name, severity, target_id, valid_from FROM Alert AS OF '2026-01-15T00:00:00' WHERE severity IN ('high', 'critical') ORDER BY valid_from DESC; ``` This is impossible in a SIEM that overwrites alerts — the historical alert state is gone the moment it ages out. Here it's a first-class query. ## Per-tenant webhooks with delivery status Configure a webhook per tenant; alerts push automatically. Crucially, the **delivery status is queryable** — worst-outcome-wins ranking collapses per-attempt outcomes into one `AlertRecord.delivery_status`: ```bash # Find alerts whose webhook delivery failed (retry loop exhausted) curl 'http://127.0.0.1:9090/alerts/list?delivery_status=failed' \ -H 'Authorization: Bearer ' ``` ```python # Python — register a webhook, then inspect delivery health client.register_webhook("https://soar.example.com/relata-alerts", event_types=["alert.high"]) failed = client.query("SELECT * FROM Alert WHERE delivery_status = 'failed'") ``` The retry loop (`deliver_webhook_retry_loop`) returns a typed `WebhookDeliveryOutcome` per attempt; failures are logged and recoverable, not silently lost. ## Stream alerts in real time For a live dashboard or a SOAR integration, stream alerts over SSE: ```python # Python for event in client.streaming_client.alerts(): print(event["severity"], event["rule_name"], event["target_id"]) ``` ```bash # Raw SSE curl -N http://127.0.0.1:9090/alerts/stream -H 'Authorization: Bearer ' ``` ## Tips & takeaways - **Write the `condition_sql` to be selective.** The commit-time bitmap filter is fast, but a rule that matches 50% of rows fires constantly and floods the audit log. Use specific thresholds and `AND` conjunctions. - **Pair rules with `mitre_technique`.** Tagging alerts with MITRE ATT&CK technique IDs makes them correlatable across rules and against threat-intel — and the field is queryable. - **Use backtesting before going live.** `SELECT FROM Alert AS OF ''` against a newly-created rule tells you what it *would* have fired — tune thresholds before the rule touches production. - **Webhook delivery status is your safety net.** Set up a periodic `?delivery_status=failed` check so a SOAR outage doesn't silently drop alerts. - **Sigma import is the fast path to coverage.** Before authoring custom rules, search the Sigma repo for your log source — chances are someone already wrote the detection. ## See also - [Use case: Cyber Sigma detection](/docs/use-cases/cyber-sigma-detection) — end-to-end worked example - [Jobs, Workflows & Detection](/docs/guides/jobs-workflows) — the typed-Job engine and scheduled detection - [Governance](/docs/concepts/governance) — ACL, purpose, and the audit chain alerts inherit - [Bi-temporal queries](/docs/reference/bitemporal) — `AS OF` alert backtesting - [Connectors & Extensions](/docs/guides/connectors) — Sigma + STIX + MISP + TAXII ingest paths ============================================================================== # Guides URL: https://relatadb.dev/docs/guides ============================================================================== # Guides Task-oriented walkthroughs for running RelataDB in production. Each guide is self-contained — pick the job you need to do. ## Configuring & ingesting - [Configuration](/docs/guides/configuration) — the full `RELATA_*` environment-variable surface - [LLM & Embedding Configuration](/docs/guides/llm-embedding) — wiring embedder sidecars and learned models - [Ingestion & SmartIngest](/docs/guides/ingestion) — getting data in and how identity detection runs - [Connectors & Extensions](/docs/guides/connectors) — pluggable sources and extension points ## Securing & isolating - [Auth & Security](/docs/guides/security) — bearer tokens, TLS, admin surfaces - [Per-Door ACL](/docs/guides/per-door-acl) — enforcing governance on each protocol door - [S3 Door](/docs/guides/s3-door) — exposing the S3-compatible door safely - [Multi-Tenancy](/docs/guides/multi-tenancy) — tenant onboarding and isolation - [Privacy & GDPR](/docs/guides/privacy) — erasure, WORM export, and trusted time ## Detecting & monitoring - [Detection Rules](/docs/guides/detection-rules) — Sigma-style rule evaluation over governed data - [Anomaly Monitoring](/docs/guides/anomaly-monitoring) — baseline drift and alerting - [Multimedia Search](/docs/guides/multimedia-search) — image / audio / video retrieval - [Observability](/docs/guides/observability) — metrics, traces, and flamegraphs ## Operating - [Backup & Restore](/docs/guides/backup-restore) — snapshots, point-in-time recovery - [Jobs, Workflows & Detection](/docs/guides/jobs-workflows) — orchestrating typed jobs - [Scaling](/docs/guides/scaling) — capacity planning and the cluster profile - [Upgrading & Migration](/docs/guides/upgrading) — version-to-version migration notes - [Troubleshooting](/docs/guides/troubleshooting) — diagnosing common issues See also: the [Reference](/docs/reference) for exact configuration values and [Deployment](/docs/deployment) for topology. ============================================================================== # Ingestion & SmartIngest URL: https://relatadb.dev/docs/guides/ingestion ============================================================================== # Ingestion & SmartIngest Relata supports multiple ingestion paths. **All writes are governed** (ACL, provenance, WAL-durable) and **bi-temporal** (every row carries `valid_from/to` + `system_from/to`), regardless of which door the data enters through. Every path funnels into the same `governed_upsert_*` write pipeline, so SmartIngest identity detection, registered ingest pipelines, and audit logging apply uniformly. | Path | Door | Best for | |---|---|---| | CSV / JSON / NDJSON | `relata ingest` · `POST /ingest` | Bulk files, ad-hoc rows | | Document (auto-chunk + embed) | `POST /ingest/document` | RAG, PDFs | | Media (image/audio/video) | `POST /ingest/media` | Multimodal vector search | | **Kafka** | `KafkaIngestAdapter` | Streaming, CDC | | **CDR** | `POST /ingest/cdr` · `relata cdr` | High-volume telco call records | | **Ingest pipelines** | `POST /pipelines` | Pre-write field transforms (grok/dissect/...) | | **`relata import --from`** | CLI | Migrate from Postgres/CSV | | OTLP | `/v1/{traces,logs,metrics}` | OpenTelemetry | ## CSV / JSON ingest ```bash # CLI relata ingest data.csv --type Person # HTTP — body format is auto-detected by first byte (no Content-Type change needed): # CSV (default) · NDJSON (leading '{') · JSON array (leading '[') curl -X POST http://localhost:9090/ingest \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \ -H "Content-Type: application/json" \ -d '{"object_type":"Person","data":[{"name":"Ada","email":"ada@example.com"}]}' ``` > The `purpose` parameter on `/ingest` is validated against `^[a-zA-Z_][a-zA-Z0-9_]{0,63}$` — use underscores, not hyphens (e.g. `threat_intel`, not `threat-intel`); max 64 chars. ## Document ingest (auto-chunking + embedding) ```bash curl -X POST http://localhost:9090/ingest/document \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \ -H "Content-Type: application/json" \ -d '{"source":"report.pdf","content":"...base64...","auto_chunk":true}' ``` Documents are automatically chunked, embedded, and indexed for vector + BM25 search. **Since v1.1 the ingest hot path no longer embeds text rows** — embeddings are either caller-supplied (`_emb_text` in the row payload) or populated asynchronously by the embedder sidecar via the media-worker drain. See [LLM & Embedding configuration](/docs/guides/llm-embedding). ## Media ingest Images, audio, and video are processed through modality-specific embedders and stored as governed blobs. Use `POST /ingest/media?modality=image|audio|video` (base64 body) or include `_emb_*` vectors inline. Perceptual-hash dedup is applied automatically. See [LLM & Embedding configuration](/docs/guides/llm-embedding) for the sidecar contract. ## Kafka adapter A pure-Rust Kafka consumer (`KafkaIngestAdapter` in `relata-storage::kafka`) — **no `rdkafka` / librdkafka C FFI**, so no `unsafe` on the poll path (repository invariants forbid `unsafe` entirely). It speaks the Kafka binary wire protocol directly for three API keys: Metadata (partition-leader discovery), Fetch (record batches), and OffsetCommit (consumer-group offset durability → at-least-once). ```bash # Configure via env (the adapter runs in the server process) RELATA_KAFKA_BOOTSTRAP=kafka:9092 \ RELATA_KAFKA_TOPIC=events \ RELATA_KAFKA_GROUP=relata-ingest \ RELATA_KAFKA_PARTITION=0 \ relata serve ``` Each consumed record flows through the same `governed_upsert_many` batch path as `/ingest/bulk`, so registered [ingest pipelines](#ingest-pipelines) run on every Kafka record. The legacy in-memory `StreamingAdapter` is retained for tests; `KafkaIngestAdapter` is the production path. ## Ingest pipelines Pipelines define **pre-write field transforms** that run before a row is committed — keeping raw-to-structured normalization inside Relata's governance boundary rather than an external pre-processor. Every write path — single-row `governed_upsert_durable`, batch `governed_upsert_many` (the path behind `/ingest/bulk`, `/ingest/cdr`, `/ingest/logs`, `/ingest/metrics`, and the Kafka drain) — runs matching pipelines before validation/persist. **Five built-in processors:** `dissect`, `grok`, `date`, `fingerprint`, `community_id`. ```http POST /pipelines Authorization: Bearer Content-Type: application/json { "name": "firewall-flow", "target_type": "NetworkFlow", "on_failure": "keep", "processors": [ {"kind": "dissect", "field": "raw", "pattern": "%{src_ip}:%{src_port} -> %{dst_ip}:%{dst_port}"}, {"kind": "fingerprint", "fields": ["src_ip", "dst_ip"]}, {"kind": "community_id"} ] } ``` | Processor | Purpose | |---|---| | `dissect` | Token-based field extraction: `%{field_name}` placeholders | | `grok` | Named-capture extraction: `%{SYNTAX:field_name}` (SYNTAX accepted for readability, not used to constrain matching) | | `date` | Parse a timestamp (i64 ns, RFC 3339, or `YYYY-MM-DD HH:MM:SS`) → Unix ns in `target_field` | | `fingerprint` | SHA-256 of selected fields → `target_field` (idempotent re-ingest dedup) | | `community_id` | Canonical direction-agnostic 5-tuple hash (`1:`) for network flows | A pipeline can only **add** fields the caller didn't already send — it never overwrites. `on_failure` is `keep` (default: log + continue) or `drop` (abort + drop the row). List with `GET /pipelines`. > Pipelines are **in-memory only** — lost on restart (persist-on-shutdown is a tracked follow-up). `DEFINE PIPELINE` SQL does not exist; `POST /pipelines` is the only registration path. ## CDR fast path A typed fast path for telco call-detail records — the highest-volume telco workload. `CdrRecord` is a flat struct (~80 bytes/row vs ~200 bytes for the generic `HashMap` row) with MSISDNs as `u64` and timestamps as `i64` ns, behind a non-blocking `IngestQueue`. ```bash curl -X POST "http://localhost:9090/ingest/cdr?purpose=investigation" \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \ -H "Content-Type: text/csv" \ --data-binary @cdrs.csv # or via the CLI helper relata cdr ingest calls.csv [--purpose law_enforcement] relata cdr analyze +919876543210 # common-contact hand-off analysis relata cdr timeline +919876543210 # most-recent-calls timeline ``` `purpose` is **required**. Column names are case-insensitive and accept aliases (`caller`/`a_number`/`a_msisdn` → `caller_msisdn`; `start`/`call_start`/`timestamp` → `call_start_ns`; etc.). MSISDNs are normalised (non-digit chars stripped; >15 digits = parse error). A timestamp `< 10^13` is treated as epoch **seconds** and scaled to ns; `>= 10^13` as ns already. ## `relata import --from` (migration connectors) Migrate an existing database straight into the governed identity fabric — no CSV export step. Postgres is a real, wired connector; `neo4j`/`mongo`/`clickhouse` are honest stubs (each prints the CSV/NDJSON workaround and exits non-zero). ```bash relata import --from postgres \ --dsn "postgresql://user:pass@localhost:5432/appdb" \ --table users --type Person \ --batch 500 --on-conflict overwrite \ --token "$RELATA_BEARER_TOKEN" --url "http://localhost:9090" ``` The connector opens a read-only transaction and a server-side cursor, `FETCH`es `--batch`-sized pages (streaming, bounded memory), and POSTs each page as one `/ingest/bulk` request. Type-faithful JSON: numeric/decimal kept as exact text (no precision loss on money columns). The source PK maps to `_pk` so `--on-conflict overwrite` updates existing rows. Use `--dry-run` to preview ≤5 mapped rows without writing. See [Connectors & Extensions](/docs/guides/connectors) for the full migration story. ## Embedder-sidecar lifecycle The ingest hot path does not embed. Vectors are either **caller-supplied** (`_emb_text`/`_emb_image`/... in the row payload) or populated **asynchronously** by the embedder sidecar: set `RELATA_ACCEL_ENDPOINT` and the media-worker drain cycle populates `_emb_*` off the request thread (typically sub-second). The built-in CPU embedder (128-dim, deterministic) is **query-side only** — it embeds the `recall()` search query when no sidecar is configured. Circuit breaker opens after 3 consecutive failures (`"embedder circuit open"`, 60 s cooldown). Full contract + reference Python sidecar: [LLM & Embedding configuration](/docs/guides/llm-embedding). ## SmartIngest identity pipeline SmartIngest runs on every ingest automatically — no configuration beyond registering canonical types and optional enrichment tables: 1. **Identity extraction** — emails, phone numbers, IBANs, MMSIs, VINs, IMEIs are detected from free-text fields using 76 canonical-type validators (eager regex/pattern phase; lazy detection deferred to materialized views). 2. **Entity matching** — extracted identifiers are matched against the existing entity graph to detect duplicates or known entities. 3. **Entity merge** — if a match is found, the new data is linked to the existing entity rather than creating a duplicate (`FUSE_IDENTITIES`). 4. **Enrichment** — registered enrichment lookup tables (CSV-backed) can augment rows at ingest time. Tune detection latency with `RELATA_ENRICH_MODE` / `RELATA_LAZY_TYPES` (eager inline vs lazy background job). See `DETECT_IDENTITIES` in the [SQL reference](/docs/reference/sql). ## OTLP ingest (traces/logs/metrics) OpenTelemetry-compatible ingest endpoints: | Endpoint | Protocol | |---|---| | `/v1/traces` · `/v1/logs` · `/v1/metrics` | OTLP JSON | | `/ingest/traces` · `/ingest/logs` · `/ingest/metrics` | Relata native | ## Enrichment lookup tables Register a CSV as a query-time enrichment table: ```bash curl -X POST http://localhost:9090/lookup/register \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \ -H "Content-Type: application/json" \ -d '{"name":"country_codes","data":[{"code":"US","name":"United States"},...]}' ``` Reference in SQL — see `REGISTER LOOKUP` / `LOOKUP` in the [SQL reference](/docs/reference/sql) and the worked example in the [Query cookbook](/docs/reference/query-cookbook). ## See also - [Connectors & Extensions](/docs/guides/connectors) — Connector trait, dbt adapter, migration connectors, packs - [LLM & Embedding configuration](/docs/guides/llm-embedding) — sidecar contract, `_emb_*` lifecycle - [Identity](/docs/concepts/identity) — canonical types and entity resolution - [Search & Retrieval](/docs/reference/search) — how ingested data becomes searchable - [Configuration](/docs/guides/configuration) — ingest-related env vars ============================================================================== # Jobs, Workflows & Detection URL: https://relatadb.dev/docs/guides/jobs-workflows ============================================================================== # Jobs, Workflows & Detection Relata ships a unified **extension + job + report framework** — six extension kinds (Connector, Detector, Enricher, Scorer, Job, Report) sharing one signed Manifest, one ACL/principal/audit contract, and one scheduler plane. This page covers the detection/analysis surface: pattern-detection jobs, the detection-rules engine, governance-aware workflows, intelligence feeds, and the domain packs that bundle them. ## Pattern detection Four typed **continuous `Job` extensions** run on a schedule, scan ingested data for investigation-significant patterns, and emit typed `AlertEvent` records with full PROV-O provenance. Each runs under the `BATCH` scheduler class (yields to interactive), writes alerts that inherit the strictest classification of the data that triggered them, and is validated by `ExtensionShakedown` against a `GoldenDataset` before promotion from `shadow` to `live`. | Job | Schedule | Scans | Detects | |---|---|---|---| | `C2BeaconDetectJob` | hourly | `NetFlowEvent`, `DnsQueryEvent` | Periodic outbound traffic (coefficient-of-variation < 0.15, small dest-IP set, consistent payload) → `C2_BEACON` | | `ConvoyDetectJob` | every 30 min | `MovementEvent`, `CellAttachEvent` | ≥3 entities within 500 m moving together ≥15 min → `CONVOY` | | `TransactionRingDetectJob` | hourly | transaction graph (`WireTransferHop`, `UpiTxn`, `ImpsTxn`, `CryptoTxn`) | Circular flow A→B→C→A (depth ≤6, ≥ tenant threshold) → `TRANSACTION_RING` | | `ContradictionDetectJob` | daily | entity/link/event assertions | Same `(entity, property)` with conflicting values from reliable sources, not explained by bi-temporal succession → `CONTRADICTION` | Jobs are discoverable via `SUGGEST_EXTENSIONS()` and surface in the MCP `list_jobs` / `job_status` / `schedule_job` tools. The detection algorithms live in `crates/relata-jobs/src/pattern_detection/`. ## Detection rules engine A lightweight detection-rules engine complements the typed pattern-detection jobs. Rules are continuously evaluated against new data; each is defined in SQL or Sigma-compatible YAML. ```bash # Register a detection rule (SQL condition against a target type) curl -X POST http://localhost:9090/rules \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "large-transfer-flag", "trigger": { "type": "Transaction", "condition": "amount > 100000 AND currency = '\''USD'\''" }, "action": "flag" }' ``` ### Sigma rule import Sigma is the vendor-neutral signature format for SIEM detection rules. Import Sigma YAML directly: ```bash relata import-sigma rules/suspicious_dns.yaml ``` Rules must pass a precision/recall gate against a golden dataset before promotion from `shadow` to `live`. ### Rule lifecycle | Endpoint | Method | Description | |---|---|---| | `/rules` | `GET` / `POST` | List / create rules | | `/rules/:id` | `DELETE` | Disable/delete rule | | `/rules/:id/snooze` | `POST` | Temporarily disable | | `/rules/:id/suppress` | `POST` | Suppress alerts | | `/rules/:id/exceptions` | `POST` | Add exception | | `/rules/:id/tuning` | `GET` | Tuning suggestions | | `/alerts/list` | `GET` | List fired alerts | | `/alerts/update/:id` | `PATCH` | Update alert status | Rule conditions are validated at create + eval time (rejects `;`, `--`, `/* */`; rejects `UNION`/`INTERSECT`/`EXCEPT` adjacent to punctuation; 2048-byte length cap) to prevent SQL injection through rule text. ### Detection modes - **`live`** — alerts fire immediately when data matches - **`shadow`** — alerts are logged but not surfaced (for validation before promotion) - **`disabled`** — rule is inactive ## Workflows (governance-aware DAGs) Workflows are DAGs of steps that automate multi-stage analysis: ```bash curl -X POST http://localhost:9090/workflows \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "fraud-investigation", "steps": [ {"name": "detect", "type": "rule", "rule": "large-transfer-flag"}, {"name": "enrich", "type": "lookup", "table": "sanctions_list"}, {"name": "report", "type": "report", "template": "sars-template"} ] }' ``` | Endpoint | Method | Description | |---|---|---| | `/workflows` | `GET` / `POST` | List / register workflows | | `/workflows/:name` | `GET` | Get definition | | `/workflows/:name/run` | `POST` | Trigger a run | | `/workflows/runs/:run_id` | `GET` | Check run status | **Governance:** every workflow step inherits the tenant context, PURPOSE, and ACL of the triggering request. Steps that would violate governance are blocked — workflows cannot bypass policy. ## The crates behind it | Crate | Role | |---|---| | `relata-jobs` | The Job/Report extension framework + 42 built-in jobs across 7 categories (storage hygiene, correctness, retention/compliance, audit/integrity, performance, cluster, egress) + the typed pattern-detection jobs | | `relata-intelligence` | Incident clustering, anomaly detection, LLM interpretation of alert clusters, detection-rule tuning suggestions | The MCP surface (`job_status`, `list_jobs`, `schedule_job`, `list_workflows`, `run_workflow`, `workflow_status`, `list_rules`, `create_rule`, `import_sigma`) mirrors these endpoints for tool-calling agents. ## RIFN — Relata Intelligence Feed Network RIFN is Relata's signed, bi-temporal **intelligence dissemination network**: publishers push knowledge fragments to a Feed Broker over HTTPS + mTLS; the broker authenticates, re-signs the envelope, and exposes them as append-only, cursor-addressed channels; subscribers (a local Relata deployment) pull diffs and apply them to their ontology-graph via SmartIngest + `EVIDENCE_INTAKE`. Every feed entry is a typed, signed `FeedEntry` with 14 payload kinds covering the full ontological surface (object/link assertions, events, ontology extensions, enrichment rules, job definitions, governance policies, reference data, identity bindings, revocations…). - The implementation lives in the `relata-feed` + `relata-feed-broker` crates. - Subscribers ingest via a typed `FeedSyncJob` on a schedule — not a daemon — applying signed `FeedEntry` values through SmartIngest. Content-hash-addressed entries deduplicate across publishers automatically; `RevokeEntry` handles retractions first-class. - The **broker is the licensed component**; the engine and subscriber-side `FeedSyncJob` are open core. A deployment may self-host a private broker for internal agency dissemination. - Progressive bootstrap: a new subscriber consumes reference-data and ontology layers first (layers 0–3), then live intelligence (layers 4–7); subscribers track a high-water mark per channel. ## Domain packs Packs bundle a domain's ontology types, detectors, jobs, reports, and detection rules into a signed, versioned unit. The repo ships ~20 domain packs + ~25 jurisdiction packs; the portal's [use-cases](/docs/use-cases/appdev-governed-rag) map onto them: | Use-case | Pack | Detection content | |---|---|---| | Financial intelligence / AML | [`finint`](/docs/use-cases/aml-sanctions-screening) | Sanctions/PEP screening, wire/crypto tracing, transaction-ring detection | | Cyber threat intel | [`cyber`](/docs/use-cases/cyber-sigma-detection) | C2 beacon detection, Sigma rules, IOC ingest | | Counter-terrorism | [`counter_terror`](/docs/use-cases/telecom-colocation-network) | Convoy detection, co-location / network analysis | | Law-enforcement telco | [`lea`](/docs/use-cases/lea-investigation-graph) | CDR analysis, ANPR trace, tower-dump | | Maritime | [`maritime`](/docs/use-cases/maritime-dark-fleet) | AIS, dark-fleet detection | | OSINT identity fusion | [`counter_intel`](/docs/use-cases/osint-identity-fusion) | Identity resolution, persona-cluster detection | | Counter-intelligence / narcotics / border / defense | `counter_intel` / `narcotics` / `border` / `defense` | Domain pattern sets | | Regional / sectoral | `gcc_mena`, `india`, `geopolitics`, `health`, `aml` + `jurisdiction-` (25) | Per-jurisdiction legal type sets | `relata-pack-stub` demonstrates the pack layout for authoring new packs. ## Jobs Background jobs (storage hygiene, correctness, retention/compliance, audit/integrity, performance, cluster, egress) run maintenance tasks: ```bash relata jobs # list all jobs relata jobs status indexer # check a specific job ``` | Endpoint | Method | Description | |---|---|---| | `/jobs` | `GET` | List jobs | | `/jobs/:name` | `GET` | Job status | ## See also - [Security](/docs/guides/security) — governance and policy enforcement - [Connectors & Extensions](/docs/guides/connectors) — the extension framework + Connector trait - [Observability](/docs/guides/observability) — monitoring detection alerts - [Ingestion](/docs/guides/ingestion) — ingest pipelines that feed detection ============================================================================== # LLM & Embedding Configuration URL: https://relatadb.dev/docs/guides/llm-embedding ============================================================================== # LLM & Embedding Configuration Relata has **two separate model-touching surfaces**, and they are configured independently. This page is the single canonical reference for both — consolidating settings that were previously scattered across [Configuration](/docs/guides/configuration), [Environment Variables](/docs/reference/env-vars), [Search](/docs/reference/search), and [Agent Memory](/docs/reference/agent-memory). | Surface | What it does | Where it runs | Config | |---|---|---|---| | **LLM endpoint** (`RELATA_LLM_URL`) | Natural-language → SQL translation, NL summaries (`nl_query`, `interpret` MCP tools) | External HTTP LLM server (Ollama / vLLM / LM Studio) | `RELATA_LLM_URL`, `RELATA_LLM_MODEL` | | **Embedder sidecar** (`RELATA_ACCEL_ENDPOINT`) | Compute `_emb_*` vectors for semantic/vector search | External HTTP embedder (sidecar process) | `RELATA_ACCEL_ENDPOINT`, `RELATA_EMBED_*` | > **Crucial invariant (since v1.1): the ingest hot path never calls a model.** Ingest is a pure validate → WAL → store loop bounded by disk I/O. LLM/embedder calls are either **caller-supplied** (vectors come in the row payload) or **asynchronous** (the media-worker drain populates `_emb_*` off the request thread). The built-in CPU embedder is **query-side only**. ## LLM endpoint (natural-language query) Point Relata at any OpenAI-compatible `/v1/chat/completions` endpoint to enable the `nl_query` and `interpret` MCP tools (natural-language → governed SQL → execute, with a deterministic fallback when unset). ```bash # Ollama (local) export RELATA_LLM_URL=http://localhost:11434/v1/chat/completions export RELATA_LLM_MODEL=llama3.1 # vLLM (self-hosted GPU server) export RELATA_LLM_URL=http://vllm-host:8000/v1/chat/completions export RELATA_LLM_MODEL=meta-llama/Meta-Llama-3.1-8B-Instruct # LM Studio (local) export RELATA_LLM_URL=http://localhost:1234/v1/chat/completions ``` | Variable | Default | Description | |---|---|---| | `RELATA_LLM_URL` | _(unset)_ | OpenAI-compatible chat-completions endpoint. Unset = `nl_query` uses a deterministic fallback (no LLM call). | | `RELATA_LLM_MODEL` | _(unset)_ | Model name passed in the request body. | Every NL-translated query still runs through the governed path (ACL, PURPOSE, cell masking, tenant scoping). The deterministic fallback when `RELATA_LLM_URL` is unset keeps the tool working offline. ## Embedder sidecar (vector embeddings) Vector search on text, image, audio, and video needs `_emb_*` fields. Since v1.1 you have **two ways** to populate them: 1. **Pre-computed (recommended for throughput)** — include `_emb_text` / `_emb_image` / `_emb_audio` / `_emb_video` (float arrays) directly in the row payload at ingest time. No model call is made. 2. **Sidecar + media-worker drain** — set `RELATA_ACCEL_ENDPOINT` to an external embedder; the media-worker drain cycle populates `_emb_*` asynchronously *after* the write returns. This is the only path that calls the embedder automatically, and it runs off the request thread. The built-in CPU embedder (128-dim, deterministic) is **query-side only** since v1.1: `recall()` uses it to embed the search query when no sidecar is configured. It is not invoked on ingest. ### Quick start ```bash # Point the server at a sidecar export RELATA_ACCEL_ENDPOINT=http://localhost:8200 cargo run -p relata-cli -- serve ``` The server probes the sidecar on startup and logs the model tag. ### Sidecar API contract The sidecar must implement HTTP endpoints returning JSON. The server waits up to **30 seconds** per batch call. Empty `"embeddings": []` is the correct response for a modality the sidecar doesn't support (the row is stored without that vector rather than failing). | Endpoint | Input | Output | Populates | |---|---|---|---| | `POST /embed` | `{"texts": ["...", "..."]}` | `{"embeddings": [[...]], "model": "..."}` | `_emb_text` | | `POST /embed-image` | `{"items": [[255,216,...], ...]}` (raw byte arrays, **not** base64) | `{"embeddings": [[...]]}` | `_emb_image` | | `POST /embed-audio` | `{"items": [[[bytes]]]}` | `{"embeddings": [[...]]}` | `_emb_audio` | | `POST /embed-video` | `{"items": [[[bytes]]]}` | `{"embeddings": [[...]]}` | `_emb_video` | | `POST /embed-face` | `{"items": [...]}` | `{"embeddings": [[...]]}` | `_emb_face` (gated on legal approval) | | `POST /rerank` *(optional)* | `{"query": "...", "documents": ["..."]}` | `{"scores": [0.95, ...]}` | Used by `HYBRID_SEARCH` + `recall`; falls back to RRF on 404 | ### MODALITY → endpoint mapping `SIMILAR TO … MODALITY ` and `similar_multimodal(modality="…")` route to the corresponding `_emb_*` field: | `MODALITY` | Row field | Sidecar endpoint | |---|---|---| | `text` | `_emb_text` | `POST /embed` | | `image` | `_emb_image` | `POST /embed-image` | | `audio` | `_emb_audio` | `POST /embed-audio` | | `video` | `_emb_video` | `POST /embed-video` | | `face` | `_emb_face` | `POST /embed-face` (legal gate) | ### Environment variables | Variable | Default | Description | |---|---|---| | `RELATA_ACCEL_ENDPOINT` | _(unset)_ | Base URL of the embedder sidecar. Unset = no sidecar; the built-in CPU embedder (128-dim) is used **query-side only** by `recall()`. Ingest does not embed — rows must carry `_emb_*` from the caller or the sidecar must be configured so the media-worker drain populates them. | | `RELATA_EMBED_BATCH_SIZE` | `32` | Texts per `/embed` call during the drain cycle. | | `RELATA_EMBED_CONCURRENCY` | `4` | Parallel drain workers sharing the embed queue. | ### Model tag and re-indexing The server derives a **model tag** from the first successful `/embed` response (include a `"model"` field to set it explicitly; otherwise `"default"`). The tag namespaces the HNSW index — **changing the model after data has been ingested requires a re-index pass or a fresh store**. ### Error handling - **Non-2xx** → server logs a warning and continues without that embedding; the row is stored text-only (still BM25-searchable). - **Connection refused / timeout** → circuit breaker opens after 3 consecutive failures; server logs `"embedder circuit open"` and stops calling the sidecar for 60 s. ### Reference sidecar (Python) A minimal sidecar using `sentence-transformers` for text and `open_clip` for images: ```python # sidecar.py — pip install flask sentence-transformers open_clip_torch Pillow from io import BytesIO import torch from flask import Flask, request, jsonify from sentence_transformers import SentenceTransformer import open_clip app = Flask(__name__) text_model = SentenceTransformer("all-MiniLM-L6-v2") # 384-dim text clip_model, _, clip_preprocess = open_clip.create_model_and_transforms( "ViT-B-32", pretrained="openai") clip_model.eval() @app.post("/embed") def embed(): texts = request.json["texts"] vecs = text_model.encode(texts, normalize_embeddings=True).tolist() return jsonify({"embeddings": vecs, "model": "all-MiniLM-L6-v2"}) @app.post("/embed-image") def embed_image(): from PIL import Image items = request.json["items"] embeddings = [] for byte_ints in items: img = Image.open(BytesIO(bytes(byte_ints))).convert("RGB") tensor = clip_preprocess(img).unsqueeze(0) with torch.no_grad(): vec = clip_model.encode_image(tensor) vec = vec / vec.norm(dim=-1, keepdim=True) embeddings.append(vec[0].tolist()) return jsonify({"embeddings": embeddings, "model": "ViT-B-32:openai"}) # Return empty lists for modalities you don't support: @app.post("/embed-audio") def embed_audio(): return jsonify({"embeddings": []}) @app.post("/embed-video") def embed_video(): return jsonify({"embeddings": []}) if __name__ == "__main__": app.run(host="0.0.0.0", port=8200) ``` ```dockerfile FROM python:3.12-slim RUN pip install --no-cache-dir flask sentence-transformers open_clip_torch Pillow COPY sidecar.py /app/sidecar.py CMD ["python", "/app/sidecar.py"] ``` ### Verify end-to-end **Option A — caller-supplied (no sidecar):** send the vector in the row payload; `_emb_text` is honoured verbatim. ```bash curl -s -X POST http://localhost:9090/ingest \ -H "Content-Type: application/json" \ -d '[{"_type":"Note","body":"The HNSW index is seeded","_emb_text":[0.12,0.34,0.56,0.78]}]' ``` **Option B — sidecar + media-worker drain:** start the sidecar, set `RELATA_ACCEL_ENDPOINT`, ingest a media row; the worker embeds it asynchronously (typically sub-second). ```bash docker run -p 8200:8200 relata-sidecar & export RELATA_ACCEL_ENDPOINT=http://localhost:8200 cargo run -p relata-cli -- serve & curl -s -X POST http://localhost:9090/ingest/media?modality=image \ -H "Content-Type: application/json" \ -d '{"_type":"Photo","image_b64":""}' sleep 2 # wait for the drain cycle curl -s http://localhost:9090/query \ -H "Content-Type: application/json" \ -d '{"query":"SELECT _emb_image FROM Photo LIMIT 1"}' ``` If `_emb_image` is non-null after the drain window, the sidecar is working. If it stays `null`, check the server log for `"embedder circuit open"` or `"embed timeout"`. ## Embedding model migration Changing the sidecar model after data has been ingested changes the model tag and the vector dimension, which breaks similarity search over existing rows. The migration story: re-index (`relata embed --type=`) or start a fresh store. See the source repo's embedding-model-migration guide for the full procedure. ## See also - [Search & Retrieval](/docs/reference/search) — `HYBRID_SEARCH`, `SIMILAR TO`, vector operators - [Vector Index Parameters](/docs/reference/vector-params) — HNSW / DiskANN tuning - [Agent Memory](/docs/concepts/agent-memory) — `recall` runs on the same BM25 + vector pipeline - [Configuration](/docs/guides/configuration) — every other runtime knob - [Environment Variables](/docs/reference/env-vars) — the full env-var reference ============================================================================== # Multi-Tenancy URL: https://relatadb.dev/docs/guides/multi-tenancy ============================================================================== # Multi-Tenancy RelataDB supports multiple organisations on a single binary. Isolation is enforced on the read path via planner guards plus `tenant_id` keying in the storage layer. One tenant cannot see another's rows, even on a shared node. ## How isolation works Every scan carries the request's tenant context into storage. The planner guard rejects any query plan that would cross tenant boundaries. Rows physically co-exist in one store but are logically unreachable across tenants. There is no default tenant. A request to `server` or `cluster` without `X-Organization-Id` is rejected. ## Set the tenant header on every request ```bash curl https://relata.example.com/query \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \ -H "X-Organization-Id: org-acme" \ -H "Content-Type: application/json" \ -d '{"sql":"SELECT name FROM Person LIMIT 5"}' ``` The SDK injects the header automatically when you set `tenant`: ```python # Python SDK from relata import RelataClient client = RelataClient( url="http://localhost:9090", bearer_token=token, tenant="org-acme", purpose="analytics", ) rows = client.query("SELECT name FROM Person LIMIT 5") ``` ## Delegation headers Use delegation when one principal acts on behalf of another. Every delegation is recorded in the audit log and validated against the delegating principal's authority. | Header | Description | |---|---| | `X-Organization-Id` | The tenant the request runs as. Required on server/cluster. | | `X-Acting-As` | User identity the request runs as (delegated). | | `X-Delegated-By` | The principal who granted the delegation. | ```bash # User alice delegates to bob for an analytics query curl https://relata.example.com/query \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \ -H "X-Organization-Id: org-acme" \ -H "X-Acting-As: user:bob" \ -H "X-Delegated-By: user:alice" \ -H "Content-Type: application/json" \ -d '{"sql":"PURPOSE '\''analytics'\'' SELECT name FROM Person LIMIT 5"}' ``` A delegation that exceeds the delegating principal's authority is rejected by the policy engine. The audit entry records both principals. ## Create and manage tenants Tenant management uses the `/tenants` REST API. Start the server with `RELATA_TENANCY_MODE=multi` and an admin bearer token. ### CLI The `relata tenant` subcommand wraps the full lifecycle: ```bash # Create relata tenant create --id org-acme --name "Acme Corp" --tier standard # List / get / update relata tenant list relata tenant get org-acme relata tenant update org-acme --name "Acme Global" # Quota relata tenant quota org-acme --max-mb 10240 # Usage relata tenant usage org-acme # Members relata tenant members org-acme --add user-1 --role analyst # Sharing agreements relata tenant sharing org-acme --add partner-org # Suspend / reactivate / delete relata tenant suspend org-acme relata tenant reactivate org-acme relata tenant delete org-acme ``` ### HTTP API Three API surfaces serve different audiences: | Surface | Audience | Key difference | |---|---|---| | `/tenants` | Tenant admin | Full CRUD + members + sharing + quota + search config | | `/api/v1/tenants` | Control-plane automation | Hard-purge delete, billing-grade usage, inline quota at create | | `/platform/tenants` | Platform operator | Tier assignment, license status, cross-tenant usage | ```bash # Create curl -X POST http://localhost:9090/tenants \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \ -H "Content-Type: application/json" \ -d '{"id":"org-acme","name":"Acme Corp","tier":"standard"}' # Set quota curl -X PUT http://localhost:9090/tenants/org-acme/quota \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \ -H "Content-Type: application/json" \ -d '{"max_mb":10240}' # Check usage curl http://localhost:9090/tenants/org-acme/usage \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" # Create a sharing agreement (org-acme shares with org-partner) curl -X POST http://localhost:9090/tenants/org-acme/sharing \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \ -H "Content-Type: application/json" \ -d '{"partner_org":"org-partner"}' # Retire a tenant (governed tombstone) curl -X DELETE http://localhost:9090/tenants/org-acme \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" ``` For control-plane automation (billing, hard purge): ```bash # Provision with inline quota curl -X POST http://localhost:9090/api/v1/tenants \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \ -H "Content-Type: application/json" \ -d '{"tenant_id":"org-acme","display_name":"Acme Corp","tier":"server","quota_mb":10240}' # Hard purge (removes all tenant rows — irreversible) curl -X DELETE http://localhost:9090/api/v1/tenants/org-acme \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" ``` See the [HTTP API reference](/docs/reference/api-reference) for the complete endpoint table covering all three API surfaces. ## Per-tenant encryption Each tenant has a distinct Data Root Key (DRK). KMS isolation means one tenant's DEK cannot unwrap another's data. The node fails closed if KMS is unavailable — no tenant data is readable in plaintext at rest. DRK rotation produces a new wrapped-DEK set without re-encrypting data rows. To rotate: ```bash curl -X POST http://localhost:9090/tenants/org-acme/rotate-key \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" ``` ## Per-tenant quotas Quotas protect against noisy-neighbour workloads. Cost is a function of rows scanned, not rows returned — a `SELECT *` against a 10 M-row table costs more budget than a `LIMIT 10`. | Variable | Default | Description | |---|---|---| | `RELATA_QUERY_QUOTA` | `10000` | Cost units per principal per window. | Set per-tenant overrides via the admin API (see above). Quota exhaustion returns `429 Too Many Requests` with a `Retry-After` hint. Monitor quota usage across all tenants: ```bash curl http://localhost:9090/tenants/usage \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" ``` ## Multi-org principals A principal may belong to multiple organisations. The `X-Organization-Id` header selects which org's data is in scope for each request. The acting org is recorded in the audit log alongside the principal identity. Verify that cross-tenant isolation holds: ```bash # Ingest into org-acme curl -X POST http://localhost:9090/ingest \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \ -H "X-Organization-Id: org-acme" \ -H "Content-Type: application/json" \ -d '{"object_type":"Person","data":[{"name":"Ada"}]}' # Query from org-other — should return zero rows curl -X POST http://localhost:9090/query \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \ -H "X-Organization-Id: org-other" \ -H "Content-Type: application/json" \ -d '{"sql":"SELECT * FROM Person"}' ``` Zero rows from `org-other` confirms isolation is working. ## Sub-tenant namespaces `NamespacePath` on `Row` partitions data within a tenant — for example, separating departments or cases inside one organisation. ```sql SELECT * FROM Person WHERE namespace = 'org-acme/dept-finance' LIMIT 5; ``` Sub-tenant namespace enforcement is partially wired. Full per-namespace policy enforcement is deferred. Do not rely on namespace filtering as a security boundary today; use tenant-level isolation for hard boundaries. ## See also - [Auth & Security](/docs/guides/security) — ABAC policy engine, egress filtering - [Configuration](/docs/guides/configuration) — quota and auth env vars - [Observability](/docs/guides/observability) — audit trail per tenant ============================================================================== # Multimedia search — image, video, audio, face URL: https://relatadb.dev/docs/guides/multimedia-search ============================================================================== # Multimedia search — image, video, audio, face RelataDB isn't just a text-and-rows engine — it ships six embedding modalities and two purpose-built multimedia SQL operators: `FACE_SEARCH` for face recognition over a gallery, and `MATCH_PDQ` for perceptual-hash near-duplicate detection (the CSAM / NCMEC / known-image workflow). Media blobs are governed (`BlobRef` + content-addressed store), ACL applies on every hit, and you can write media via the S3 door and search it over SQL. > All six embed routes and both operators are verified against source — `FACE_SEARCH` and `MATCH_PDQ` are SQL-reachable (`crates/relata-query/src/scenario_e2e.rs`); the embed routes live at `/embed/{,batch,image,face,audio,video}`. Media embed routes return `503` if the active embedder doesn't support the modality (the CPU default doesn't — wire the GPU sidecar via `RELATA_ACCEL_ENDPOINT`). ## The 6 embedding modalities | Modality | Route | SDK method (py / ts / go) | Model family | |---|---|---|---| | Text | `POST /embed` + `/embed/batch` | `embed` / `embed_batch` | CPU lexical default (128-dim); GPU sidecar via `RELATA_ACCEL_ENDPOINT` | | Image | `POST /embed/image` | `embed_image` / `embedImage` / `EmbedImage` | CLIP | | Face crop | `POST /embed/face` | `embed_face` / `embedFace` / `EmbedFace` | ArcFace | | Audio | `POST /embed/audio` | `embed_audio` / `embedAudio` / `EmbedAudio` | CLAP | | Video keyframe | `POST /embed/video` | `embed_video` / `embedVideo` / `EmbedVideo` | CLIP keyframe | | Near-dup hash | (computed on ingest into `_emb_*` slots) | — | aHash + dHash (pure-Rust, in-tree) | ```python vc = client.vector_client e = vc.embed("a red car at sunset") # → {embedding, model, dim} eim = vc.embed_image(base64_bytes) # CLIP — multimodal RAG ef = vc.embed_face(base64_bytes) # ArcFace — face gallery eau = vc.embed_audio(base64_bytes) # CLAP — audio retrieval ev = vc.embed_video(base64_bytes) # CLIP keyframe — video frame search ``` Embeddings land on `_emb_*` columns, keyed on `(object_type, modality, model_tag, tenant_id)`, and ride the same HNSW + DiskANN index as text vectors. See [Vector params](/docs/reference/vector-params) for `ef_construction`/`M`/quantization tuning. ## FACE_SEARCH — face recognition as a SQL operator Index a face crop into a gallery, then match a probe face against it in one SQL call. Returns the top-k identities above a threshold, governed by ACL. ```sql PURPOSE 'investigation' -- Match a probe face embedding against the 'watchlist' gallery, top 5 SELECT * FROM FACE_SEARCH('[0.12, 0.087, ...]', 'watchlist', LIMIT => 5); -- With an explicit similarity threshold SELECT identity_id, score FROM FACE_SEARCH('[...]', 'casework', THRESHOLD => 0.85) ORDER BY score DESC; ``` ```python # Python — equivalent via the SDK hits = client.face_search("watchlist", probe_embedding, k=5, threshold=0.85, purpose="investigation") # → QueryResult ``` The MCP `face_match` tool wraps the same operator for agent-driven investigation: ```python mcp.call_tool("face_match", {"probe_id": "probe-7", "threshold": 0.85, "top_k": 10, "purpose": "security_incident"}) ``` > **Governance note:** biometric data is high-sensitivity. Put face galleries behind a strict Cedar policy (a `forbid` on `Face.embedding` for non-clearance principals), declare a dedicated `PURPOSE`, and let the audit chain record every probe. See [Per-Door ACL](/docs/guides/per-door-acl) and [Governance](/docs/concepts/governance). ## MATCH_PDQ — perceptual-hash near-duplicate detection PDQ is Facebook's perceptual hash for images — the standard for CSAM / NCMEC / known-image matching. Relata computes PDQ hashes on ingest into a slot and exposes `MATCH_PDQ` as a governed SQL operator: hash a probe image, match against a corpus, return near-duplicates above a threshold. ```sql PURPOSE 'trust-and-safety' -- Match a probe PDQ hash against the 'ncmec' corpus SELECT object_id, similarity FROM MATCH_PDQ('ffff...', 'ncmec', THRESHOLD => 0.5) ORDER BY similarity DESC; -- Tunable threshold — higher = stricter (fewer false positives) SELECT * FROM MATCH_PDQ('', 'intake', THRESHOLD => 0.9); ``` ```python hits = client.match_pdq("ncmec", probe_hash, threshold=0.9, purpose="trust-and-safety") # → QueryResult ``` Use this for trust-and-safety pipelines (detect known-bad images across uploads), copyright / dedup, and CSAM scanning where the hash corpus is loaded via `relata import` or a sanctions-style pull. ## Image near-dup (aHash + dHash) — lightweight, no sidecar For "is this image a re-encoding of one we've seen?" without a GPU — Relata computes aHash + dHash in-tree (pure Rust, decodes the image itself) and finds near-copies within a small Hamming distance. Faster and cheaper than PDQ for the common case; PDQ is the higher-precision choice for adversarial inputs. This is wired into ingest for image-bearing types — a re-encoded crop of an existing image surfaces as a near-dup automatically. ## Governed media blobs (`BlobRef`) Media content doesn't bloat the row store. `MediaContent` rows carry a `BlobRef` pointer to the content-addressed blob store + the `_emb_*` embedding, never inline bytes: ```bash # Async media ingest — enqueue the blob + an embed task curl -X POST http://127.0.0.1:9090/ingest/media \ -H 'Authorization: Bearer ' -H 'Content-Type: application/json' \ -d '{ "object_type": "SurveillanceFrame", "mime_type": "image/jpeg", "body_b64": "", "partition_key": "case-7", "purpose": "investigation" }' # → {task_id: "itsk_..."} — poll GET /ingest/media/:task_id for completion ``` Bodies ≥ `RELATA_S3_BLOB_THRESHOLD_MB` spill to the content-addressed store; smaller ones inline. Cross-protocol: write media via the **S3 door** (`put_object`), query it over **SQL** (`SELECT key, size FROM S3Object`), embed it via `/embed/image`. See [S3 door](/docs/guides/s3-door). ## Multimodal RAG — text → image retrieval Because CLIP image embeddings live in the same vector index as text, a text query retrieves matching images: ```python # Embed a text probe, retrieve matching images e = vc.embed("a red sedan parked at night") hits = vc.knn_search("SurveillanceFrame", "_emb_image", e["embedding"], k=10, purpose="investigation") for h in hits: print(h["object_id"], h["case_id"]) ``` The same flow drives the MCP `search_video_frames` tool — text-query a corpus of video keyframes: ```python mcp.call_tool("search_video_frames", {"query_id": "", "media_type": "VideoFrame", "top_k": 20, "purpose": "security_incident"}) ``` ## Tips & takeaways - **Wire the GPU sidecar for media.** The CPU embedder handles text only — `embed_image/face/audio/video` return `503` until you set `RELATA_ACCEL_ENDPOINT`. Text-only workloads stay on CPU. - **Pick the right hash for the job.** aHash/dHash = fast, cheap, good for re-encoding detection. PDQ = higher precision, adversarial inputs, CSAM/NCMEC. Face identification = ArcFace via `FACE_SEARCH`, not a hash. - **Threshold tuning is per-corpus.** A 0.85 face threshold that works on a clean watchlist may need 0.90 on a noisy intake feed. Backtest against labelled pairs before going live. - **Partition by case / tenant.** Use `partition_key` on `/ingest/media` so a probe search is scoped to one case — much faster and tenant-safe. - **Biometrics deserve biometric-grade governance.** `forbid` clauses on face embeddings, dedicated `PURPOSE`, audit-chain on every probe. Treat the gallery like the sensitive PII it is. - **Cross-door is the payoff.** Your SOC uploads evidence via the S3 door; the investigator searches it via SQL `FACE_SEARCH` + `MATCH_PDQ`; the agent retrieves via MCP `search_video_frames`. One store, four front doors. ## See also - [Vector parameters](/docs/reference/vector-params) — HNSW/DiskANN + quantization tuning - [Hybrid Search](/docs/concepts/hybrid-search) — fuse BM25 + vector + graph - [S3 door](/docs/guides/s3-door) — governed object store + bi-temporal versioning - [Per-Door ACL](/docs/guides/per-door-acl) — Cedar principals for sensitive media doors - [For AI Agent Builders](/docs/use-cases/for-ai-agents) — multimodal RAG in context - [For Security Teams](/docs/use-cases/for-security-teams) — trust-and-safety framing ============================================================================== # Observability URL: https://relatadb.dev/docs/guides/observability ============================================================================== # Observability RelataDB emits structured logs, Prometheus metrics, OpenTelemetry traces, and health/readiness probes from a single binary with no external dependencies. Every layer is opt-in except logging, which is always on. ## Structured logging Set log format to `json` in production so log shippers can parse fields directly: ```bash RELATA_LOG_FORMAT=json RELATA_LOG_LEVEL=info relata serve ``` | Variable | Default | Options | |---|---|---| | `RELATA_LOG_FORMAT` | `pretty` | `pretty` (human-readable) \| `json` (production) | | `RELATA_LOG_LEVEL` | `info` | `trace` \| `debug` \| `info` \| `warn` \| `error` | A JSON log line carries `ts`, `level`, `target`, `msg`, and request-scoped fields (`request_id`, `tenant`, `principal`) when a request is in context. Feed into any log shipper (Loki, Elasticsearch, CloudWatch) without a parsing plugin. ## Health and readiness probes Wire these into your load balancer and container orchestrator: ```bash # Liveness — always 200 if the process is running curl http://localhost:9090/health # Readiness — 503 (problem+json) if any of 12 conditions fail curl http://localhost:9090/health/ready # Profile, role, query quota curl http://localhost:9090/status # Build info: version, git SHA, build time, profile curl http://localhost:9090/version ``` The readiness response (200): ```json { "status": "ready", "profile": "server", "node_id": "node-1", "queue_depth_pct": 0, "replication_lag": 0, "uptime_secs": 642, "node_count": 1, "license_tier": "server" } ``` When a condition fails, the endpoint returns `503` with an RFC 7807 `application/problem+json` body whose `type` names the failing check. Common failure reasons: | Reason (`type`) | What it means | What to do | |---|---|---| | `embedder-unhealthy` | Embedding circuit open (consecutive errors) | Check sidecar logs; circuit resets after `RELATA_EMBED_CIRCUIT_COOLDOWN_MS` | | `wal-unavailable` | WAL write failures above threshold | Check disk space and `RELATA_DATA_DIR` permissions | | `audit-backpressure` | Audit log dropped entries | Compliance event — isolate node, preserve WAL, investigate | | `queue-backpressure` | Ingest queue at capacity | Scale ingest or raise queue capacity | Route traffic only to nodes returning `200` from `/health/ready`. ## Prometheus metrics `/metrics` serves operational counters in Prometheus text format. Key metrics: | Metric | Type | Description | |---|---|---| | `relata_ingested_rows_total` | counter | Cumulative ingested rows | | `relata_query_count_total` | counter | Cumulative queries | | `relata_uptime_seconds` | gauge | Server uptime | | `relata_audit_chain_valid` | gauge | `1` = valid, `0` = tampered | | `relata_embed_queue_depth` | gauge | Current embedding backlog | | `relata_embed_queue_capacity` | gauge | Max backlog (`RELATA_EMBED_QUEUE_MAX`) | | `relata_wal_failures_total` | counter | WAL write failures | | `relata_background_task_panics_total` | counter | Background task panics | By default `/metrics` requires a bearer token (fail-closed). For Prometheus scrapers that authenticate at the network layer (NetworkPolicy, mTLS sidecar): ```bash RELATA_METRICS_PUBLIC=true relata serve ``` Prometheus scrape config: ```yaml scrape_configs: - job_name: relatadb static_configs: - targets: ["relata:9090"] # Remove bearer_token if RELATA_METRICS_PUBLIC=true bearer_token: "" metrics_path: /metrics scrape_interval: 15s ``` Alert on `relata_audit_chain_valid == 0` — that is a security event requiring immediate investigation. ## OpenTelemetry traces Set `RELATA_OTLP_ENDPOINT` to export spans. When the variable is unset, OpenTelemetry is fully disabled — zero overhead, no threads, no allocations. ```bash RELATA_OTLP_ENDPOINT=http://otel-collector:4318/v1/traces \ RELATA_OTLP_SAMPLE_RATIO=0.1 \ relata serve ``` | Variable | Default | Description | |---|---|---| | `RELATA_OTLP_ENDPOINT` | — | OTLP/HTTP endpoint. Unset = OTel fully off. | | `RELATA_OTLP_SAMPLE_RATIO` | `0.01` | Fraction of traces sampled. `1.0` = sample everything. | Use `1.0` temporarily when debugging a specific request flow; drop back to `0.01` for steady-state production to avoid overhead. ## docker-compose with OpenTelemetry Collector A working local observability stack: ```yaml services: relata: image: ghcr.io/relatadb/relata:latest environment: RELATA_PROFILE: server RELATA_BEARER_TOKEN: "change-me" RELATA_LOG_FORMAT: json RELATA_LOG_LEVEL: info RELATA_OTLP_ENDPOINT: "http://otel-collector:4318/v1/traces" RELATA_OTLP_SAMPLE_RATIO: "0.1" RELATA_METRICS_PUBLIC: "true" ports: - "9090:9090" depends_on: - otel-collector otel-collector: image: otel/opentelemetry-collector-contrib:latest volumes: - ./otel-collector.yaml:/etc/otel-collector.yaml command: ["--config=/etc/otel-collector.yaml"] ports: - "4318:4318" # OTLP/HTTP receiver - "8888:8888" # Collector metrics prometheus: image: prom/prometheus:latest volumes: - ./prometheus.yaml:/etc/prometheus/prometheus.yml ports: - "9091:9090" ``` Minimal `otel-collector.yaml`: ```yaml receivers: otlp: protocols: http: endpoint: "0.0.0.0:4318" exporters: logging: verbosity: detailed service: pipelines: traces: receivers: [otlp] exporters: [logging] ``` Minimal `prometheus.yaml`: ```yaml scrape_configs: - job_name: relatadb static_configs: - targets: ["relata:9090"] metrics_path: /metrics scrape_interval: 15s ``` ## Correlation IDs Every request gets an auto-generated `X-Request-ID` (UUID v7, `serve.rs:13248`). Pin your own to trace a specific request end-to-end: ```bash curl -H "X-Request-ID: $(uuidgen)" \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \ http://localhost:9090/query \ -d '{"sql":"SELECT * FROM Person LIMIT 1"}' ``` The server stamps the ID on error responses. RFC 7807 `application/problem+json` bodies carry `request_id` so a user-visible error can be traced through logs, spans, and the audit chain without a session replay. ## Audit chain verification ```bash # Quick check curl -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \ http://localhost:9090/audit/count # { "entries": 1248, "chain_valid": true } # Deep check: chain + WAL + object-store config relata check ``` `chain_valid: false` means an audit entry was modified or deleted. Treat it as a security event: isolate the node, preserve the WAL directory, and open an incident. The `relata_audit_chain_valid` Prometheus metric exports this as a gauge — alert on `== 0`. ## Profiling (CPU / heap) Production profiling is **off by default** and admin-gated — a CPU profile leaks workload shape and resolved symbol names, so it is never silently on. ```bash RELATA_PPROF_ENABLE=true RELATA_ADMIN_TOKEN=changeme relata serve # capture 15s of CPU as a pprof protobuf curl -H "Authorization: Bearer changeme" \ "http://localhost:9090/debug/pprof/profile?seconds=15" -o cpu.pprof ``` `/debug/pprof/profile` returns Google's standard **pprof protobuf** (`application/octet-stream`, `cpu.pprof`) — not a server-rendered SVG. Render it with the standard toolchain: ```bash go tool pprof -http=:8080 cpu.pprof # interactive flamegraph / top / source view # or drop cpu.pprof into https://speedscope.app for a no-install browser view ``` `GET /debug/pprof/heap` reports the coarse memory counters the store already tracks (`implemented: false` — a real per-allocation-site heap profile needs a jemalloc build with `--enable-prof`; the default allocator is mTLS mimalloc). Bounds: `?seconds=` is clamped to `[1, 30]`; only one profile may run per process at a time (a second gets `429`). --- ## Operational debug endpoints | Endpoint | Description | |---|---| | `GET /debug/stats` | Engine counts: records, states, snapshot rows, log leaves, tokens | | `GET /metrics.json` | Same data as `/metrics` in JSON (useful for scripted checks) | | `GET /audit/count` | Audit entries count + chain validity | ```bash # Get engine stats without parsing Prometheus format curl -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \ http://localhost:9090/metrics.json | jq . ``` ## MCP observability tools The MCP surface exposes operational tools for agent runbooks: | Tool | Mirrors | |---|---| | `server_health` | `/health/ready` | | `metrics` | `/metrics.json` | | `job_status` | Continuous detection jobs | | `get_audit_trail` | Paginated, filtered audit trail | ## See also - [Configuration](/docs/guides/configuration) — logging and OTLP env vars - [Auth & Security](/docs/guides/security) — audit hash chain, security events - [Backup & Restore](/docs/guides/backup-restore) — WAL durability model ============================================================================== # Per-door ACL — least privilege per integration URL: https://relatadb.dev/docs/guides/per-door-acl ============================================================================== # Per-door ACL — least privilege per integration When your MongoDB ETL scraper, your Postgres BI tool, your S3 archive job, and your HTTP app all connect to one database, **how do you grant each integration the least privilege it needs — and nothing more?** In a traditional database the answer is "you can't, really" — every connection from the same user shares one principal, and an attacker who compromises the read-only scraper can write if the shared user can. Relata solves this by giving **every wire door a distinct Cedar principal**. A request arriving over the S3 door is principal `s3-client`; over pgwire it's `pgwire-client`; over MongoDB it's `mongo-client`. One Cedar policy can say "the S3 door may read `Person` but not write it" — and that holds regardless of which credentials the S3 client used, because the principal is the *door*, not the user. > **Why this matters:** turn "this read-only S3 scraper can't write even if compromised" into a one-liner, and make every audit-log row forensically attributable to the protocol that wrote it. ## The 11 door principals Every request is classified into exactly one door principal before the ACL evaluates: | Door | Cedar principal | |---|---| | HTTP REST | `http-client` | | gRPC | `grpc-client` | | PostgreSQL wire (psql, psycopg2, pgvector) | `pgwire-client` | | MongoDB wire | `mongo-client` | | Redis RESP | `redis-client` | | Neo4j HTTP (Cypher) | `neo4j-client` | | Bolt | `bolt-client` | | ClickHouse HTTP / native | `clickhouse-client` | | S3-compatible | `s3-client` | | Arrow Flight | `flight-client` | | MCP (`/mcp`) | `mcp-client` | | Server-internal | `system` | The door principal is independent of the **user principal** (the bearer token's identity) and the **session principal** — Cedar sees all three, so a policy can express "the S3 door acting for user U in tenant T." ## Writing per-door policies The door principal is exposed in Cedar as `principal == User::"-client"`. Combine with the user, action, and resource as usual: ```cedar // The S3 door may read Person and S3Object but never write Person. permit( principal == User::"s3-client", action == Action::"read", resource in Resource::"Person" ); permit( principal == User::"s3-client", action == Action::"read", resource in Resource::"S3Object" ); // No permit clause for s3-client + Action::"write" + Person → denied by default. ``` ```cedar // pgwire gets full Person read/write — your BI tool's psql connection. permit( principal == User::"pgwire-client", action in [Action::"read", Action::"write"], resource in Resource::"Person" ); ``` ```cedar // The Mongo door is write-only (ingest only, no exfiltration). permit( principal == User::"mongo-client", action == Action::"write", resource in Resource::"MongoDocument" ); ``` Deny-wins: an explicit `forbid` overrides any `permit`, so a compliance lock-down is one line: ```cedar // No door may read the `ssn` column on Person, regardless of who asks. forbid( principal, action == Action::"read", resource == Resource::"Person.ssn" ); ``` ## Faster than Cedar — the env-var grant shorthand For common cases, skip the policy file entirely and use the `RELATA_ACL_GRANT` env var (now targets `ALL_DOOR_ROLES`): ```bash # Person: read for every door, write only for pgwire + http RELATA_ACL_GRANT="Person:read+write" RELATA_ACL_GRANT_PGWIRE="Person:write" \ relata serve ``` This compiles into Cedar at startup; you can mix and match with hand-written policy files. ## Audit attribution — the forensic payoff Every governed write/flip carries the door principal in the audit log as the `purpose` field — so a later investigation can answer "did this row come in over Mongo, S3, or HTTP?": ```bash # Every row the S3 door touched: curl 'http://127.0.0.1:9090/audit/entries?purpose=s3-client&limit=50' \ -H 'Authorization: Bearer ' ``` ```sql -- Which doors have written to Person in the last 24h? SELECT purpose AS door, COUNT(*) AS writes FROM _audit WHERE resource_type = 'Person' AND system_at > now() - INTERVAL '24' hours GROUP BY purpose; ``` This is tested in production paths (`crates/relata-cli/tests/serve_hardening.rs` — the `s3-client` door role is asserted to flow through `governed_get` into the audit log on every S3 door read). ## Tips & takeaways - **Default to least privilege per door.** A new integration gets a new door principal with `read` only; escalate to `write` only when the integration needs it. This is the single biggest containment win for the "compromised scraper" threat. - **The door principal is the *first* factor, not the only one.** Combine with user principal (who) and tenant (where) for full ABAC — `principal == User::"s3-client" && resource.tenant == "org-acme"`. - **Use `forbid` for compliance lock-downs.** Cell-level `forbid` on `Person.ssn` survives any future `permit` someone adds — deny-wins is your safety net. - **Audit by door to spot anomalies.** "Why is the Redis door writing to `Person`?" is a one-query check once every door's writes are tagged. - **Door principals + `RELATA_TENANCY_MODE=multi`.** In multi-tenant mode the door principal still applies *within* each tenant's scope — the per-tenant policy can override per-door defaults. ## See also - [Auth & Security](/docs/guides/security) — bearer tokens, OIDC, mTLS, the full ACL model - [Governance](/docs/concepts/governance) — Cedar-inspired ABAC, PURPOSE tracking, cell masking - [Compatibility & Doors](/docs/compatibility) — the 13 wire doors this principal set covers - [Deploying Protocol Doors](/docs/deployment/protocol-doors) — door bind/publish/enable wiring - [Audit & Provenance](/docs/concepts/provenance) — the audit chain that records every door's writes ============================================================================== # Privacy & GDPR URL: https://relatadb.dev/docs/guides/privacy ============================================================================== # Privacy & GDPR RelataDB provides first-class data subject rights management: data subject access requests (DSAR), erasure (cryptographic and physical), consent registers, and retention policies. ## Data Subject Access Request (DSAR) Export all data associated with a subject: ```bash # CLI relata dsar --subject user@example.com # HTTP curl "http://localhost:9090/gdpr/dsar?subject=user@example.com" \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" ``` The response includes every row across every type where the subject's identifier appears, plus the provenance chain for each row. ## Erasure ### Cryptographic erasure The data root key (DRK) for a tenant is destroyed in the KMS. All data encrypted under that key becomes permanently unreadable — no row-by-row deletion needed. This is the fastest path for a full-tenant right-to-be-forgotten. ```bash curl -X POST http://localhost:9090/gdpr/erase \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \ -H "Content-Type: application/json" \ -d '{"subject":"user@example.com","scope":"all"}' ``` ### Physical erasure For per-subject erasure, the system closes the subject's rows (`valid_to` = now) and removes them from all indexes (FTS, vector, graph, identity). Bi-temporal history is preserved for audit — the row's provenance chain survives, but the PII fields are cryptographically shredded. ### Media erasure Face/voice biometric templates are removed from the searchable index, and media blobs are deleted from S3. This is critical for biometric data under GDPR Article 17. ## Consent register Every query may carry a PURPOSE tag that is matched against a consent register: ```sql PURPOSE 'marketing' SELECT name, email FROM Person WHERE id = 'p1' ``` If the subject has not consented to `'marketing'`, the query returns zero rows for that subject. Consent is managed as governed rows and can be withdrawn at any time. ## Retention policies | Endpoint | Method | Description | |---|---|---| | `/retention/policies` | `GET` | List policies | | `/retention/holds` | `GET` | List legal holds | | `/retention/holds` | `POST` | Place a legal hold | | `/retention/holds/:case` | `DELETE` | Lift a legal hold | | `/retention/worm` | `GET` | List WORM policies | Legal holds prevent retention enforcement on held data, even if the retention period has expired. ## See also - [Governance](/docs/concepts/governance) — PURPOSE, consent, and policy - [Security](/docs/guides/security) — encryption and KMS - [Multi-tenancy](/docs/guides/multi-tenancy) — per-tenant encryption keys ============================================================================== # The S3 door — your existing S3 client, governed URL: https://relatadb.dev/docs/guides/s3-door ============================================================================== # The S3 door — your existing S3 client, governed Relata serves a real S3-compatible API. Your existing `boto3` / `aws` CLI / `rclone` / MinIO client works unchanged — point it at Relata's S3 port and use the bearer token as the access key (or set up SigV4). Every `PutObject` lands as a governed `S3Object` row with PROV-O provenance, bi-temporal history, and Cedar ACL — so you can write via S3 and read the same data back over SQL, with the audit chain recording every object event. > **Why this matters:** MinIO/S3 give you blob storage. Relata's S3 door gives you blob storage **plus** provenance, bi-temporal versioning, cell-level ACL, and cross-protocol reads — your SQL analytics, your graph traversal, and your hybrid search all see the same objects. ## Connect The door auto-enables when `RELATA_BEARER_TOKEN` is set (see [Compatibility & Doors](/docs/compatibility)). ```bash RELATA_BEARER_TOKEN=change-me relata serve # S3 door auto-starts on port 9191 ``` ```python import boto3 from botocore.config import Config s3 = boto3.client( "s3", endpoint_url="http://127.0.0.1:9191", aws_access_key_id="change-me", # = RELATA_BEARER_TOKEN aws_secret_access_key="unused", # SigV4 secret defaults to the bearer token config=Config(signature_version="s3v4", s3={"addressing_style": "path"}), ) ``` The SigV4 secret defaults to `RELATA_BEARER_TOKEN` unless you set `RELATA_S3_SECRET_KEY` to a dedicated secret. When a secret is configured, the door **requires verified SigV4** and rejects plaintext bearer auth (set `RELATA_S3_ALLOW_PLAINTEXT=true` only for dev — cleartext credentials are forgeable). ## Supported operations | Operation | Notes | |---|---| | `ListBuckets` | governed — only buckets the caller's ACL permits | | `CreateBucket` / `HeadBucket` / `DeleteBucket` | bucket must be empty to delete | | `ListObjectsV2` | with prefix + delimiter | | `PutObject` / `GetObject` / `DeleteObject` / `HeadObject` | bodies ≥ `RELATA_S3_BLOB_THRESHOLD_MB` (default 4 MiB) spill to content-addressed blob store | | `ListObjectVersions` | **bi-temporal versioning for free** — see below | | Multipart upload | parts in-memory only (lost on restart) | ETag is SHA-256. ## Bi-temporal object versioning Because every `S3Object` is a governed bi-temporal row, **S3 object versioning comes for free** — no separate versioning flag, no separate store. `PUT` creates a new row version (the prior one's `valid_to` closes); `DELETE` is a delete-marker; history is `AS OF`-queryable. ```bash # List every version of an object (the bi-temporal history) curl 'http://127.0.0.1:9191/cases/exhibit-1.txt?versions' \ -H 'Authorization: Bearer ' # Fetch a specific version by its system_from_ns curl 'http://127.0.0.1:9191/cases/exhibit-1.txt?versionId=1735490000000000000' \ -H 'Authorization: Bearer ' ``` > **Honest caveat:** `ListObjectVersions` is **not wired for cluster fan-out yet** (`s3_server.rs` follow-up). On `RELATA_PROFILE=cluster` it returns a clear typed error rather than silently returning partial results. Use it on `free`/`server` today; cluster support lands in a future release. ## Cross-protocol — read your S3 objects over SQL The killer feature: the same objects you wrote via `boto3` are queryable over SQL, joinable to your typed rows, and searchable via hybrid search: ```sql -- What's in the 'cases' bucket? SELECT key, size, content_hash, system_from FROM S3Object WHERE bucket = 'cases' ORDER BY system_from DESC LIMIT 20; -- Join objects to typed investigation rows SELECT s.key, s.size, p.name AS suspect FROM S3Object s JOIN Person p ON s.metadata->>'case_id' = p.case_id WHERE s.bucket = 'cases'; -- Object version history at a point in time SELECT key, size FROM S3Object WHERE bucket = 'cases' AND key = 'exhibit-1.txt' AS OF '2026-01-15T00:00:00'; ``` Governed identically: ACL, purpose, tenant isolation, and the audit chain apply on every S3 read/write just as they do on SQL. ## Event notifications Wire S3 events to downstream pipelines (SOAR, Lambda, dgrep) via `RELATA_S3_NOTIFY_URL`: ```bash RELATA_S3_NOTIFY_URL=https://soar.example.com/relata-s3-events relata serve ``` After each mutating S3 operation, a JSON payload `{"event":"","bucket":"…","key":"…"}` is POSTed fire-and-forget — errors are logged and discarded (non-blocking, won't slow the write path). ## Tips & takeaways - **Use a dedicated SigV4 secret in production.** `RELATA_S3_SECRET_KEY=$(openssl rand -hex 32)` keeps your S3 door credential separate from the master bearer token — blast-radius containment if the S3 key leaks. - **Tune `RELATA_S3_BLOB_THRESHOLD_MB` for your workload.** Small default (4 MiB) inlines objects for fast SQL reads; raise it for media-heavy workloads to keep the row store lean. - **Versioning is always on.** Unlike AWS S3 where you opt in per bucket, every Relata S3 object is bi-temporal from the first `PUT` — `?versions` works everywhere. - **Join objects to your graph.** Metadata you attach at `PUT` (`s3.put_object(Metadata={"case_id": "case-7"})`) is queryable in SQL — link exhibits to suspects without a separate join table. - **Audit the S3 door by principal.** Per-door ACL (`s3-client` principal) means every S3 write is forensically attributable — see [Per-Door ACL](/docs/guides/per-door-acl). ## See also - [Compatibility & Doors](/docs/compatibility) — full port + credential table - [Deploying Protocol Doors](/docs/deployment/protocol-doors) — Docker/K8s wiring for the S3 port - [Bi-temporal queries](/docs/reference/bitemporal) — `AS OF` over `S3Object` versions - [Per-Door ACL](/docs/guides/per-door-acl) — the `s3-client` Cedar principal - [Backup & Restore](/docs/guides/backup-restore) — the object store is also the backup backend ============================================================================== # Scaling URL: https://relatadb.dev/docs/guides/scaling ============================================================================== # Scaling RelataDB scales from a laptop (`free` profile, ~1 B entities) to a multi-node cluster (`cluster` profile, 100 B–1 T+ entities). The key levers: RAM walls that engage automatically on `server`/`cluster`, paged backends that spill to object storage, streaming execution that never OOMs, and lazy cold-restart that brings large nodes back in seconds. ## Choose the right profile ```bash # Local dev, CI, demos — unbounded RAM, eager restart RELATA_PROFILE=free relata serve # Single-node production — 1 GB RAM wall, disk-first walls, lazy restart on RELATA_PROFILE=server RELATA_BEARER_TOKEN= relata serve # Multi-node (alpha) — same as server plus coordination RELATA_PROFILE=cluster RELATA_BEARER_TOKEN= relata serve ``` | Profile | Scale target | Row-store RAM cap | Lazy restart default | |---|---|---|---| | `free` | ~1 B entities | unbounded | off | | `server` | ~10 B entities | 1024 MB | on | | `cluster` | ~100 B–1 T+ (alpha) | 1024 MB | on | Single-node `server` is the production-recommended profile today. `cluster` is alpha — petabyte-scale sharding is still in progress. ## All scaling knobs Set any of these to override the profile default. Explicit values always win. | Variable | Default (server/cluster) | Description | |---|---|---| | `RELATA_STORE_MAX_RAM_MB` | `1024` | Row-store RAM budget before spill to disk segments. Unbounded on `free`. | | `RELATA_GRAPH_RAM_BUDGET_MB` | `1024` | Graph adjacency RAM before paging out to `PagedCsrGraph`. | | `RELATA_IDENTITY_RAM_BUDGET_MB` | `1024` | Identity index RAM before going live-paged. | | `RELATA_DISKANN_MAX_RESIDENT` | `0` (unbounded) | Soft cap on RAM-resident HNSW vectors. Warns to shard/restart when exceeded. | | `RELATA_VECTOR_COLD_RESIDENT_MAX` | `100000` | Max staging vectors in IVF cold bucket before spill to `PagedAnnIndex`. | | `RELATA_MV_MAX_ROWS` | `1000000` | Max rows in an incremental materialized-view cache. `0` = unbounded. | | `RELATA_TOMBSTONE_CACHE_MAX_ROWS` | `1024` | Per-type tombstone cache entries. Lower saves RAM; raises on-disk re-reads. | | `RELATA_DECODED_SEGMENT_CACHE_MAX` | `64` | Max decoded disk-segment entries cached in RAM. | | `RELATA_LAZY_RESTART` | `true` on server/cluster | `true` loads manifest catalog only on restart — O(manifest) not O(rows). | | `RELATA_HYDRATE_RECENT_SEGMENTS` | `0` | With lazy restart, pre-warm the newest N segments. `0` = fully lazy. | | `RELATA_FLUSH_SEGMENT_MAX_ROWS` | `250000` | Max rows per Parquet segment flush. Larger deltas split into `ceil(delta/N)` segments. | ## Tune RAM walls for your node The 1 GB default is conservative. On a 32 GB node running only RelataDB, raise it: ```bash RELATA_PROFILE=server \ RELATA_BEARER_TOKEN= \ RELATA_STORE_MAX_RAM_MB=16384 \ RELATA_GRAPH_RAM_BUDGET_MB=8192 \ RELATA_IDENTITY_RAM_BUDGET_MB=4096 \ relata serve ``` The caps are byte-aware — small datasets never spill. The wall only bites when you actually exceed the budget. ## Disk-first walls and paged backends Every large structure has a paged backend. When the RAM budget is exceeded, data pages out to the object store and pages back in on demand. RAM becomes a cache; the object store is the truth. | Structure | Paged backend | Engagement condition | |---|---|---| | Authoritative rows | Disk segments (Parquet) | `RELATA_STORE_MAX_RAM_MB` exceeded | | Graph adjacency (CSR) | `PagedCsrGraph` | `RELATA_GRAPH_RAM_BUDGET_MB` exceeded | | Identity index | Live-paged | `RELATA_IDENTITY_RAM_BUDGET_MB` exceeded | | Vector index (HNSW) | `PagedAnnIndex` + IVF cold tier | `RELATA_DISKANN_MAX_RESIDENT` exceeded | | FTS postings + range indexes | `DiskIndexSource` | Automatic when segments spill | Paging adds disk-read latency on cold paths but eliminates OOM. On hot paths, the cache hierarchy absorbs most reads. ## Streaming execution The execution engine never materialises full intermediate results. Big joins, high-cardinality aggregates, and large result sets all stream: ```bash # This query on a 50 M-row table streams results — does not OOM relata query "SELECT dept, COUNT(*) FROM Person GROUP BY dept" ``` Hash-join and aggregate operators spill intermediate batches to the object store when they exceed the RAM budget, then merge-read in a streaming pass. ## Columnar analytics Filter-free `GROUP BY` uses a vectorised columnar path that reads only the referenced columns: ```sql -- Fast: only reads the "dept" column SELECT dept, COUNT(*) FROM Person GROUP BY dept; -- Slower: filter needs the full row to evaluate "active" SELECT dept, COUNT(*) FROM Person WHERE active GROUP BY dept; ``` For analytical workloads on large types, design queries to avoid per-row filters where possible, or add a bloom-filtered column to narrow the scan before the filter. ## Lazy restart `RELATA_LAZY_RESTART=true` (default on `server`/`cluster`) loads only the manifest catalog on startup. Rows hydrate on the first query that touches them. A 10 M-row node returns to ready in seconds instead of ~55 s. ```bash # Pre-warm the 5 most recent segments at startup, stay lazy for the rest RELATA_LAZY_RESTART=true \ RELATA_HYDRATE_RECENT_SEGMENTS=5 \ relata serve ``` This is the right setting for most production nodes: fast restart with warm cache for recent data. ## Cache hierarchy | Tier | Latency | Scope | |---|---|---| | L1 foyer (NVMe, S3-FIFO admission) | sub-ms | Single node | | L2 consistent-hash ring (~2 hot replicas) | ~1 ms | Reader pool | | L3 object store (S3/MinIO/GCS/Azure) | 20–100 ms | All data, durable | S3-FIFO admission at L1 is scan-resistant — a one-off full-table scan does not evict hot working set. L2 consistent-hashing means a request routed to the right reader node hits warm cache without cross-node fetch. ## Graph and vector scaling **Graph** — CSR adjacency stays RAM-resident up to `RELATA_GRAPH_RAM_BUDGET_MB`, then pages to `PagedCsrGraph`. The incremental degree index keeps `DEGREE()` queries O(1) regardless of graph size. **Vectors** — HNSW is the in-memory graph for recall. DiskANN is the warm tier backed by object-store segments. The IVF cold bucket stages vectors in RAM up to `RELATA_VECTOR_COLD_RESIDENT_MAX` before spilling to `PagedAnnIndex`. Increase `RELATA_DISKANN_MAX_RESIDENT` on nodes with headroom to avoid premature spill warnings. ## Cluster mode (alpha) Cluster adds coordinator/reader/writer/indexer roles, hash partitioning, and multi-region replication. Single-node `server` is production-recommended until the petabyte-scale sharding work lands; do not run cluster in production for data you cannot afford to lose until the alpha label is removed. ➡️ **The canonical cluster recipe — every required env var (plain `NODE_ID` / `CLUSTER_ROLE` / `CLUSTER_PEERS` — not the `RELATA_`-prefixed names, which are different, unrelated vars), a tested local 3-node example, and the gotchas (gRPC port must match across nodes, `CLUSTER_AUTH_TOKEN` fails silently, three different bind-var conventions) — lives in [Cluster Setup](/docs/deployment/cluster).** Scaling-wise, cluster inherits the same RAM walls, paged backends, streaming execution, and lazy restart described above; the cluster-specific levers are the partition-count (`RELATA_CLUSTER_SHARDS`, default 8) and the cross-region merge cadence (`RELATA_CROSS_REGION_MERGE_INTERVAL_SECS`). ## Performance characteristics Measured on representative hardware at 10 M rows. Use these as order-of-magnitude guides. | Operation | Overhead | Notes | |---|---|---| | Bitmap row filtering (ACL) | ~1.0× raw-scan | Effectively free — branch-predicted bitset | | Conditional ACL | ~1.32× raw-scan p50 | Budget gate: under 2.5× | | Cell masking | ~2.6× raw-scan p50 | Avoid on hot scan paths | | Cold-restart (eager) | ~55 s at 10 M rows | Use lazy restart instead | | Cold-restart (lazy) | seconds | O(manifest), rows hydrate on demand | Cell masking is expensive because it must inspect every field of every returned row. Apply it to specific columns in specific policies, not globally. ## Run the benchmark suite Before a capacity change, establish a baseline: ```bash # Quick gate — under 1 min, run before every merge cargo run -p relata-bench --release -- full --scale 100k --gate # Full suite without HNSW builds (~8 min) cargo run -p relata-bench --release -- full --scale 100k --no-ann # Full suite including HNSW builds (~15 min+) cargo run -p relata-bench --release -- full --scale 100k # All scales cargo run -p relata-bench --release -- full # Memory recall at scale cargo run -p relata-bench --release -- memory --scale 100k --all ``` Set `RELATA_BENCH_NO_SAVE=1` to suppress writing `relata-bench.json`. ## See also - [Configuration](/docs/guides/configuration) — every scaling env var - [Backup & Restore](/docs/guides/backup-restore) — object-store setup, lazy restart - [Observability](/docs/guides/observability) — RAM wall metrics, queue depth ============================================================================== # Auth & Security URL: https://relatadb.dev/docs/guides/security ============================================================================== # Auth & Security Security in RelataDB is in the query path. Every read runs through an ABAC engine, every write is audit-logged with a tamper-evident hash chain, and classified types are redacted at egress before serialisation — regardless of whether the query itself succeeded. ## Dev mode warning The `free` profile with no token set runs completely unauthenticated. pgwire is disabled, the admin surface (`/admin/*`) is open, and no auth is checked. This is intentional for local dev. If the node is reachable beyond localhost, secure it now: ```bash # Minimum viable secure setup RELATA_PROFILE=server \ RELATA_BEARER_TOKEN=$(openssl rand -hex 32) \ relata serve ``` The `server` and `cluster` profiles refuse to start without `RELATA_BEARER_TOKEN`. There is no way to accidentally run them unauthenticated. ## Step 1 — Set a bearer token Generate a token and set it before starting the server: ```bash export RELATA_BEARER_TOKEN=$(openssl rand -hex 32) export RELATA_ADMIN_TOKEN=$(openssl rand -hex 32) RELATA_PROFILE=server relata serve ``` Every request to every endpoint (HTTP, gRPC, Arrow Flight, pgwire, all protocol doors) now requires: ``` Authorization: Bearer ``` Test that auth is working: ```bash # Should return 401 curl -s -o /dev/null -w "%{http_code}" http://localhost:9090/query # Should return 200 curl -s -o /dev/null -w "%{http_code}" \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \ http://localhost:9090/health ``` The admin token gates all `/admin/*` management operations separately. On `server` and `cluster` profiles, **`RELATA_ADMIN_TOKEN` is required** — if unset, every `/admin/*` route returns `503 Service Unavailable`. The regular bearer token (`RELATA_BEARER_TOKEN`) is not a fallback for admin access (privilege separation). ```bash # Provision a tenant token via admin API curl -X POST http://localhost:9090/admin/tokens \ -H "Authorization: Bearer $RELATA_ADMIN_TOKEN" \ -H "Content-Type: application/json" \ -d '{"description":"acme-prod","expires_in_days":365}' # → { "token": "rlt_7f3a9b2c...", "id": "tok_..." } ``` On the `server` and `cluster` profiles the data-plane HTTP listener (`RELATA_HTTP_BIND`) and gRPC listener (`RELATA_GRPC_BIND`) bind `0.0.0.0` by default — auth/TLS posture is uniform across every profile. The env bearer token (`RELATA_BEARER_TOKEN`) is accepted on every interface; protect it with TLS (`RELATA_TLS_CERT`/`RELATA_TLS_KEY`) or a reverse proxy / sidecar that terminates auth before traffic reaches the node. The admin surface (`/admin/*`, `/platform/*`) is on a **separate, loopback-only** listener (`RELATA_ADMIN_BIND`, default `127.0.0.1:9091` — Zero-Trust control plane) and is never mounted on the data-plane listener. For network-exposed client traffic, prefer per-tenant registry tokens (rotatable, revocable, narrowly scoped) over reusing the env bearer token. ### Token lifecycle Tokens support expiry, self-service rotation, and per-tenant audit: ```bash # Create a token with a 30-day expiry curl -X POST http://localhost:9090/admin/tokens \ -H "Authorization: Bearer $RELATA_ADMIN_TOKEN" \ -H "Content-Type: application/json" \ -d '{"description":"short-lived","expires_in_days":30}' # Rotate your own token (tenant self-service — no admin token needed) curl -X POST http://localhost:9090/tokens/rotate \ -H "Authorization: Bearer rlt_7f3a9b2c..." # → { "token": "rlt_new...", "old_token_revoked": true } # View your token's last-use audit log (last 100 uses) curl http://localhost:9090/tokens/audit \ -H "Authorization: Bearer rlt_7f3a9b2c..." # Platform-wide audit (admin only) curl http://localhost:9090/admin/tokens/audit \ -H "Authorization: Bearer $RELATA_ADMIN_TOKEN" ``` Tokens within 30 days of expiry are logged at `WARN` on each use as a rotation reminder. ## Step 2 — Wire OIDC (optional) For production SSO, configure OIDC. The `oidc` mode trusts a front-proxy's verified principal; `oidc-verify` validates the JWT signature against the provider's JWKS endpoint in-process. ```bash RELATA_AUTH_MODE=oidc \ RELATA_OIDC_ISSUER=https://auth.example.com \ RELATA_OIDC_CLIENT_ID=relata \ RELATA_OIDC_JWKS_URI=https://auth.example.com/.well-known/jwks.json \ RELATA_OIDC_AUDIENCE=relata \ RELATA_PROFILE=server \ relata serve ``` For in-process token signature verification (recommended): ```bash RELATA_AUTH_MODE=oidc-verify \ RELATA_OIDC_ISSUER=https://auth.example.com \ RELATA_OIDC_JWKS_URI=https://auth.example.com/.well-known/jwks.json \ RELATA_OIDC_AUDIENCE=relata \ RELATA_PROFILE=server \ relata serve ``` Verify OIDC is working by obtaining a token from your provider and querying: ```bash TOKEN=$(curl -s -X POST https://auth.example.com/token \ -d "grant_type=client_credentials&client_id=relata&client_secret=$SECRET" \ | jq -r .access_token) curl http://localhost:9090/health/ready \ -H "Authorization: Bearer $TOKEN" ``` ## Step 3 — Wire mTLS (optional) For service-to-service auth where client certificates are already managed by your mesh: ```bash RELATA_AUTH_MODE=mtls \ RELATA_MTLS_CA_CERT_PATH=/etc/relata/tls/ca.crt \ RELATA_PROFILE=server \ relata serve ``` mTLS requires a CA cert (`RELATA_MTLS_CA_CERT_PATH`); client-cert requirement defaults to on (`RELATA_MTLS_REQUIRE_CLIENT_CERT=true`). To terminate TLS in-process on the listener, also set `RELATA_TLS_CERT` and `RELATA_TLS_KEY`. Clients must present a certificate signed by the configured CA. No bearer token is required when mTLS is the auth mode — the client cert is the credential. Test with curl: ```bash curl --cert client.crt --key client.key --cacert ca.crt \ https://localhost:9090/health/ready ``` ## Purpose enforcement `PURPOSE` is optional at the SQL layer. When you declare it, it is recorded in the audit log and evaluated by the ACL engine. ```sql -- With purpose (recorded in audit, ACL-evaluated) PURPOSE 'analytics' SELECT name, email FROM Person LIMIT 10; -- Without purpose (valid — purpose is optional) SELECT name FROM Person LIMIT 10; ``` In production, lock down to a registered list: ```bash RELATA_PURPOSE_MODE=strict \ RELATA_PURPOSES=analytics,audit,compliance,security_incident \ relata serve ``` Queries declaring an unregistered purpose return `403` (see [Error codes](/docs/reference/error-codes) — `MissingPurpose`/`UnknownPurpose`). Use `open` mode only in dev. ## EXPLAIN POLICY Before deploying a policy, validate what it does: ```sql EXPLAIN POLICY FOR PURPOSE 'analytics' ON Person; ``` The output shows which rows are visible, which columns are masked, and which deny rules fired. Run this whenever you change ACL policies — it catches overly broad denies before they hit production queries. ## Policy engine (ABAC) RelataDB ships its own Cedar-inspired ABAC engine — deny-wins semantics, bitmap row filtering, and cell masking. | Rule type | Performance | |---|---| | Bitmap row filtering | ~1.0× raw-scan overhead (effectively free) | | Conditional ACL | ~1.32× raw-scan p50 | | Cell masking | ~2.6× raw-scan p50 — avoid on hot paths | Policy example: ```cedar permit( principal == user::"alice", action == action::"read", resource in department::"finance" ) when { resource.purpose == "audit" }; forbid( principal, action == action::"read", resource ) when { resource.classification == "restricted" }; ``` Deny-wins means any matching `forbid` overrides all `permit` rules. Always test with `EXPLAIN POLICY` after adding a deny. ## Egress filtering Classified types are redacted at serialisation regardless of whether the query succeeded. This applies uniformly across HTTP, gRPC, Arrow Flight, pgwire, SPARQL, and every protocol door. Redacted types include `SourceTrueIdentity`, `SigintIntercept`, `AccessScopedIntercept`, and `LawfulInterceptRecord`. These never appear in tool results, query rows, or SDK responses. ## Rate limits Rate limits are per-IP and enforced on every request path: ```bash # Production defaults on server/cluster (per-IP token bucket) # RELATA_RATE_LIMIT_RPS=100000 (default) # RELATA_RATE_LIMIT_AUTH_FAIL_RPS=10 (default) ``` On exhaustion the server returns `429 Too Many Requests` with a `Retry-After` header. Setting `AUTH_FAIL_RPS=0` is treated as `1` — use `99999` to effectively disable. Auth-failure rate limiting is a brute-force guard. Keep it low in production. ## GDPR Art. 17 erasure The `ERASE SUBJECT` operator performs a governed right-to-erasure: shreds rows, destroys orphaned blobs, destroys the per-subject DEK via KMS, and returns a signed Art. 17 receipt. ```sql ERASE SUBJECT 'person-42' REASON 'gdpr-art17' CERTIFY; ``` The same operation is available via CLI, SDK, and MCP: ```bash # CLI relata query "ERASE SUBJECT 'person-42' REASON 'gdpr-art17' CERTIFY" # SDK (Python) client.identity.erase_subject("person-42", reason="gdpr-art17") # MCP tool # { "tool": "erase_subject", "subject_id": "person-42", "reason": "gdpr-art17" } ``` The returned receipt is content-addressed and verifiable against the audit chain. Store it — regulators may ask for it. ## Audit hash chain Every write is recorded in an append-only log with principal, timestamp, purpose, cost units, and a hash linking each entry to the previous one. ```bash # Check chain validity curl -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \ http://localhost:9090/audit/count # { "entries": 1248, "chain_valid": true } # Deep health check (chain + WAL + object-store) relata check ``` `chain_valid: false` is a security event. Treat it as a potential breach: isolate the node, preserve the WAL, and investigate. ## Protocol door security The protocol-compatibility doors bind to `127.0.0.1` by default; override per-door with `RELATA__BIND` on any profile (no license needed). pgwire fails closed without `RELATA_BEARER_TOKEN`. Put other doors behind a network policy or mTLS sidecar before exposing them beyond localhost. Every door presents a **distinct Cedar principal** (`s3-client`, `pgwire-client`, `mongo-client`, …) so you can grant least privilege per integration and audit-log which protocol wrote each row — see [Per-Door ACL](/docs/guides/per-door-acl). For full Docker/Kubernetes door wiring, see [Deploying Protocol Doors](/docs/deployment/protocol-doors). ## See also - [Per-Door ACL](/docs/guides/per-door-acl) — least privilege per integration via Cedar door principals - [Multi-Tenancy](/docs/guides/multi-tenancy) — org isolation, per-tenant DRK, quotas - [Configuration](/docs/guides/configuration) — all auth env vars - [Observability](/docs/guides/observability) — audit trail, correlation IDs ============================================================================== # Troubleshooting URL: https://relatadb.dev/docs/guides/troubleshooting ============================================================================== # Troubleshooting Relata fails **closed** and **loud**: it refuses to start on a bad configuration rather than silently degrading. Most issues are a single env var. If a symptom isn't here, see [Error Codes](/docs/reference/error-codes) and [Environment Variables](/docs/reference/env-vars). --- ## Startup failures (FATAL) Relata exits at startup with a clear `FATAL` line. Common causes: | Symptom | Cause | Fix | |---|---|---| | `RELATA_PROFILE=lite has been removed` | `lite` was removed outright | Use `RELATA_PROFILE=free` | | FATAL on a `RELATA_*` value | Strict parsing rejects malformed values | Correct the value — typos no longer fall back to defaults | | server/cluster refuses to start | Auth required on these profiles | Set `RELATA_BEARER_TOKEN` | | `address already in use` | Port 9090 (or a door port) is taken | Change `RELATA_PORT` / `RELATA__PORT`, or stop the other process | | `data dir is locked by another process` | Only one `relata serve` per data dir (exclusive flock) | Stop the other instance or use a different `RELATA_DATA_DIR` | > Tip: bump verbosity to see exactly where startup stalls — `RELATA_LOG_LEVEL=debug relata serve`. --- ## Doors won't connect / not reachable Every protocol door is **opt-in and off by default**, and on `free` the server binds to loopback only. | Symptom | Cause | Fix | |---|---|---| | Connection refused to a door | Door not enabled | Set `RELATA__ENABLE=true` (e.g. `RELATA_S3_ENABLE`, `RELATA_MONGO_ENABLE`) | | Reachable locally, not from another host/container | `free` binds `127.0.0.1` | `RELATA_HTTP_BIND=0.0.0.0` (no license needed); in Docker also `-p` publish the port | | S3 `403 SignatureDoesNotMatch` | Plaintext bearer sent where SigV4 is required | Sign with SigV4; the secret defaults to the bearer token — set `RELATA_S3_SECRET_KEY` to customise | | pgwire / MCP `401 Unauthorized` | Missing/invalid bearer | Send `Authorization: Bearer $RELATA_BEARER_TOKEN` | | `/debug/pprof/*` returns `404` or `401` | Profiling is off by default and admin-gated | `RELATA_PPROF_ENABLE=true` + `RELATA_ADMIN_TOKEN`, then send the admin bearer | --- ## Write rejected — `402 Payment Required` You hit the **Free-tier 10 GB storage cap** (the only paid limit). ```bash curl -s localhost:9090/metrics | grep relata_store_total_stored_bytes ``` - Reduce or expire old data, **or** activate a [license](/docs/reference/licensing) to lift the cap. - A soft warning is logged at 90% (9 GB); `402` is the hard stop at 10 GB. --- ## `403` on multi-tenant writes/reads Under `RELATA_TENANCY_MODE=multi`, **tenant-less requests fail closed** to prevent cross-tenant leakage. - Send the caller's tenant: header `X-Relata-Tenant-Id: ` (or a verified OIDC org claim). - `session_id` is **not** a security boundary — the tenant (org) is. - In genuine single-tenant dev, keep `RELATA_TENANCY_MODE=single`. --- ## Cluster reads return `206 Partial Content` A fan-out read couldn't reach every peer but returned what it could — Relata tells you honestly instead of a silent, incomplete `200`. - Check `CLUSTER_PEERS` — every peer URL must be reachable from the coordinator. - The response body carries `_relata_warnings` listing the failed peer(s). - Writes to the wrong shard are blocked by the `CROSS_SHARD_WRITE` guard — verify branch/shard routing. --- ## `relata_audit_chain_valid == 0` The tamper-evident audit chain detects a modified/deleted entry. **Treat as a security event:** isolate the node, preserve the WAL directory, and investigate. See [Observability](/docs/guides/observability). --- ## Queries return nothing (or are slow) - **`AS OF` scans:** temporal reads scan + bloom-prune segments today (a version index is landing to make this O(log n)). Narrow the time window or type. - **Multi-tenant scope:** in `single` mode, a global sanity gate may block broad scans — set `RELATA_GLOBAL_SCAN_ALLOWED=true` only for trusted diagnostics. - Inspect the plan with `EXPLAIN ANALYZE ` (per-operator timing). --- ## Getting more detail from logs ```bash RELATA_LOG_LEVEL=debug RELATA_LOG_FORMAT=json relata serve ``` - Levels: `trace` · `debug` · `info` · `warn` · `error`. - Every error response carries a `request_id` (RFC 7807 `application/problem+json`) — grep the logs/audit chain for it to trace a user-visible failure end-to-end. --- ## Still stuck - [Error Codes](/docs/reference/error-codes) — every `code` the API returns. - [Environment Variables](/docs/reference/env-vars) — the full `RELATA_*` surface. - [GitHub — relatadb](https://github.com/relatadb) · or [talk to enterprise](/get-license). ============================================================================== # Upgrading & Migration URL: https://relatadb.dev/docs/guides/upgrading ============================================================================== # Upgrading & Migration This page records the compatibility facts the code actually guarantees. Anything not verified by a test or an explicit version check is marked **untested — verify before relying** rather than claimed as a guarantee. Source: the repo's upgrading guide. > **Always back up first.** Take and `relata verify-backup ` a full snapshot before touching any node. Restore forward (old → new), not backward — backward format/version compatibility is untested. ## Upgrading to 2.0.0 2.0.0 is a **major** release with breaking changes (semver-mandated by the Zero-Trust / licensing / tenancy work). Review before upgrading from 1.x: - **Multi-tenant gating.** `RELATA_TENANCY_MODE=multi` now FATALs on `free` and `server` (both fixed at `max_tenants=1`). Multi-tenant mode is **cluster-only** and requires an effective `max_tenants > 1` (license value, or the `RELATA_MAX_TENANTS` override). If you ran `multi` on `free`/`server` in 1.x, move to `cluster` with a multi-tenant license or stay on `single`. See [Deployment](/docs/deployment). - **Licensing model v3.** The binary `"unlimited-storage"` capability is gone. Licenses carry two numeric parameters on the signed `NodeConfig` — `storage_max_gb` and `max_tenants` (`0` = unlimited). Re-issue licenses with the v3 tool; old `.lic` files without these fields are rejected. See [Licensing & Tiers](/docs/reference/licensing). - **Zero-Trust admin surface.** `/admin/*` and `/platform/*` moved to a separate loopbound listener (`RELATA_ADMIN_BIND`, default `127.0.0.1:9091`); they are no longer served on `RELATA_HTTP_BIND`. Set `RELATA_ADMIN_TOKEN` and reach the surface via port-forward / sidecar. - **Auth posture uniform.** No more implicit `free`-profile dev bypass — set `RELATA_OPEN_DEV_ALLOWED=true` explicitly for unauthenticated local dev, on every profile. - **Go SDK → `github.com/relatadb/sdk-go/v2`.** The major-version import-path suffix is mandatory; update your imports. See [SDKs](/docs/sdks/overview). - **Version lockstep.** Server and all SDKs/tray/Grafana/Helm now ship one version, enforced by the repo's `check_versions.py`. Validate on a staging copy of your data first; back up before upgrading. ## On-disk / artefact format compatibility ### Manifest (object-store layout) `relata-storage::manifest::ManifestVersion` is the source of truth for the commit-manifest layout. Two variants exist: `V1Single` (legacy single file) and `V2Sharded` (sharded layout) — **new writes always emit `V2Sharded`**. Readers detect the version from the index object. There is no in-place V1→V2 rewriter; the migration story for an existing V1 store is **untested — verify before relying** (prefer a fresh V2 store seeded from a restore over an in-place flip). ### Backup snapshot format `relata-storage::backup::BackupPayload` is a self-describing JSON artefact (`full-.json` / `incr-.json`) with `schema_version: u32` (currently **`1`**). Cross-tenant restores **are** enforced (`assert_agency` aborts a payload/organisation mismatch). **Tenant-scoped restore (`POST /admin/restore {"tenant": "..."}`) is rejected outright (`501`)** — the store-swap is an unconditional whole-store replacement, so a `tenant`-scoped request would silently wipe every *other* tenant. Restore only without a `tenant` field until a true per-tenant merge ships. Restoring a backup taken by a **newer** build into an **older** binary is **untested — verify before relying**. Always run `relata verify-backup ` before depending on a snapshot. ## Ontology version monotonicity `relata-ontology::OntologyVersion(u64)` is a monotonically-increasing counter, bumped on every type addition/modification and stored in the commit manifest. The version only ever moves **forward** within a branch; it never rewrites history. Downgrading the binary does not roll the ontology version back — an older binary reading a higher ontology version is **untested — verify before relying**. ## Rolling-upgrade ordering (recommendation) This ordering is a **recommendation**, not an enforced/tested invariant: 1. **Back up first** — take and `verify-backup` a full snapshot before touching any node. 2. **Validate config** — the CLI fails fast (exit `78`, `EX_CONFIG`) on an invalid `RELATA_*` enum/numeric value. Apply config changes to one node and confirm it starts clean before rolling out. 3. **Upgrade followers before the coordinator/writer** — in a cluster, roll reader/follower nodes first so the write path stays on the known-good version longest; promote the coordinator/writer last. 4. **`cluster` profile only — drain the node before stopping it.** Run `relata cluster drain --wait` and confirm `safe_to_stop: true` before deleting/restarting a writer's pod. Otherwise the passive heartbeat-timeout rebalancer (`RELATA_CLUSTER_DEAD_AFTER_SECS`, default 90 s) may evict and re-move partitions during a slow restart — real, avoidable data movement. 5. **One pod at a time** — the operator's StatefulSet uses `RollingUpdate` (default one pod at a time, or `spec.upgrade.maxUnavailable`). Wait for `GET /health/ready` 200 and (cluster) for `relata_replication_lag_seconds` to settle before continuing. 6. **Roll back by restoring**, not by downgrading in place — backward format/version compatibility is untested. ## Config migration ```bash relata config --migrate # migrate relata.toml / env vars across versions ``` The CLI now fails fast on invalid `RELATA_*` enum/numeric values and logs the offending variable, so config drift surfaces at startup rather than at runtime. ## Data migration | Path | When | Docs | |---|---|---| | `relata import --from postgres\|csv` | Migrate an existing database into Relata | [Connectors & Extensions](/docs/guides/connectors) · [Ingestion](/docs/guides/ingestion) | | `relata config --migrate` | Migrate config across versions | above | | Embedding-model migration | Change the sidecar model / vector dimension | [LLM & Embedding configuration](/docs/guides/llm-embedding) | | Backup → restore | Whole-store migration / disaster recovery | [Backup & Restore](/docs/guides/backup-restore) | For Neo4j / MongoDB / ClickHouse, `relata import --from ` is an honest stub today — each prints the documented CSV/NDJSON export workaround and exits non-zero. ## v1.4.2 → v1.5.0 (reference) A backward-compatible upgrade. `/query` responses added `processing_time_ms` alongside `elapsed_ms` (the latter retained for one release, removed in v1.6.0). `QueryError` exposed stable `REL_*` codes; the RFC 7807 `type` URI changed from `about:blank` to `https://relatadb.dev/errors/{code}`. 429 responses added `X-RateLimit-*` headers. Admission control moved to a real cost estimate (row count × join multiplier). Rolling: upgrade reader/indexer nodes first, writers last; all in-flight writes are safe (formats compatible). ## See also - [Deployment](/docs/deployment) — the three profiles and profile-specific gates - [Backup & Restore](/docs/guides/backup-restore) — snapshots, `verify-backup`, restore semantics - [Connectors & Extensions](/docs/guides/connectors) — `relata import --from` migration connectors - [Licensing & Tiers](/docs/reference/licensing) — the two-parameter licensing model - [Cluster Setup](/docs/deployment/cluster) — graceful-restart / drain procedure ============================================================================== # RelataDB — one database for messy, sensitive, connected data URL: https://relatadb.dev/docs/ ============================================================================== # RelataDB — one database for messy, sensitive, connected data RelataDB turns records arriving from many places — phone logs, bank transfers, social profiles, sanctions lists, app logs — into one connected, trustworthy, provable picture. It does the work most teams do by hand (cleaning, matching, proving, governing) **inside the database**, so your team spends its time asking questions, not reconciling data. > **Already running MongoDB / Postgres / Redis / Neo4j / ClickHouse / an S3 client?** > Point your existing client at Relata in 60 seconds — no SDK, no rewrite. Your driver, your ORM, your GUI tool all keep working. See **[Compatibility & Doors](/docs/compatibility)**. ## Is this for me? RelataDB earns its keep when your data is **at least one** of: - **Messy** — the same entity wears disguises across sources (`+44 7700…` in a call log, `07700…` in a CRM, an email at signup, a CustomerID in billing). You can't tell it's the same human without weeks of cleanup. - **Sensitive** — who-can-see-what rules differ by team, country, and purpose, and you have to enforce them on read. - **Needs proving** — every fact must be traceable to its source (audit, legal, compliance, court-grade replay). - **Connected** — you need the graph of who-knows-who, who-paid-who, who-called-who — without hand-building it. If your data is clean, public, and low-stakes, a regular database is simpler and cheaper. See **[Relata vs others](/docs/concepts/relata-vs-others)** for an honest side-by-side (including *when not to use Relata*). ## Try it in 60 seconds The server is one binary, no JVM, no external services. Pick the path that matches your stack — **both paths read and write the same governed store**. ### Path A — keep your existing client (drop-in) Best if you have a working app and want governance, history, and provenance **without rewriting code**. ```bash docker run -d -p 9090:9090 -p 27017:27017 -p 5433:5433 -p 6379:6379 \ -e RELATA_BEARER_TOKEN=change-me \ ghcr.io/relatadb/relata:2.0.0 ``` ```javascript // Your MongoDB client, unchanged except host/port + password = your token const { MongoClient } = require("mongodb"); const c = new MongoClient("mongodb://localhost:27017", { auth: { username: "relata", password: "change-me" }, }); await c.db("cases").collection("logs").insertOne({ _id: "r1", body: "hello" }); ``` Same story for `psql`, `redis-cli`, Neo4j, ClickHouse, `boto3`, Arrow Flight — one binary speaks 13 wire protocols. Full port table + per-protocol quickstarts: **[Compatibility & Doors](/docs/compatibility)**. ### Path B — use the SDK (full surface) Best for greenfield, or when you want first-class SQL, graph traversals, agent memory, MCP tools, and hybrid search. ```bash docker run -d -p 9090:9090 ghcr.io/relatadb/relata:2.0.0 # or: curl -sSf https://relatadb.dev/install.sh | sh && relata serve ``` ```python # pip install relata-sdk from relata import RelataClient with RelataClient("http://localhost:9090", purpose="analytics") as client: client.ingest("Person", [{"name": "Alice", "email": "alice@example.com"}]) for row in client.query("SELECT * FROM Person LIMIT 10"): print(row["name"], row["email"]) ``` ```typescript // npm install @zysec-ai/relata-sdk import { createClient } from "@zysec-ai/relata-sdk"; const relata = createClient("http://localhost:9090", { defaultPurpose: "analytics" }); const result = await relata.query("SELECT * FROM Person LIMIT 10"); console.log(result.rows); ``` SDK quickstart (Python / TypeScript / Go): **[Quickstart](/docs/quickstart)**. ## What you get by default Four things every other database makes **you** build yourself — RelataDB does them in the write and read paths, not as an afterthought: | What | Means | Why it matters | |---|---|---| | **Identity resolution** | The same person across phone / email / CustomerID is auto-merged into one entity, deterministically (no LLM guessing) | Stop hand-matching; query a graph that formed itself | | **Bi-temporal history** | Every row carries `valid_from/to` + `system_from/to`; rewind with `AS OF ''` | "What did we know on Tuesday?" is one query, not a restore | | **Provenance on every fact** | Where it came from, when, who put it there — tamper-evident audit hash chain | Court-grade replayable; auditors and lawyers get answers fast | | **Cell-level governance** | Cedar-inspired ABAC: who can see which cell, by purpose / team / country | Share the data without leaking the sensitive fields | Plus **hybrid search** (BM25 + HNSW vector + identity fusion in one query), **10 cognitive memory verbs** for AI agents (remember · recall · recognize · justify · consolidate · forget · associate · episodes · resolve · summarise), and **13 wire protocols** so your existing tools work unchanged. ## Where to go next A clear path from "is this for me?" to production — not 20 links. **Evaluate (~10 minutes)** - [Compatibility & Doors](/docs/compatibility) — bring your existing MongoDB / Postgres / Redis / Neo4j / ClickHouse / S3 client - [Quickstart](/docs/quickstart) — first query in 5 minutes (Python / TypeScript / Go) - [Relata vs others](/docs/concepts/relata-vs-others) — honest comparison, including *when not to use Relata* **See it in action** - [Use cases](/docs/use-cases/aml-sanctions-screening) — AML / sanctions, law-enforcement investigation graphs, telecom co-location, maritime dark-fleet, cyber Sigma detection, OSINT identity fusion, governed RAG **Build** - [SDK overview](/docs/sdks/overview) — Python · TypeScript · Go - [SQL reference](/docs/reference/sql) — the query plane (incl. graph, identity, and temporal verbs) **Run in production** - [Installation](/docs/installation) — Docker, binary, build-from-source - [Deployment](/docs/deployment) — profiles (`free` / `server` / `cluster`), persistence, graceful shutdown - [Deploying Protocol Doors](/docs/deployment/protocol-doors) — expose the compat doors safely in Docker / Kubernetes - [Configuration](/docs/guides/configuration) — the `RELATA_*` env-var surface ## What's honestly shipping RelataDB is on the path to 5.0. The governed core is real and smoke-tested — bi-temporal store, planner with ACL + organisation isolation, provenance/audit hash chain, SmartIngest identity detection, and all 8 compatibility doors + 5 native protocols (13 wire surfaces) from one binary. The honest gap list lives at **[Limits & Caveats](/docs/reference/limits)**. ============================================================================== # Installation URL: https://relatadb.dev/docs/installation ============================================================================== # Installation RelataDB ships as a single Rust binary — no JVM, no external services, no runtime dependencies. Docker is optional. The same binary serves all three deployment profiles (`free`, `server`, `cluster`); switching is one environment variable. ## Prerequisites | Requirement | Version | Notes | |---|---|---| | Rust toolchain | 1.85+ (MSRV, edition 2024) | `rustup default stable` — build from source only | | `protoc` | 3.x+ | Required for gRPC codegen at build time | | OS | Linux, macOS, Windows (WSL2) | x86\_64 and aarch64 supported | | Memory | 512 MB minimum | `server` profile caps at 1024 MB by default | | External services | None | No Redis, no Kafka, no JVM required | > **Note:** Docker users skip Rust and protoc entirely — the image is self-contained. --- ## Docker (recommended) The fastest path. The published image bundles the `relata` binary and defaults that work out of the box. The canonical image is `ghcr.io/relatadb/relata` (the same image is mirrored to Docker Hub as `openworkbench/relata-db` — pick whichever registry your environment prefers). ```bash docker run -d -p 9090:9090 --name relata \ ghcr.io/relatadb/relata:latest # or, the Docker Hub mirror (same image): docker run -d -p 9090:9090 --name relata \ openworkbench/relata-db:latest ``` Verify: ```bash curl http://127.0.0.1:9090/health # {"status":"ok"} ``` To persist data to a host directory: ```bash docker run -d -p 9090:9090 \ -v "$PWD/relata-data:/data/relata" \ --name relata \ ghcr.io/relatadb/relata:latest ``` > **Production:** pin to a specific tag (e.g. `ghcr.io/relatadb/relata:2.0.0`) and run `relata check` after startup. Using `:latest` in production is a reliability risk. --- ## Pre-built Binary The signed install script downloads the correct pre-built binary for your platform and verifies its SHA-256 checksum: ```bash curl -sSf https://relatadb.dev/install.sh | sh relata --version ``` Supported targets: `x86_64-unknown-linux-gnu`, `aarch64-unknown-linux-gnu`, `x86_64-apple-darwin`, `aarch64-apple-darwin`. --- ## Build from Source Source access is available to licensees. Once you have access, build with Rust 1.85+, `protoc`, and `cargo`: ```bash cargo install --path crates/relata-cli --locked # → ~/.cargo/bin/relata # or, without installing system-wide: cargo build --workspace --release # → target/release/relata ``` `--locked` uses the exact dependency versions in `Cargo.lock`. Request source access via [github.com/relatadb](https://github.com/relatadb). > **Note:** `unsafe` is forbidden (`#![forbid(unsafe_code)]` in every crate) and `missing_docs` is denied. Supply-chain check (optional but recommended before production): ```bash cargo deny check ``` --- ## Verify the Install ```bash relata --version # relata 2.0.0 ``` Start the server and confirm it responds: ```bash relata serve & curl http://127.0.0.1:9090/health # {"status":"ok"} ``` Run the full integration check suite (150+ checks covering storage, query, auth, and protocol compatibility): ```bash relata check ``` > Run `relata check` after every install or upgrade. It exercises the full server surface and catches misconfiguration before your workload does. --- ## First-Run Checklist 1. Start the server: `relata serve` 2. Confirm health: `curl http://127.0.0.1:9090/health` 3. Run a query: `relata query "SELECT * FROM Person LIMIT 5"` 4. Set a bearer token: `RELATA_BEARER_TOKEN=your-secret relata serve` 5. Choose a profile: `RELATA_PROFILE=server relata serve` 6. Run integration checks: `relata check` --- ## Deployment Profiles Set `RELATA_PROFILE` to switch profiles. The binary is identical across all three. | Profile | Default for | Key behaviour | |---|---|---| | `free` | Local dev, evaluation | Unbounded RAM, eager restart on boot, no auth enforced by default | | `server` | Single-node production | 1024 MB RAM cap, lazy restart (O(manifest) not O(rows)), auth gated | | `cluster` | Multi-node scale-out | Coordinator / reader / writer / indexer roles, hash partitioning, multi-region replication | `free` is the default so `relata serve` works immediately for evaluation. Move to `server` before handling real data. > `lite` was a legacy alias for `free` and is now rejected outright — startup fails if `RELATA_PROFILE=lite` is set. Use `free`. --- ## Key CLI Commands | Command | Purpose | |---|---| | `relata serve` | Start the HTTP/Postgres wire/gRPC server | | `relata query "SQL"` | Run a one-off query | | `relata check` | Run 150+ integration checks | | `relata backup` / `relata restore` | Snapshot backup and restore | All commands respect `RELATA_PROFILE` and the full `RELATA_*` environment variable matrix documented in [Configuration](/docs/guides/configuration). --- ## Air-Gapped / Demo Mode There's no single air-gap switch — outbound calls are opt-in already. Leave `RELATA_LLM_URL`/`RELATA_LLM_API_KEY` and `RELATA_OTLP_ENDPOINT` unset and the node makes no LLM or telemetry calls. To disable rate limits for load testing, benchmarks, or demo environments, raise the per-IP limiter values directly: ```bash RELATA_RATE_LIMIT_RPS=99999 \ RELATA_RATE_LIMIT_AUTH_FAIL_RPS=99999 \ relata serve ``` > **Warning:** Do not disable rate limits in production. They are part of the abuse-prevention posture. Use network-layer controls (mTLS sidecar, NetworkPolicy) if you need to enforce them at the infrastructure level instead. Setting `RELATA_RATE_LIMIT_AUTH_FAIL_RPS=0` is treated as `1` by the backend — zero is clamped to the minimum. Use `99999` to effectively disable the auth-fail bucket. --- ## See Also - [Quickstart](/docs/quickstart) — get running and issue your first query in five minutes - [Configuration](/docs/guides/configuration) — full environment variable reference - [Deployment](/docs/deployment) — production topology, TLS, object storage ============================================================================== # Quickstart — first query in 5 minutes URL: https://relatadb.dev/docs/quickstart ============================================================================== # Quickstart — first query in 5 minutes Pick the path that matches your stack. > **Already running MongoDB / Postgres / Redis / Neo4j / ClickHouse / an S3 client?** > You don't need an SDK — point your existing client at Relata's compat port and use your bearer token as the password. Full port table + 3-step quickstarts per protocol: **[Compatibility & Doors](/docs/compatibility)**. This page is the SDK path. ## Prerequisites ```bash # Start the server (terminal 1) — Docker is the fastest path docker run -d -p 9090:9090 --name relata ghcr.io/relatadb/relata:2.0.0 # or from source: cargo run -p relata-cli -- serve # Check it's live (terminal 2) curl http://127.0.0.1:9090/health ``` No token is needed for local dev — the server starts in unauthenticated mode. Set `RELATA_BEARER_TOKEN` before handling real data. --- Pick your language. Each example below connects to a local Relata server, inserts a row, and queries it back. --- ## Python ```bash pip install relata-sdk ``` ```python from relata import RelataClient # 1. Connect (no token needed for local dev). client = RelataClient("http://localhost:9090", purpose="analytics") # 2. Insert a row. client.query("INSERT INTO Person (_pk, name, email) VALUES ('p1', 'Alice', 'alice@example.com')") # 3. Query it back. result = client.query("SELECT * FROM Person LIMIT 5") for row in result: print(row["name"], row["email"]) # 4. Search (BM25 + hybrid). hits = client.search("alice", "Person", limit=5, highlight=True) for hit in hits.hits: print(hit.score, hit.fields.get("name")) # 5. Memory (agent cognitive verbs). from relata import Memory mem = Memory("http://localhost:9090", bearer_token="", purpose="agent") mid = mem.add("Alice prefers dark mode") results = mem.search("ui preferences", top_k=3) ``` ### Jupyter notebook ```python %load_ext relata.ipython %%relata --purpose analytics SELECT * FROM Person LIMIT 10 ``` Results appear as a pandas DataFrame automatically. --- ## TypeScript ```bash npm install @zysec-ai/relata-sdk ``` ```typescript import { RelataClient } from "@zysec-ai/relata-sdk"; // 1. Connect. const client = new RelataClient({ baseUrl: "http://localhost:9090" }); // 2. Insert a row. await client.query({ purpose: "analytics", sql: "INSERT INTO Person (_pk, name, email) VALUES ('p1', 'Alice', 'alice@example.com')" }); // 3. Query it back. const result = await client.query({ purpose: "analytics", sql: "SELECT * FROM Person LIMIT 5" }); for (const row of result.data) { console.log(row.name, row.email); } // 4. Search with matching strategy. const hits = await client.search({ query: "alice", type: "Person", limit: 5, matchingStrategy: "all" }); // 5. Memory. await client.remember("Alice prefers dark mode", { purpose: "agent" }); const memories = await client.recall("ui preferences", { topK: 3 }); ``` --- ## Go ```bash go get github.com/relatadb/sdk-go/v2 ``` ```go package main import ( "context" "fmt" "time" "github.com/relatadb/sdk-go/v2/relata" ) func main() { ctx := context.Background() // 1. Connect. client := relata.New("http://localhost:9090", &relata.ClientOptions{ BearerToken: "", DefaultPurpose: "analytics", Timeout: 30 * time.Second, }) // 2. Insert a row. client.Query(ctx, "INSERT INTO Person (_pk, name, email) VALUES ('p1', 'Alice', 'alice@example.com')") // 3. Query it back. result, _ := client.Query(ctx, "SELECT * FROM Person LIMIT 5") for _, row := range result.Rows { fmt.Println(row["name"], row["email"]) } // 4. Search with typo tolerance. hits, _ := client.Search(ctx, "alice", "Person", relata.WithSearchLimit(5), relata.WithMatchingStrategy("all"), ) // 5. Memory. mem, _ := relata.NewMemory("http://localhost:9090", "agent", &relata.MemoryOptions{ Timeout: 30 * time.Second, }) mem.Add(ctx, "Alice prefers dark mode") results, _ := mem.Search(ctx, "ui preferences", relata.WithTopK(3)) _ = results _ = hits } ``` --- ## Parameterized queries Use `$1`, `$2`, … placeholders to bind values server-side — no concatenation, no injection risk. **Python** — `?` placeholders are auto-rewritten to `$1`, `$2`, … ```python result = client.query_params( "SELECT * FROM Person WHERE age = $1 AND city = $2", [25, "Karachi"], purpose="analytics", ) # ? form also works result = client.query_params("SELECT * FROM T WHERE id = ?", [42]) ``` **TypeScript** ```typescript const r = await relata.queryWithParams( "SELECT * FROM Person WHERE age = $1 AND city = $2", [25, "Karachi"], { purpose: "analytics" }, ); ``` **Go** ```go result, err := client.QueryWithParams(ctx, "SELECT * FROM Person WHERE age = $1 AND city = $2", []any{25, "Karachi"}, relata.WithPurpose("analytics"), ) ``` ## Text embedding via VectorClient The TypeScript `VectorClient` exposes `embed` and `embedBatch` to call the server's `/embed` endpoint directly. The server uses its built-in CPU lexical embedder (128-dim) when `RELATA_ACCEL_ENDPOINT` is unset, or the GPU sidecar when configured. ```typescript import { createClient, VectorClient } from "@zysec-ai/relata-sdk"; const relata = createClient("http://localhost:9090", { bearerToken: process.env.RELATA_TOKEN, }); const vectors = new VectorClient(relata); // Single text const { embedding, model, dim } = await vectors.embed("Alice Smith"); console.log(`dim=${dim} model=${model}`); // Batch const { embeddings, count } = await vectors.embedBatch(["Alice", "Bob"]); console.log(`${count} embeddings, each dim=${embeddings[0].length}`); ``` --- ## What's next - **Search cookbook**: [Query cookbook](/docs/reference/query-cookbook) - **SQL grammar**: [SQL reference](/docs/reference/sql) - **SDK capability matrix**: [SDK overview](/docs/sdks/overview) - **API explorer**: open `http://localhost:9090/api-docs` in your browser ============================================================================== # Agent memory surface URL: https://relatadb.dev/docs/reference/agent-memory ============================================================================== # Agent memory surface RelataDB is the **agentic-first ontological database** — one governed, bi-temporal knowledge base that speaks every database protocol. This page covers its **agent-memory surface**: the governed memory layer where what an agent knew, when, and why is **defensible**, not just convenient. Most agent-memory products (Mem0, RushDB, Zep) optimize for *convenience*: push JSON, get semantic recall, schema-free, with embeddings computed for you. Relata optimizes for *accountability*: every belief is bi-temporal, provenance-stamped, and access-controlled. Since v1.1 embeddings are caller-supplied — either pre-compute `_emb_text` in the row payload or run the embedder sidecar for async drain — so the ingest hot path is pure throughput (no per-row model call). If you need memory that *feels* smart, those are great; if the memory must answer **"what did the agent know, when, and why — and who was allowed to see it?"**, that's Relata. ## Relata vs. the convenience-memory cohort | | **Mem0 / RushDB / Zep** | **Relata** | |---|---|---| | Optimized for | Convenience, fast onboarding | Accountability, defensibility | | History | Current state | **Bi-temporal** — `AS OF` valid + system time | | Provenance | None / light | **PROV-O per row + tamper-evident hash chain** | | Access control | ACID only | **Cell-level ACL (Cedar) + org isolation + per-tenant encryption/quotas/namespaces** | | Schema | Schema-free | **Schema-as-code ontology** (governed) | | Model | On Neo4j / hosted | **Own Rust engine**, object-store native, single binary | | License | AGPL-3.0-only / hosted | **AGPL-3.0-only** (source on request) | | Best fit | Personalization, RAG, schema-free apps | Regulated, audited, intel/LEA/FININT, court-grade | ## Why Relata for agent memory | Concern | Relata answer | |---|---| | **Bi-temporal recall** | Every memory carries `valid_from/to` (when it was true) *and* `system_from/to` (when Relata learned it). Query "what did the agent believe at T?" exactly. | | **Tamper-evident audit** | Hash-chained commit manifests mean any deletion or mutation leaves a forensic trail. Compliance and incident-response needs are met out of the box. | | **Provenance chain** | Every MemoryItem links back to the ToolCall and AgentSession that produced it. Replay exactly how a decision was reached. | | **Governed access** | Cedar-inspired ABAC + PURPOSE restrict which agents can read which memories. Multi-tenant isolation is enforced at the query-planner level. | ## Problems we solve | Problem | Relata solution | |---|---| | **Finite context window** — agents forget anything that doesn't fit the prompt | External governed store + `recall` injects a bounded, ranked slice per turn (`LIMIT N BUDGET T`) | | **Memory bloat degrades recall** — as memory grows, noise drowns signal | Hybrid retrieval (BM25 + vector + graph) with early pruning; `consolidate` supersedes stale facts instead of accumulating them | | **No "current truth"** — facts change but old beliefs linger | Bi-temporal supersession: old belief closed at `valid_to=now`, new inserted, both `AS OF`-reconstructable | | **Hallucination amplification / memory poisoning** — retrieved "memories" have no source | PROV-O provenance per row + tamper-evident hash chain; every result is `justify`-able; no source = not a memory | | **No time-travel** — can't ask "what did the agent know at T?" | Bi-temporal `recall … AS OF ''` reconstructs the agent's state at any moment | | **Multi-agent collision / privacy leak** | Per-agent/session scoping + cell-level ACL (Cedar) + org isolation + sub-tenant namespaces | | **No compliance** — GDPR delete, audit, access logs | `forget` with retention + legal-hold, hash-chained audit, PURPOSE recording | | **Retrieval is slow / expensive** | Lazy detection + deterministic canonical ops on the hot path (no LLM/GPU) + tiered cache (RAM/SSD/object-store) | ## Unlimited memory Relata gives an agent **unbounded memory with bounded prompts** by separating *what the agent knows* (unlimited) from *what it sees per turn* (bounded): - **Capacity is unbounded** — durable storage is object-store (S3 / self-hosted S3-compatible), not RAM-bound like a vector DB. You run out of bucket, not memory. - **The agent never loads it all** — each turn, `recall` returns only the small, relevant, ranked slice, capped by `LIMIT N BUDGET T`. A 10-year, billion-row memory and a 1 MB memory cost the *same* prompt budget; retrieval is selective, not exhaustive. - **Cold vs hot** — cold history lives on object storage; active-case data is promoted into RAM/SSD via the tiered cache. Archive-scale capacity with interactive latency. ## Fast and efficient on constrained hardware Relata is RAG-on-memory, engineered to run on a laptop or edge box with **no GPU**: - **No LLM on the hot path.** Ingest canonicalizes + validates declared identities (deterministic, cheap). Auto-detection/extraction is **lazy via materialized views** — you pay detection cost only on the slice you query, not on 100% of rows at write time. - **Early-pruned retrieval.** IdentityIndex bloom filters + graph pushdown + a BM25 shortlist mean the vector path scans a tiny candidate set, not the whole corpus. - **Single binary, embedded `free` profile** — one process, in-memory + optional object-store, running alongside the agent's own loop. No separate vector server, Redis, and graph DB to operate. - **Token-efficient** — hybrid (keyword + vector + graph) returns higher-precision results, so fewer tokens are injected for the same answer quality, and the budget operator sets a hard ceiling. > **Honest scope:** single-node `free`/`server` with object-store + tiered cache is the > shipped answer today. Multi-node cluster scale (deep partitioning) is the cluster profile > and still maturing. ## The five canonical memory types | Type | Description | |---|---| | `MemoryItem` | The atomic unit. Holds `content`, `session_id`, `confidence [0,1]`, and bi-temporal timestamps. Created by `remember`, consumed by `recall`. | | `DecisionRecord` | A choice an agent made, with the inputs it considered and the rationale it logged. Created by `justify`. | | `AgentSession` | A bounded interaction window grouping related MemoryItems and ToolCalls. Created implicitly on first `remember` for a new `session_id`. | | `ToolCall` | One invocation of a tool: name, arguments, result, latency. Linked from MemoryItems so the provenance chain is complete. | | `Episode` | A higher-order grouping of related sessions forming a coherent narrative arc. Retrieved by `episodes_in`. | ## The 10 cognitive verbs | Verb | HTTP surface | Description | |---|---|---| | `remember` | `POST /memory/remember` | Store a new MemoryItem. Returns `id`, `confidence`, `valid_from`. | | `recall` | `GET /memory/recall?q=...` | Semantic + temporal search over MemoryItems. Returns ranked list with provenance. | | `recognize` | `GET /memory/recognize/:id` | Fetch one MemoryItem by id with full provenance chain attached. | | `episodes` | `GET /memory/episodes?session_id=...` | List Episodes for a session, ordered by `valid_from`. | | `justify` | `GET /memory/justify/:id` | Trace the decision provenance for a MemoryItem — returns the chain of ToolCalls and DecisionRecords that produced it. | | `consolidate` | `POST /memory/consolidate` | Supersede an existing MemoryItem with updated content. The old item is retained in history; a new one is created with higher confidence. | | `forget` | `DELETE /memory/forget/:id` | Schedule a MemoryItem for retention-policy deletion. Does not hard-delete immediately — the item remains queryable until the retention window expires. | | `associate` | `POST /memory/associate` | Link two memory items / entities with a typed, provenance-stamped association (`from_id`, `to_id`, `relation`). | | `resolve` | `GET /memory/resolve/:id` | Resolve a memory reference through its supersession chain to the canonical (live) MemoryItem. | | `summarise` | `POST /memory/summarise` | Produce a governed, provenance-stamped summary of a session or topic. | > **Batch helpers:** `POST /memory/remember/batch` and `POST /memory/associate/batch` are high-throughput variants that amortise the per-record index flush across a whole batch; per-item errors are returned per element and valid items still commit. > **Surface note:** The cognitive verbs are **MCP tools and REST endpoints** — > they are **not** SQL keywords. The `LIMIT N BUDGET T` retrieval knobs > and `recall … AS OF ''` time-travel are expressed as **verb/REST parameters** > (`?top_k=`, `?as_of=` on `GET /memory/recall`). The underlying bi-temporal `AS OF` > is available as real SQL over the base types > (`SELECT … FROM MemoryItem AS OF ''`). ## Recall-quality knobs — tuning *what* comes back Recall isn't a black box. Five keyword parameters (the "retrieval-quality operators") let you shape recall for your domain — confidence floors, memory decay, hard token budgets, Ebbinghaus forgetting, and early-cancel. All five are accepted by `GET /memory/recall`, the `recall` MCP tool, and every SDK's `search(...)` / `search_detailed(...)` method. | Parameter | Underlying operator | What it does | Example | |---|---|---|---| | `min_confidence` | `CONFIDENCE(f)` | Drop memories below this confidence floor. Use to keep low-quality / speculative beliefs out of the prompt. | `min_confidence=0.7` | | `recency_half_life_secs` | `RECENCY(λ)` | Exponential score decay half-life, in seconds. Recent memories rank higher; old ones don't vanish, they just decay. | `recency_half_life_secs=604800` (1 week) | | `budget_tokens` | `BUDGET(t)` | **Hard ceiling** on the cumulative token cost of returned memories. The server stops emitting once the budget is hit — your prompt literally cannot overflow. | `budget_tokens=2000` | | `stability_days` | `FORGETTING_CURVE(d)` | Ebbinghaus stability parameter, in days. Mirrors human-memory reinforcement: memories that haven't been re-touched decay faster. | `stability_days=30` | | `cancel_threshold` | `CANCEL_WHEN(threshold)` | Short-circuit the scan the moment a hit exceeds this score. Use when you want "the first great match, then stop." | `cancel_threshold=0.95` | ### Reading the effect back — `search_detailed` Two response fields let you *observe* the knobs' effect, not just set them: - `recall_cost_tokens` — the running token total under `BUDGET` (how much of the budget was consumed). - `cancelled` — whether `CANCEL_WHEN` short-circuited the scan (`true`) or the full ranking ran (`false`). These appear on the detailed recall envelope (`Memory.search_detailed(...)` in all three SDKs; `GET /memory/recall?detailed=true`). ### SDK examples **Python** ```python from relata import Memory mem = Memory("http://localhost:9090", bearer_token="", purpose="agent") # Tight budget, recency-weighted, stop at the first near-certain hit result = mem.search_detailed( "how do we reset the IR sensor?", top_k=10, min_confidence=0.6, # CONFIDENCE floor recency_half_life_secs=259200, # 3-day half-life (RECENCY) budget_tokens=1500, # hard prompt budget (BUDGET) cancel_threshold=0.92, # stop early on a great match (CANCEL_WHEN) ) print(result["recall_cost_tokens"], result["cancelled"]) ``` **TypeScript** ```typescript const detail = await mem.searchDetailed("how do we reset the IR sensor?", { topK: 10, minConfidence: 0.6, recencyHalfLifeSecs: 259200, budgetTokens: 1500, cancelThreshold: 0.92, }); console.log(detail.recall_cost_tokens, detail.cancelled); ``` **Go** ```go res, _ := mem.SearchDetailed(ctx, "how do we reset the IR sensor?", relata.WithTopK(10), relata.WithMinConfidence(0.6), relata.WithRecencyHalfLife(259200), relata.WithBudgetTokens(1500), relata.WithCancelThreshold(0.92), ) fmt.Println(res.RecallCostTokens, res.Cancelled) ``` ### Tips & takeaways - **Start with `budget_tokens`.** It's the single biggest win for agent loops — the prompt literally cannot overflow the model's context window. Pick a budget that leaves room for the system prompt + tool output + the response. - **Pair `recency_half_life_secs` with `stability_days` only if it matters.** For most RAG use cases `recency_half_life_secs` alone is enough; `stability_days` (Ebbinghaus) is for long-lived agents that should "remember" frequently-revisited facts. - **`cancel_threshold` trades coverage for latency.** Set it when one excellent match is enough (FAQ retrieval, lookups); leave it unset when you want a ranked slate (brainstorming, summarization). - **`min_confidence` is a safety net, not a ranking signal.** It filters; it doesn't sort. Combine with `recency_*` for the actual ranking shape you want. - **Bi-temporal recall composes with all five.** Add `as_of=''` to reconstruct what the agent *believed at T* under the same quality constraints — "what would we have recalled on Tuesday under a 1500-token budget?" Cross-ref: [Concepts: Agent Memory](/docs/concepts/agent-memory) · [MCP tools](/docs/reference/mcp-tools) · [Bi-temporal queries](/docs/reference/bitemporal) · [Limits](/docs/reference/limits) --- ## Quick-start via MCP Relata exposes all the cognitive verbs as MCP tools. Point your agent's MCP client at the server and call tools directly: ```json // MCP initialize POST /mcp/initialize {"protocolVersion":"2024-11-05","clientInfo":{"name":"my-agent","version":"1.0"}} // MCP list tools GET /mcp/tools // → returns the memory tools: remember, remember_batch, recall, recognize, episodes_in, justify, consolidate, forget, associate, resolve, summarise // MCP call — remember POST /mcp/tools/call { "name": "remember", "arguments": { "content": "User prefers concise summaries", "session_id": "sess_abc123", "confidence": 0.9, "purpose": "personalisation" } } // → {"isError": false, "content": [{"type":"text","text":"{\"id\":\"\",\"confidence\":0.9,...}"}]} // MCP call — remember_batch (high-throughput write path) POST /mcp/tools/call { "name": "remember_batch", "arguments": { "items": [ {"content": "User prefers dark mode", "session_id": "sess_abc123", "confidence": 0.9}, {"content": "User timezone is UTC+5:30", "session_id": "sess_abc123", "confidence": 0.85}, {"content": "User speaks English and Hindi", "session_id": "sess_abc123", "confidence": 0.95} ], "purpose": "personalisation" } } ``` ## Quick-start via HTTP REST ### Step 1 — Remember a fact ```http POST /memory/remember Content-Type: application/json { "content": "User prefers dark mode", "session_id": "sess_abc123", "confidence": 0.9, "purpose": "personalisation" } ``` Response: ```json { "isError": false, "content": [{ "type": "text", "text": "{\"id\":\"550e8400-e29b-41d4-a716-446655440000\",\"confidence\":0.9,\"valid_from\":1750000000000000000}" }] } ``` ### Step 2 — Recall related memories ```http GET /memory/recall?q=user+preferences&top_k=5&purpose=personalisation ``` Response: ```json { "isError": false, "content": [{ "type": "text", "text": "{\"memories\":[{\"id\":\"550e8400...\",\"content\":\"User prefers dark mode\",\"score\":0.97}],\"total\":1}" }] } ``` ### Step 3 — Justify a memory (provenance) ```http GET /memory/justify/550e8400-e29b-41d4-a716-446655440000?purpose=audit ``` Returns the full ToolCall → MemoryItem → DecisionRecord provenance chain. ### Step 4 — Consolidate (update with higher confidence) ```http POST /memory/consolidate Content-Type: application/json { "id": "550e8400-e29b-41d4-a716-446655440000", "content": "User prefers dark mode (confirmed across three sessions)", "confidence": 0.98, "purpose": "personalisation" } ``` Returns `{"superseded": "", "new_id": ""}`. ### Step 5 — Schedule forgetting ```http DELETE /memory/forget/550e8400-e29b-41d4-a716-446655440000?retain_days=30&purpose=gdpr ``` Returns `{"scheduled": true, "policy": "delete_after_30d"}`. ## Governance model Every memory operation passes through three governance layers: 1. **PURPOSE** — The `purpose` field (e.g. `"personalisation"`, `"audit"`, `"gdpr"`) is recorded in the audit log and may be required by ACL policy. When `PURPOSE` is omitted, the operation is recorded as unpurposed but still allowed in `open` mode. 2. **ACL** — Cedar-inspired ABAC rules control which agent identities can write, read, or delete memories in a given organisation. Cell-level masking applies to sensitive fields. 3. **Provenance chain** — Every MemoryItem references the `AgentSession` and `ToolCall` that produced it. The hash-chained manifest makes the chain tamper-evident. `justify` replays the chain on demand. ## Multi-agent isolation When multiple agents share one Relata instance, pass a `tenant_id` header: ```http POST /memory/remember X-Relata-Agency: agent-team-alpha ``` The planner enforces `tenant_id` keying so agents never see each other's memories without an explicit cross-organisation grant. > **`session_id` is not a tenant boundary.** The `session_id` (typically > an agent's Ed25519 pubkey) groups a conversation — it is **not** an isolation > key. Two organisations reusing the same `session_id` are kept apart only by > their **tenant** (org). On a multi-tenant profile (`server`/`cluster` with more > than one tenant, or `RELATA_TENANCY_MODE=multi`) memory writes with **no** tenant > are rejected `403`. Present a per-tenant credential — `X-Organization-Id` or a > tenant-scoped bearer token — for every memory write. ## See also - [MCP tools reference](/docs/reference/mcp-tools) - [Bi-temporal queries](/docs/reference/bitemporal) - [Error codes](/docs/reference/error-codes) ============================================================================== # HTTP API Reference URL: https://relatadb.dev/docs/reference/api-reference ============================================================================== # HTTP API Reference RelataDB exposes a REST API on the HTTP port (default 9090). All endpoints require Bearer token authentication via the `Authorization` header. ## Authentication ```http Authorization: Bearer ``` Create tokens via the CLI: `relata tenant members --add --role admin`, or via `POST /tokens`. ## Query | Method | Endpoint | Description | |---|---|---| | `POST` | `/query` | Execute a SQL (or Cypher) query | | `POST` | `/query/arrow` | Execute and return Arrow IPC | | `POST` | `/query/stream` | Streaming query results (SSE) | | `GET` | `/export` | Export query results as CSV/JSON | ```bash curl -X POST http://localhost:9090/query \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \ -H "Content-Type: application/json" \ -d '{"sql":"SELECT * FROM Person LIMIT 5","purpose":"analytics"}' ``` ## Search | Method | Endpoint | Description | |---|---|---| | `POST` | `/search` | Full-text / vector / hybrid search | | `POST` | `/multi-search` | Multi-query with rank fusion | ## Ingest | Method | Endpoint | Description | |---|---|---| | `POST` | `/ingest` | Ingest CSV/JSON data | | `POST` | `/ingest/bulk` | Bulk ingest | | `POST` | `/ingest/document` | Ingest a document (with auto-chunking) | | `POST` | `/ingest/media` | Ingest media (image/audio/video) | | `GET` | `/ingest/tasks/:id` | Check ingest task status | ## Tenants (3 surfaces) ### `/tenants` — tenant admin | Method | Endpoint | Description | |---|---|---| | `POST` | `/tenants` | Create tenant | | `GET` | `/tenants` | List tenants | | `GET` | `/tenants/me` | Caller's tenant context | | `GET` | `/tenants/:id` | Tenant detail | | `PATCH` | `/tenants/:id` | Update name/classification | | `POST` | `/tenants/:id/suspend` | Suspend | | `POST` | `/tenants/:id/reactivate` | Reactivate | | `DELETE` | `/tenants/:id` | Soft-delete | | `GET/PUT` | `/tenants/:id/quota` | Get/set quota | | `GET` | `/tenants/:id/usage` | Usage | | `POST/GET/DELETE/PATCH` | `/tenants/:id/members[/:pid]` | Membership CRUD | | `POST/GET/DELETE` | `/tenants/:id/sharing[/:aid]` | Sharing agreements | | `GET/PUT` | `/tenants/:id/config/search` | Search config | ### `/api/v1/tenants` — control-plane | Method | Endpoint | Description | |---|---|---| | `POST` | `/api/v1/tenants` | Provision with inline quota | | `GET` | `/api/v1/tenants` | List with usage | | `GET` | `/api/v1/tenants/:id` | Config + usage | | `POST` | `/api/v1/tenants/:id/suspend` | Suspend (402 on queries) | | `POST` | `/api/v1/tenants/:id/resume` | Resume | | `DELETE` | `/api/v1/tenants/:id` | Hard purge (removes rows) | | `GET` | `/api/v1/tenants/:id/usage` | Billing snapshot | | `GET` | `/api/v1/tenants/usage/summary` | Cross-tenant summary | ### `/platform/tenants` — platform admin | Method | Endpoint | Description | |---|---|---| | `GET` | `/platform/tenants` | List all | | `GET` | `/platform/tenants/:id` | Detail | | `PATCH` | `/platform/tenants/:id/tier` | Assign tier | | `POST` | `/platform/tenants/:id/suspend` | Suspend | | `POST` | `/platform/tenants/:id/reactivate` | Reactivate | | `DELETE` | `/platform/tenants/:id` | Tombstone | | `GET` | `/platform/license` | License status | | `GET` | `/platform/usage` | Usage summary | ## Audit & Provenance | Method | Endpoint | Description | |---|---|---| | `GET` | `/audit/count` | Entry count | | `GET` | `/audit/entries` | List entries | | `GET` | `/audit/proof` | Merkle proof | | `GET` | `/audit/log` | Raw log | | `POST` | `/api/v1/audit/export` | Export (control-plane) | ## Health & System | Method | Endpoint | Description | |---|---|---| | `GET` | `/health` | Health check | | `GET` | `/health/live` | Liveness | | `GET` | `/health/ready` | Readiness | | `GET` | `/version` | Version | | `GET` | `/status` | Detailed status | | `GET` | `/metrics` | Prometheus metrics | | `GET` | `/config` | Server config | | `GET` | `/openapi.json` | OpenAPI spec | ## Admin | Method | Endpoint | Description | |---|---|---| | `POST` | `/admin/backup` | Trigger backup | | `GET` | `/admin/backups` | List backups | | `POST` | `/admin/restore` | Restore | | `POST` | `/admin/compact` | Compact storage | | `POST` | `/admin/reindex` | Reindex a type | | `POST` | `/admin/rotate-dek` | Rotate encryption key | | `GET` | `/admin/system` | System info | | `GET` | `/admin/dashboard` | Admin dashboard | ## Cluster | Method | Endpoint | Description | |---|---|---| | `GET` | `/cluster/nodes` | List nodes | | `GET` | `/cluster/topology` | Topology view | | `POST` | `/cluster/rebalance` | Rebalance | | `POST` | `/cluster/drain/:node` | Drain a node | ## See also - [Tenant management](/docs/guides/multi-tenancy) — CLI + onboarding guide - [SDKs](/docs/sdks/overview) — Python / TypeScript / Go clients - [Error codes](/docs/reference/error-codes) — HTTP error reference ============================================================================== # Bi-temporal query reference URL: https://relatadb.dev/docs/reference/bitemporal ============================================================================== # Bi-temporal query reference Relata is bi-temporal by default. Every row and every edge carries **four timestamps** — two for the real-world validity period, two for the database's knowledge period. This page covers the SQL surface, the index path, and the common query patterns. ## The four timestamps | Field | Type | Meaning | |---|---|---| | `valid_from` | `i64` ns UTC | When the row became true in the real world | | `valid_to` | `i64` ns UTC | When the row stopped being true (or `i64::MAX` for "still true") | | `system_from` | `i64` ns UTC | When the database first knew about the row | | `system_to` | `i64` ns UTC | When the database stopped believing the row (or `i64::MAX`) | The pair `(valid_from, valid_to)` is the **valid time** axis (business reality). The pair `(system_from, system_to)` is the **system time** axis (database knowledge). The four together give a 2-D point in time-travel space. The pair is half-open: `[valid_from, valid_to)` — the row is visible at `valid_from` inclusive, invisible at `valid_to` exclusive. ## SQL surface ### `AS OF` — time travel Point-in-time query on either axis. The bare form `AS OF ''` selects the **valid-time** axis; `AS OF SYSTEM TIME ''` selects the **system-time** axis; `AS OF CURRENT` is sugar for both axes at `NOW`. ```sql -- What did we know about Alice as of 2024-06-01 (system time)? SELECT * FROM Person AS OF SYSTEM TIME '2024-06-01T00:00:00Z' WHERE id = 'p1' -- What was true in the real world as of 2024-06-01 (valid time)? SELECT * FROM Person AS OF '2024-06-01T00:00:00Z' WHERE id = 'p1' ``` A single `AS OF` clause addresses one axis at a time — Relata does not accept two consecutive `AS OF` clauses in one statement. To combine valid- and system-time filters, write the predicates out explicitly (e.g. `WHERE valid_from <= ts AND system_from <= ts`). ### `WITH PROVENANCE` — show the source ```sql SELECT id, name FROM Person WHERE id = 'p1' WITH PROVENANCE ``` Returns the standard columns plus a parallel `provenance` array (one entry per row) carrying `source` (the ingest batch / API call that produced the row), `method`, `confidence`, `recorded_at`, and `derived_from` (hex `ProvenanceRef` of the source row this one was derived from, or `null` for genesis). ### `EXPLAIN_REPLAY` — re-derive an exhibit seal ```sql EXPLAIN_REPLAY('', SEQ => ) ``` Re-derives a logged exhibit link's seal byte-identically. To diff two points in time, query the audit log or compare two `AS OF` snapshots. ## How `AS OF` works The bi-temporal model is enforced by `BiTemporalRange` on every row. The executor's `scan_as_of` filters in-memory by `visible_as_of(&r.temporal, valid_t, system_t)`. For spilled (on-disk) segments, zone maps and per-column bloom filters prune segments pre-decode — segments whose `valid_from` range doesn't overlap the queried timestamp are skipped entirely. ## Common patterns ### "What did we know at time T?" ```sql SELECT * FROM Person AS OF SYSTEM TIME '2024-06-01T00:00:00Z' ``` Equivalent to: `system_from <= T AND system_to > T`. ### "What was true at time T?" ```sql SELECT * FROM Person AS OF '2024-06-01T00:00:00Z' ``` Equivalent to: `valid_from <= T AND valid_to > T`. ### "When did Alice's email change?" ```sql SELECT valid_from, email FROM Person AS OF CURRENT WHERE id = 'p1' ORDER BY valid_from ``` `AS OF CURRENT` returns every version the database currently knows about (all rows whose `system_to = i64::MAX`); it is sugar for `AS OF ` on both axes. ### "Show me the history of Alice" ```sql SELECT valid_from, valid_to, system_from, email FROM Person WHERE id = 'p1' ORDER BY system_from ``` Without `AS OF`, every version is returned (including superseded ones). This is the full audit trail for the row. ### "Restore Alice's email as it was on June 1" ```sql -- Read the system-time value at T: SELECT email FROM Person AS OF SYSTEM TIME '2024-06-01T00:00:00Z' WHERE id = 'p1' -- Write it back as a new version (the old version is not modified — bi-temporal -- is append-only): INSERT INTO Person (_pk, id, email, valid_from) VALUES ('p1-v2', 'p1', '', '2024-07-01T00:00:00Z') ``` ### "What changed in the last hour?" ```sql SELECT id, system_from FROM Person WHERE system_from > NOW() - INTERVAL '1 hour' ORDER BY system_from DESC ``` Or use the audit log directly: ```bash curl -H "Authorization: Bearer $RELATA_TOKEN" \ "http://localhost:9090/audit/entries?since=3600&object_type=Person" | jq ``` ### Temporal graph queries (edges are bi-temporal too) ```sql -- Who did Alice know? (PATHS_BETWEEN accepts from_id, to_id, MAX_HOPS => n only — -- to filter by time, scope the underlying edge rows via their valid_from/system_from.) PURPOSE 'investigation' SELECT * FROM PATHS_BETWEEN('p1', '?', MAX_HOPS => 3) ``` Edges carry the same four timestamps as rows, so an `AS OF` snapshot of the link types feeds the path expansion; `PATHS_BETWEEN` itself does not accept an `AS_OF` parameter. ## Indexes The bi-temporal index is the **range index** (`BTreeMap>`) on `(valid_from, system_from)`. Range queries on either axis use this index. Equality on `id` uses the live-index (`HashMap`) for O(1) point lookup at the latest version. ## Performance notes - `AS OF CURRENT` is the fastest path — same as a regular scan with `system_to = i64::MAX`. - `AS OF ''` walks the range index; on spilled segments, the zone map prunes non-overlapping segments. - For temporal joins (`Person AS OF T1 JOIN CdrRecord AS OF T2`), use the same timestamp for both axes to avoid pinning two different snapshots. ## Bi-temporal model gotchas ### `valid_to = i64::MAX` means "still true" Don't write `WHERE valid_to IS NULL` — there is no null. Use `WHERE valid_to = 9223372036854775807` or use `AS OF CURRENT` which is the intended API. ### Updates are append-only Updating a row closes the old version (sets `system_to = NOW`) and inserts a new version (`system_from = NOW`). The old version is preserved for audit. This is why a single `id` can have many rows in the store. ### Deletes are soft `DELETE` sets `valid_to = NOW` on the matching rows; the rows themselves stay in the store. Hard deletes (`FORGET`) are a separate GDPR-style operation (`POST /memory/forget/:id` or `DELETE /types/:name/:id?hard=true`). ### Compaction preserves history Compaction merges adjacent versions where safe but never drops a version that still has `system_to = i64::MAX` (live data) or that falls inside any configured retention window. ## See also - [Query cookbook](/docs/reference/query-cookbook) - [SQL reference](/docs/reference/sql) - [Agent memory](/docs/reference/agent-memory) ============================================================================== # Branching & Namespaces URL: https://relatadb.dev/docs/reference/branching ============================================================================== # Branching & Namespaces "Branching" in Relata refers to **three related but distinct surfaces** that the rest of the docs touch separately. This page unifies them under one vocabulary so the three API names (which are easy to conflate) are unambiguous. | Surface | What it forks | Granularity | Speed | API | |---|---|---|---|---| | **Namespace branch** | An entire data namespace — every type and every row | Whole namespace | **O(types), constant time** (copy-on-write; no row copy) | `BRANCH FROM ` (SQL) · `POST /v1/namespaces/{name}/branch` (HTTP) | | **Schema branch** | The ontology/schema only (git-branched ontology) | Schema only | O(types) | `POST /schema/branches/{name}` (HTTP) | | **Sub-tenant namespace** | A hierarchical path *within* a tenant | Per-row `NamespacePath` field | Enforced on read/write | `X-Organization-Id: acme/eu/hr` + query predicates | > **The naming collision, resolved.** `BRANCH ... FROM ...` and `POST /v1/namespaces/{name}/branch` fork the **whole data namespace** (types + rows) — they are the same operation exposed over two doors. `POST /schema/branches/{name}` forks the **schema/ontology only**. They are *not* interchangeable. ## Namespace branching (data fork) Relata can fork an entire namespace — every registered type and every row — into a new named branch in **constant time**, regardless of how much data the source branch holds. Writes on either branch are isolated: the fork shares its parent's existing data via copy-on-write, and new writes on each branch go to that branch's own active segment. This is the same primitive that backs `POST /schema/branches/:name` for the schema-only case. A namespace fork is `O(types)`, not `O(rows)`. ### SQL ```sql BRANCH dev FROM main ``` `BRANCH FROM ` requires no `PURPOSE` clause — like `DELETE` and `UPSERT`, it is namespace management, not a governed data read/write. `` and `` accept a bare identifier or a quoted string. ### HTTP ```bash curl -X POST http://localhost:9090/v1/namespaces/dev/branch \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \ -H "Content-Type: application/json" \ -d '{"branch_from": "main"}' ``` ```json { "created": true, "branch": "dev", "source": "main" } ``` Returns `409 Conflict` if `dev` already exists, `404 Not Found` if `main` (the `branch_from` source) does not exist, and `400 Bad Request` if the branch name is empty or over 128 characters. ### Listing and deleting branches Branches created either way are ordinary Relata branches: they show up in `GET /schema/branches`, and `DELETE /schema/branches/:name` removes them. `main` cannot be deleted. ```bash curl http://localhost:9090/schema/branches \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" curl -X DELETE http://localhost:9090/schema/branches/dev \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" ``` ### How the fork works (constant time) - Creating a branch wraps every existing row in the source branch's tables in a shared, immutable `Arc` slab — **no row is copied**. A 10-row namespace and a 10-billion-row namespace fork in the same amount of time. - A write to either the source or the new branch after the fork goes into that branch's own fresh active segment. The parent's data is never mutated, so reads on one branch never see writes made on the other. - Branch-from-a-branch (`BRANCH c FROM b` where `b` was itself forked from `a`) works the same way, any number of levels deep. - Deleting a branch frees its tables and evicts its partition-lock entries. ### Canonical use-cases 1. **Per-developer sandbox** — give every developer (or every VCS feature branch) an isolated, full copy of production data, without an O(rows) copy: `BRANCH alice-feature-142 FROM main`. Discard with `DELETE /schema/branches/alice-feature-142`. 2. **CI test pipeline** — fork a fresh branch per test run from a known-good `fixtures` branch, run the suite's writes against it, discard it — a clean, isolated dataset per run without restoring a snapshot from disk. 3. **Point-in-time snapshot** — `BRANCH pre-migration-2026-08-01 FROM main` immediately before a risky bulk migration or schema change, for an instant rollback target that needs no WAL replay or backup restore. 4. **Codebase / RAG indexing** — run multi-pass indexing jobs against a scratch branch so partial or failed runs never corrupt the namespace other consumers read from. ## Schema branches (git-branched ontology) A schema branch forks the **ontology only** — develop schema changes (new types, state-machine constraints, computed columns) on a branch without affecting production, then merge or discard. ```bash # Create a schema branch curl -X POST http://localhost:9090/schema/branches/dev-schema \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \ -d '{"from": "main"}' # ... make schema changes on the branch, merge when ready ... # Delete if discarded curl -X DELETE http://localhost:9090/schema/branches/dev-schema \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" ``` Note the body key is `"from"` here (not `"branch_from"` as on the namespace-fork door) — these are two different endpoints. See [Ontology & Schema](/docs/concepts/ontology). ## Sub-tenant namespaces A `NamespacePath` partitions data *within* a tenant — for example, separating departments or cases inside one organisation — expressed as `/`-separated paths on the tenant identity: ``` acme ← top-level tenant acme/eu ← regional sub-tenant acme/eu/hr ← team sub-tenant (acme/eu sees acme/eu AND acme/eu/hr) acme/us ← separate regional sub-tenant (does NOT see acme/eu) ``` ```bash # Write into a sub-tenant curl -X POST "http://127.0.0.1:9090/ingest?object_type=Employee&purpose=hr" \ -H "Content-Type: text/csv" \ -H "X-Organization-Id: acme/eu/hr" \ --data-binary $'name\nBob' ``` > **Honest status:** sub-tenant namespace **enforcement is partially wired** — the `Row.namespace_path` field exists but is not universally populated on the write path today. Do not rely on namespace filtering as a hard security boundary yet; use **tenant-level** (`X-Organization-Id`) isolation for hard boundaries. See [Multi-Tenancy](/docs/guides/multi-tenancy) for the current state. ## Cache pinning for namespaces A namespace can be **pinned** in the cache tier (`RELATA_PINNED_NAMESPACES`) so it reserves a dedicated NVMe slice and is evict-immune under pressure. A branch of a pinned namespace starts unpinned (exact-match membership, not prefix). See [Environment Variables](/docs/reference/env-vars). ## See also - [HTTP API Reference](/docs/reference/api-reference) — full endpoint list - [SQL Reference](/docs/reference/sql) — `BRANCH` statement grammar and the rest of the dialect - [Ontology & Schema](/docs/concepts/ontology) — schema branches and online schema evolution - [Multi-Tenancy](/docs/guides/multi-tenancy) — tenant-level isolation ============================================================================== # Cypher & SQL-PGQ Graph Queries URL: https://relatadb.dev/docs/reference/cypher ============================================================================== # Cypher & SQL-PGQ Graph Queries RelataDB supports three graph query languages over the same underlying CSR adjacency engine. ## Cypher (Neo4j-compatible) Any SQL query starting with `MATCH` is auto-detected and translated to SQL before execution. No separate endpoint is needed. ```sql MATCH (n:Person {id: 'p1'}) RETURN n.name, n.email ``` Multi-hop traversal: ```sql MATCH (a:Person)-[:KNOWS]->(b:Person) WHERE a.id = 'p1' RETURN b.name ``` ### Supported Cypher clauses | Clause | Status | |---|---| | `MATCH` / `OPTIONAL MATCH` | ✅ | | `WHERE` | ✅ | | `RETURN` | ✅ | | `RETURN DISTINCT` / `ORDER BY` / `SKIP` / `LIMIT` | ✅ (labelled-node MATCH) | | `UNION` / `UNION ALL` | ✅ | | `CALL traverse.*` / `CALL gds.*` | ✅ | | `CREATE` / `MERGE` (writes) | ✅ via governed write door | Connect via the **Bolt protocol** on port 7687 (default) using the official Neo4j Python/Java/Go drivers or `cypher-shell`. ## SQL/PGQ SQL/PGQ graph patterns inside standard SQL `SELECT`: ```sql SELECT p1.name, p2.name FROM MATCH (p1:Person) -[e:KNOWS]-> (p2:Person) ON graph_schema WHERE p1.id = 'p1' ``` ## Graph SQL operators RelataDB provides 10+ graph operators as SQL table-valued functions: ```sql -- Shortest path SELECT * FROM SHORTEST_PATH('Person', 'p1', 'p2', 'KNOWS') -- Traverse with depth limit SELECT * FROM TRAVERSE('Person', 'p1', 'KNOWS', 3) -- Degree centrality SELECT id, DEGREE(id, 'out') AS out_degree FROM Person -- Connected components SELECT * FROM WEAKLY_CONNECTED('Person', 'KNOWS') -- PageRank SELECT id, pagerank FROM PAGERANK('Person', 'KNOWS') ``` ## GQL (ISO/IEC 39075) The ISO GQL surface is reachable on `POST /query` with the `x-query-dialect: gql` header (#3265). GQL is header-selected only — without the header, a `MATCH` body is auto-detected as Cypher, so the two grammars are never silently confused. ```bash curl -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -H "x-query-dialect: gql" \ -d '{"purpose":"analytics","sql":"MATCH (n:Person) WHERE n.age > 35 RETURN n.name ORDER BY n.age DESC LIMIT 5"}' \ http://localhost:9090/query ``` Supported subset: single-node and edge `MATCH`, `WHERE`/`FILTER`, `OPTIONAL MATCH`, `RETURN` with `DISTINCT`/`ORDER BY`/`LIMIT`/`SKIP`, `UNION [ALL]`, and `CALL` procedures. Deferred constructs (writes, quantified paths, path modes, `SHORTEST`/`ANY`/`ALL` prefixes) return a typed error — GQL-status `42G04` (syntax, HTTP 400) or `0A501` (feature not supported, HTTP 501) — never a silent mis-translation. All SDKs expose the dialect: `client.query(stmt, dialect="gql")` (Python), `relata.query(stmt, { dialect: "gql" })` (TypeScript), `client.Query(ctx, stmt, relata.WithDialect("gql"))` (Go). ## See also - [SQL reference](/docs/reference/sql) — full SQL grammar - [GraphQL](/docs/reference/graphql) — the GraphQL query door - [Protocol compatibility](/docs/reference/protocols) — Neo4j/Bolt/Cypher door ============================================================================== # Environment Variables URL: https://relatadb.dev/docs/reference/env-vars ============================================================================== # Environment Variables > **Strict parsing.** Malformed values **FATAL at startup** — Relata > refuses to boot rather than silently using a wrong default. Removed/renamed > vars (e.g. `RELATA_ORG_MODE`, `RELATA_REQUIRE_ORG`, `RELATA_ALLOWED_ORIGINS`, > `RELATA_REQUIRE_MTLS`, `RELATA_MAX_CONNECTIONS`) also FATAL — run > `relata config --migrate` when upgrading from 1.x. > **Config precedence:** `env > file > built-in default`. File search order: > (1) `--config <path>` flag, (2) `RELATA_CONFIG` env var, (3) `./relata.toml`, > (4) `~/.relata/relata.toml`. Run `relata config --print-template` for a starter. ## Core server | Variable | Default | Description | |---|---|---| | `RELATA_PROFILE` | `free` | Deployment profile: `free` (dev/CI — storage-capped, otherwise identical posture to licensed tiers), `server` (single-node prod), `cluster` (multi-node). `lite` was a legacy alias for `free` and is now rejected. **Startup fails (FATAL) on `lite` or any other value.** Auth/TLS/diagnostics posture no longer varies by profile — see `RELATA_BEARER_TOKEN`, `RELATA_OPEN_DEV_ALLOWED`, `RELATA_PLAINTEXT_OK` below. | | `RELATA_PORT` | `9090` | HTTP API port. | | `RELATA_HOST` | — | Advertised host name (used for the A2A Agent Card base URL when `RELATA_A2A_BASE_URL` is unset). Falls back to the bind address. | | `RELATA_HTTP_BIND` | profile: `127.0.0.1` (free) / `0.0.0.0` (server/cluster) | HTTP bind address for the **data plane** (`/query`, `/ingest`, doors, etc.). This listener no longer serves `/admin/*` or `/platform/*` at all — see `RELATA_ADMIN_BIND` below. Also the input to `is_loopback_bind()` (`127.0.0.1`/`[::1]` prefix, or unset), which gates `RELATA_OPEN_DEV_ALLOWED` and the `RELATA_PLAINTEXT_OK` default uniformly across every profile. | | `RELATA_ADMIN_BIND` | `127.0.0.1:9091` | Bind address for a SECOND, dedicated `TcpListener` that exclusively serves `/admin/*` and `/platform/*` on a loopback-only Zero-Trust control plane. These routes are not mounted on the data-plane listener (`RELATA_HTTP_BIND`) at all — a request to them there gets a plain `404` (the route doesn't exist), not `401`/`403`, so a remote peer cannot even confirm an admin surface exists. **Must resolve to a loopback address (`127.0.0.0/8` or `::1`) — startup fails (FATAL) otherwise**; this is deliberate (loopback is enforced by TCP, not by an HTTP header check), not a bug — widening it would silently reintroduce a network-reachable admin token. Reach it via `kubectl port-forward svc/relata 9091:9091` or an mTLS control-plane sidecar in the same pod/network namespace; never expose it directly. | | `RELATA_BEARER_TOKEN` | — | `Authorization: Bearer` token for all auth-gated endpoints. **Mandatory on `server`/`cluster` (startup fails if unset).** On the `free` profile a dev token may be minted on first run and printed to stderr — set it persistently to reuse it. pgwire is disabled when unset. With no `RELATA_BEARER_TOKEN`/`RELATA_ADMIN_TOKEN` and no `RELATA_OPEN_DEV_ALLOWED`, every data/admin/diagnostics endpoint returns 401 on every profile — `cargo run -- serve` no longer silently opens anything. | | `RELATA_ADMIN_TOKEN` | — | Admin-only token for the `/admin/*` + `/platform/*` surface (privilege-separated from `RELATA_BEARER_TOKEN`), gated on the loopbound listener described under `RELATA_ADMIN_BIND`. Gates: `/admin/console` (HTML), `/admin/tokens`, `/admin/backup`, `/admin/backups`, `/admin/restore`, `/admin/compact`, `/admin/rotate-dek`, and every `/platform/*` tenant-lifecycle route. When unset, the entire admin surface returns 401 on `free` (or, on `server`/`cluster`, `503` via `admin_surface_guard`) — unless `RELATA_OPEN_DEV_ALLOWED=true` (see below). Browser access: `GET /admin/console?token=<RELATA_ADMIN_TOKEN>` (the client JS reads `?token=` from the URL). **Env-only by design:** this is a privileged secret and is deliberately *not* persisted to disk or config — it must be re-supplied on every restart. A restart without it leaves `/admin/*` and admin-gated features unprovisioned (a client sees this as a degraded/"stub" mode); startup logs a loud `Admin surface: UNPROVISIONED` warning so the state is never silent. **Note:** there is an open gap where `check_admin_auth` falls back to the bearer token when `RELATA_ADMIN_TOKEN` is unset — set both tokens explicitly in production until this is resolved. | | `RELATA_OPEN_DEV_ALLOWED` | `false` | Explicit, uniform-across-every-profile opt-in for the unauthenticated local-dev convenience that used to be a silent `RELATA_PROFILE=free` special case. When `true`, AND no `RELATA_BEARER_TOKEN`/`RELATA_ADMIN_TOKEN`/registered token is configured, AND `RELATA_AUTH_MODE` resolves to `none` (unauthenticated), AND the relevant listener is bound to loopback (data plane: `RELATA_HTTP_BIND`; admin surface: `RELATA_ADMIN_BIND` — checked fresh per request, never by profile), every data endpoint (`check_auth_with_registry`), the admin surface (`check_static_admin_auth`), and the diagnostics surface (`diagnostics_auth_ok`) are open with no token. Fail-closed on every axis: unset, a configured credential, a non-`None` auth mode, or a non-loopback bind all keep the gate closed regardless of profile. **Do not set this outside local development.** | | `RELATA_PLAINTEXT_OK` | unset (defaults to `true` only on a loopback bind) | Uniform across every profile, including free. Set `true`/`false` explicitly to allow/require TLS termination in front of a plaintext HTTP listener when `RELATA_TLS_CERT`/`RELATA_TLS_KEY` are not configured. If unset, the effective default is `true` when the bind is loopback (`RELATA_HTTP_BIND` unset or `127.0.0.1`/`[::1]`) and `false` (TLS required, startup FATALs) otherwise — the same loopback predicate `RELATA_OPEN_DEV_ALLOWED` uses. There is no longer a `free`-only unconditional plaintext pass: a `free`-profile server bound to `0.0.0.0` needs `RELATA_TLS_CERT`/`RELATA_TLS_KEY` or an explicit `RELATA_PLAINTEXT_OK=true` exactly like `server`/`cluster`. | | `RELATA_DATA_DIR` | `./data/relata` | Root directory for config files and WAL state. | | `RELATA_CONFIG_DIR` | — | Override directory for TOML config files (takes priority over `RELATA_DATA_DIR`). | | `RELATA_CONFIG` | — | Inline TOML config blob; takes priority over config files. | | `RELATA_PUBLIC_URL` | — | Externally-visible base URL (used for CORS `Allow-Origin` and link generation). | | `RELATA_URL` | `http://127.0.0.1:9090` | Base URL the CLI client uses to reach a running `relata serve` — covers `relata import`, `relata status`, `relata query`, jobs/workflows subcommands, and cluster-admin calls. Falls back to `http://127.0.0.1:{RELATA_PORT}`. (`RELATA_STATUS_URL` is a deprecated alias — see the Deprecated section.) | | `RELATA_A2A_BASE_URL` | — | Explicit base URL advertised in the A2A Agent Card (`.well-known/agent.json`). Overrides the `RELATA_HOST`/`RELATA_PORT` derivation. | | `RELATA_DRAIN_TIMEOUT_SECS` | `30` | Graceful-shutdown drain timeout. On SIGTERM/Ctrl-C the server first waits (up to this bound) for the ingest queue to reach empty so every **acknowledged** row is flushed to the WAL before exit. The outcome is observable: an info log `ingest drain complete — N rows persisted` on success, or a LOUD error `ingest drain INCOMPLETE — acked rows may be unpersisted` plus a **non-zero exit** if the timeout fires with acked rows still queued (fail-closed, so an orchestrator never treats an acked-data-loss shutdown as clean). | | `RELATA_DEMO_MODE` | — | Set `true` (or `1`/`yes`/`on`) to enable demo-mode restrictions (read-only ingest, synthetic data). Case-insensitive. **Startup fails on unrecognised values.** | | `RELATA_SHELL_PURPOSE` | `shell` | Default purpose for the `relata shell` interactive REPL. | > `lite` was a silent legacy alias for `free`; it is now rejected outright — startup fails (FATAL) if `RELATA_PROFILE=lite` is set. --- ## Storage Relata selects a backend in priority order: in-memory opt-in → S3 → local disk (default). | Variable | Default | Description | |---|---|---| | `RELATA_IN_MEMORY` | — | Set `true` (or `1`/`yes`/`on`) to run fully in-memory (dev/CI only — **data lost on restart**). Case-insensitive. **Startup fails on unrecognised values.** | | `RELATA_LOCAL_DATA_DIR` | — | Explicit local-disk path for the object store. When unset, falls back to `RELATA_DATA_DIR/objects`. | | `AWS_ENDPOINT_URL` | — | S3-compatible endpoint (AWS S3, R2, GCS, self-hosted). When set, S3 takes priority over local disk. | | `AWS_S3_BUCKET` | `relata` | S3 bucket name. | | `AWS_ACCESS_KEY_ID` | — | S3 access key. | | `AWS_SECRET_ACCESS_KEY` | — | S3 secret key. | | `AWS_REGION` | `us-east-1` | AWS region (S3 path, KMS). | | `RELATA_DURABILITY` | `s3` | Per-backend WAL recovery posture: `s3` (strong), `r2` (eventual on failure), `s3compat`, `azure`. | | `RELATA_ALLOW_HTTP_OBJECT_STORE` | — | Set `true` to silence the plaintext-`http://` object-store warning (local S3-compatible dev). Credentials/data travel unencrypted. | | `RELATA_STORE_MAX_INDEX_MB` | — (adaptive) | Cap on per-store index RAM in MiB. When unset, derived from detected RAM (index ≈11% of the ~75% pool). `0` = unbounded. | | `RELATA_TEMPORAL_INDEX_MAX_ENTRIES` | `4000000` | Cap on `Table`'s per-entity bi-temporal version-chain index (`version_index`), in entries per object type. Past the cap, the index stops growing and is marked incomplete — `AS OF` reads fall back to an exact full scan for that type rather than trusting a truncated index. | | `RELATA_WAL_SYNC` | `interval` | Process-global WAL fsync mode. `always` (fsync on every flush boundary, RPO≈0) \| `interval` (default — batched fsync every ~10 ms, RPO≈10 ms) \| `off` (page-cache only — faster, crash-but-not-power-loss-durable; `batch` and `os` are accepted as aliases of `off`). **Any other value (including `true`/`on`/`1`) is a FATAL startup error** — use the exact vocabulary above. For a **per-write** RPO=0 knob use the `X-Relata-Durability: sync` request header instead of a deployment-wide switch (see [guarantees → durability levels](/docs/architecture/storage)). | | `RELATA_WAL_STRICT` | — | Set `true` to escalate mid-stream WAL corruption to a hard startup error instead of dropping the corrupted suffix. When unset, recovery drops the unparseable tail and increments `relata_wal_records_dropped_total`. | | `RELATA_PERSIST` | — | Legacy persistence flag (still read; prefer `RELATA_LOCAL_DATA_DIR`). ⚠ still read at `crates/relata-cli/src/serve.rs:16107` — removal tracked separately. | | `RELATA_COMPACTION_TARGET_SEGMENTS` | — | Target segment count before WAL compaction triggers. | | `RELATA_COMPACTION_MIN_AGE_SECS` | — | Minimum segment age before compaction considers it. | | `RELATA_COMPACTION_GC_GRACE_SECS` | — | GC grace period after compaction before old segments are deleted. | | `RELATA_COMPACT_STRATEGY` | `size-tiered` | Compaction segment-selection strategy. `size-tiered` uses all segments for the target type (default). `leveled` compacts the first level of up to 10 segments. `time-window` compacts only segments sharing the earliest `partition_date` bucket. | | `RELATA_COMPACT_MAX_PARALLEL` | `4` | Maximum compaction parallelism: bounds *both* how many object-store GET+decode pipelines run concurrently within one type's compaction, and how many types the auto-compaction scheduler / `/admin/compact` compact concurrently. Higher values reduce wall-clock time at the cost of additional network/CPU concurrency. | | `RELATA_BENCH_NO_SAVE` | — | Set `1` in `relata-bench` to skip auto-saving JSON results to `docs/benchmarks/results/<git-sha>/`. Best-effort: a missing git binary or unwritable path just prints a warning. | | `RELATA_BENCH_STRICT` | — | Set any non-empty value in `relata-bench` to enable server-class (strict) gate thresholds: B-INGEST ≥350K rows/s, B-SCAN p99 ≤10ms, B-FILTER p50 ≤7ms, G-VECTOR-INGEST ≥3000 vecs/s. Without this, loose thresholds accommodate thermal throttle on developer hardware. | | `RELATA_OBJECT_STORE` | — | Object-store URL selecting a **native** cloud backend ahead of the S3-compatible path: `gcs://bucket/prefix` or `azure://container/prefix` (`crates/relata-storage/src/remote.rs`). When unset, Relata uses the `AWS_ENDPOINT_URL` S3-compatible path. | | `RELATA_SPILL_FORMAT` | `sstable` | On-disk format for cold spill segments (`crates/relata-storage/src/sstable.rs`). `sstable` (default) writes LSM-style sorted blocks with a binary-searchable sparse index; `json` keeps the legacy newline-delimited text format. | | `RELATA_WAL_FORMAT` | `binary` | WAL payload encoding (`crates/relata-storage/src/wal.rs`). `binary` = compact `0xB1`-magic layout (default); `json` = legacy/interop human-readable. Any non-`json` value selects `binary`. | | `RELATA_WAL_IO` | `tokio` | WAL writer I/O backend (`crates/relata-storage/src/async_io.rs`). `tokio` (default) \| `direct` (`O_DIRECT`, unbuffered). Any other value — including `io_uring`, which is recognised syntax but has no working backend yet — exits the process at startup with a FATAL message; it is never silently accepted then failed at use time. | | `RELATA_WAL_ARCHIVE_DIR` | — | Directory where rotated WAL segments are archived for point-in-time recovery / disaster recovery (`crates/relata-storage/src/backup.rs`). When unset, WAL archiving returns `NotConfigured` and rotated segments are recycled. | | `RELATA_HYDRATE_MAX_PARALLEL` | `8` | Maximum number of concurrent remote-segment fetches during lazy hydration on restart (`crates/relata-storage/src/store/remote_io.rs`). Clamped to ≥ 1. Raise to warm the cache faster on high-bandwidth object-store links. | --- ## Memory and cache > **Adaptive defaults.** When the budget vars below are **unset**, > Relata derives them from detected hardware at startup — one shared ~75%-of-RAM > pool split store ≈34% / index ≈11% / cache ≈11% / vectors ≈19%, plus > `RELATA_EMBED_CONCURRENCY` = cores and `RELATA_SPECULATE_MAX_CONCURRENT` = > cores/4. `free` clamps the pool to the 10 GB ceiling. Explicit env vars > always win; if the RAM probe fails the legacy flat constants apply. See > [`capacity-planning`](/docs/guides/scaling) and > `crates/relata-cli/src/resource.rs`. | Variable | Default | Description | |---|---|---| | `RELATA_STORE_MAX_RAM_MB` | — (adaptive) | Global RAM budget cap across all stores. When unset, derived from detected RAM (store ≈34% of the ~75% pool); on `server`/`cluster` the legacy fallback is 1024 MB. Crossing the cap no longer spills synchronously on the inserting thread: an insert only flips an in-memory pressure flag, and a supervised background task (`serve::maintenance::background_spiller_task`, polling every 200 ms) performs the actual disk write + `fdatasync` off the hot path. | | `RELATA_SESSION_DRAFT_TTL_HOURS` | `24` | How long a session write-buffer (draft) is retained before automatic eviction. Set lower in memory-constrained environments. See `X-Session-Draft` header. | | `RELATA_VECTOR_RAM_BUDGET_MB` | `64` | Vector index (DiskANN) RAM budget. | | `RELATA_GRAPH_RAM_BUDGET_MB` | `64` | Graph (CSR) in-memory budget. When set, `GRAPH_*` ops build their adjacency into a disk-paged CSR, streaming edges block-by-block so peak build RAM is O(page budget), not O(edges). | | `RELATA_GRAPH_RESIDENT_EDGE_WARN` | `10000000` | Edge-count at which a **resident** (whole-RAM) graph build logs a `warn!` that it may OOM without paging — a nudge to set `RELATA_GRAPH_RAM_BUDGET_MB`. `0` disables the warning. | | `RELATA_GRAPH_DELTA_LOG_CAP` | `50000` | Max entries retained in `LinkStore`'s bounded edge-delta log (`crates/relata-graph/src/link_store.rs`). Once exceeded, the oldest entries are dropped, so a cached CSR older than the oldest retained generation can no longer be reconciled via `edge_deltas_since` and falls back to a full rebuild. | | `RELATA_GRAPH_CSR_DELTA_REBUILD_RATIO` | `0.25` | Delta-size ÷ graph-edge-count ratio above which `LinkStore`'s CSR-cache reconciliation prefers a full rebuild over `CsrGraph::apply_delta` — mirrors the tombstone-ratio-triggers-compaction convention in `relata_storage::vector::COMPACT_TOMBSTONE_RATIO`. | | `RELATA_GRAPH_CSR_COUNTING_SORT_MAX_SPARSITY` | `8.0` | Max `node_count ÷ edge_count` ratio at which `build_pair` (CSR/CSC construction) still prefers an O(V + E) counting sort over a comparison sort; past this ratio (and above a 1024-node floor) the histogram's `O(V)` cost dwarfs the actual edge work, so a comparison sort is used instead. | | `RELATA_IDENTITY_RAM_BUDGET_MB` | `64` | Identity index RAM budget. | | `RELATA_EXEC_RAM_BUDGET_MB` | — | Query execution working-memory cap. | | `RELATA_JOIN_STRATEGY` | `auto` | Join algorithm selection: `hash` (always hash join), `sort-merge` (always sort-merge join O((n+m) log n)), or `auto` (hash for small datasets, sort-merge when both sides exceed 100 000 rows). | | `RELATA_DISKANN_MAX_RESIDENT` | — (adaptive) | Max DiskANN pages to keep resident. When unset, derived from detected RAM (vectors ≈19% of the ~75% pool, ≈3.5 KiB/vector). | | `RELATA_BLOOM_COLUMNS` | `tenant_id,object_type` | Comma-separated column names for which per-column Bloom filters are built at segment flush time. Set to `none` to disable. Columns missing from a segment fail open (the segment is never falsely pruned). Both the segment-key bloom and per-column blooms are consulted live on the scan path (equality predicates prune disk segments pre-decode — `executor.rs` bloom hint → `scan_as_of_arc_with_bloom` in `scan.rs`). | | `RELATA_SKETCH_COLUMNS` | `tenant_id,object_type,status` | Comma-separated column names for which HyperLogLog (NDV) and Count-Min Sketch (value frequency) are maintained in-memory and updated on every row insert. Used by the cost-based optimizer to replace hardcoded 0.20 selectivity constants with data-driven per-value estimates. Set to `none` to disable; cold-start or disabled columns fall back to the NDV-sample heuristic. Memory: ~96 KB per tracked column per type. | | `RELATA_VECTOR_QUANT` | `full` | Vector quantization tier for HNSW node embeddings. `full` = raw f32 (no compression); `fp16` = IEEE 754 half-precision, 2× memory reduction, ~0.1% error; `binary` = 1-bit per dimension packed into u64 words, 32× memory reduction, used as a hamming pre-filter before exact distance re-ranking. | | `RELATA_VECTOR_COLD_RESIDENT_MAX` | `100000` | Soft cap on RAM-resident vectors in an IVF cold bucket's staging area before the batch spills to the object-store-backed `PagedAnnIndex`. Only pages when an object store is configured. | | `RELATA_MAX_VECTOR_K` | `1000` | Hard cap on a vector query's requested `k`. `0` disables the cap. | | `RELATA_CACHE_IDLE_TTL_SECS` | `3600` | Evict TieredCacheTracker entries idle longer than N seconds. | | `RELATA_CACHE_DECAY_KEEP_FACTOR` | `0.9` | Temperature decay factor applied per decay sweep — entries below `temperature × factor` lose heat. | | `RELATA_CACHE_L2_TO_L1_THRESHOLD` | `4.0` | Temperature threshold above which a segment is promoted from L2 ring-buffer to L1 hot set. | | `RELATA_PINNED_NAMESPACES` | — | Comma-separated list of namespaces (the `branch`/tenant identity segment `CacheCoordinator::partition_key_for` composes into its cache-partition keys) to pin: each reserves a dedicated NVMe cache slice at startup and is evict-immune under cache pressure — eviction always falls back to non-pinned namespaces first. A branch of a pinned namespace starts unpinned (exact-match membership, not prefix-match); each shard of a pinned namespace pins independently. Unknown/malformed tokens (containing the reserved `::` separator) are skipped with a `warn!`, never fail startup. | | `RELATA_PINNED_NVME_SLOTS_PER_NAMESPACE` | `64` | NVMe cache slots reserved per namespace listed in `RELATA_PINNED_NAMESPACES`. Parsed strictly — a non-integer value fails startup. Surfaced per-namespace as the `relata_pinning_utilization` gauge (`tracked_partitions / this value`, clamped to `[0.0, 1.0]`). | | `RELATA_TOMBSTONE_CACHE_MAX_ROWS` | `1024` | Max entries in the per-type tombstone cache before an arbitrary cold entry is evicted. Lower values save RAM at the cost of re-parsing on-disk tombstone files more frequently. | | `RELATA_DECODED_SEGMENT_CACHE_MAX` | `64` | Max decoded disk-segment entries cached in RAM before the oldest-mtime entry is evicted. Lower values save RAM at the cost of re-decoding cold segments. | | `RELATA_ROW_SLOT_MAX` | `256` | Max fields in a type's slotted-row schema dictionary before further rows of that type fall back to the legacy map representation instead of a dense per-row slot array. Guards wide/sparse types (hundreds of optional fields, few set per row) from wasting RAM on a slot array sized to the widest field ever seen. | | `RELATA_PARQUET_MAX_DECOMPRESSED_MB` | `4096` | Hard cap on the decompressed size of a single Parquet segment in MiB. Segments that expand beyond this limit are rejected at load time to prevent decompression-bomb OOM. | | `RELATA_ENRICH_MODE` | `eager` | Identity enrichment mode. `lazy` enqueues enrichment for background processing via `EnrichmentQueue` (lower write latency, slight graph-visibility delay). `eager` enriches inline before the write acks (default, preserves graph consistency). **Startup fails on any other value.** | | `RELATA_MIN_FREE_DISK_MB` | `256` | Minimum free disk space in MiB before Relata sheds inbound writes to protect the WAL and spill segments. `0` disables the guard entirely. | | `RELATA_HTTP_MAX_CONNS` | `4096` | Maximum concurrent in-flight HTTP connections. On the non-TLS path, a `tower::limit::ConcurrencyLimitLayer` back-pressures requests over the limit at the router. On the TLS path, a `tokio::sync::Semaphore` gates connections at accept time before they reach the router. This is the single knob for both paths — do not confuse with per-route rate limiting (`RELATA_RATE_LIMIT_RPS`). | | `RELATA_REDIS_CACHE_MB` | `64` | Redis-protocol-door value cache budget in MiB. `0` disables — every GET falls back to `governed_get` (the zero-regression path). The governed `KvEntry` row remains the single source of truth. | | `RELATA_RESULT_CACHE_ENABLED` | `true` | Enable the 16-shard query result cache. Set to `false` to disable globally (all queries hit the executor). | | `RELATA_RESULT_CACHE_ENTRIES` | `4096` | Historical entry-count knob, kept for API/config compatibility. Each shard is now a foyer S3-FIFO cache admitted purely by byte weight, so this no longer drives a separate structural cap — see `RELATA_RESULT_CACHE_MAX_BYTES`. | | `RELATA_RESULT_CACHE_TTL_SECS` | `300` | Default TTL for cached result sets in seconds. `0` means entries never expire (relying on type-tagged invalidation only). Per-query TTL override (`WITH CACHE TTL <duration>`) is not yet implemented. | | `RELATA_RESULT_CACHE_MAX_BYTES` | `134217728` | Starting total byte budget across all 16 shards (default 128 MiB). This is now the *initial* value only — the `tick()` feedback loop (wired into the 60 s background task) adjusts it ±5% toward `RELATA_RESULT_CACHE_TARGET_HIT_RATIO` at runtime, bounded by a RAM-derived ceiling. | | `RELATA_RESULT_CACHE_MAX_ROWS` | `10000` | Result sets with more rows than this are not cached (prevents pathological cache entries from exhausting the byte budget). | | `RELATA_RESULT_CACHE_PROMOTE_ON_HIT` | `true` | Historical LRU promote-on-hit knob, kept for compatibility. The S3-FIFO backend (foyer) always records access frequency on every `get()` — that's inherent to frequency-based admission — so the old peek-vs-promote distinction from the `lru`-crate backend no longer applies. | | `RELATA_RESULT_CACHE_TARGET_HIT_RATIO` | `0.85` | Target hit ratio the result-cache `tick()` adaptive-sizing loop steers the byte budget toward. Below target → budget grows 5%/tick; at or above target → budget shrinks 5%/tick (reclaiming RAM for other consumers), bounded to `[16 MiB, detected_ram × 11%]`. Clamped to `[0.0, 1.0]`. | | `RELATA_BENCH_CACHE_HIT_GATE_US` | `50` | Hit-latency gate (µs, p99) for the `relata-bench result-cache` suite. Raise on slower shared/cloud hardware where the fixed 50µs ceiling breaches on scheduler jitter rather than real contention. | | `RELATA_VECTOR_RESULT_CACHE_ENABLED` | `true` | Enable the vector KNN query-result cache (`KnnResultCache`, `crates/relata-storage/src/vector_cache.rs`) — memoises `(query_vector, k, filter_hash, tenant_id, model_tag) -> top-k ids/scores` below the full-SQL result cache, so a repeated embedding skips the hot `TurboVec` scan and any cold-tier IVF paging. Set to `false` to disable globally. | | `RELATA_VECTOR_RESULT_CACHE_MB` | `64` | Byte budget for the vector KNN result cache. LRU-evicted when exceeded. | | `RELATA_VECTOR_RESULT_CACHE_TTL_SECS` | `60` | TTL for cached KNN results in seconds. Independent of the write-triggered bucket invalidation (any embedding write into a `(object_type, modality, model_tag, tenant_id)` bucket evicts every cached result for that bucket regardless of TTL). | > **Usage example:** `RELATA_VECTOR_RESULT_CACHE_MB=128 RELATA_VECTOR_RESULT_CACHE_TTL_SECS=120 cargo run -p relata-cli -- serve` raises the KNN result-cache budget to 128 MiB and its TTL to 120s; `RELATA_VECTOR_RESULT_CACHE_ENABLED=false cargo run -p relata-cli -- serve` disables the cache entirely (every KNN/`HYBRID_SEARCH` vector sub-query re-runs the hot scan + cold-tier paging in full). | `RELATA_CACHE_DISK_PATH` | — | Directory path for the optional disk-tier segment cache (foyer `HybridCache`). When unset, the cache is RAM-only (`RowGroupCache`). Set together with `RELATA_CACHE_DISK_GB` to enable the disk tier. | | `RELATA_CACHE_DISK_GB` | — | Maximum disk space (GiB) reserved for the disk-tier segment cache. Ignored when `RELATA_CACHE_DISK_PATH` is unset. Integer; `0` or unset disables the disk tier. | | `RELATA_QUERY_ARENA` | `true` | When `true` (default), the in-memory sort path arena-allocates its keyed buffer via `bumpalo` — one bump allocation for the whole intermediate result set instead of repeated `malloc`/`free` per row. Set to `false` to fall back to standard heap allocation (useful for memory-profiling or if a bumpalo bug is suspected). | ### Query pattern tracker | Variable | Default | Description | |---|---|---| | `RELATA_PATTERN_HISTORY_LEN` | `16` | Number of recent query template hashes kept per session for Markov chain transitions. Higher values improve prediction accuracy at the cost of per-session memory. | | `RELATA_PATTERN_SESSION_TTL_SECS` | `300` | Seconds of inactivity before a session is GC'd from the pattern tracker. | | `RELATA_PREDICT_MIN_CONFIDENCE` | `0.3` | Minimum Markov transition confidence (fraction of observations) for a prediction to be returned. Range: `0.0`–`1.0`. | | `RELATA_PREDICT_TOP_K` | `2` | Maximum number of predicted next queries returned per prediction request. | | `RELATA_PREDICT_INFER_PARAMS` | `false` | Propagate changed literals from the triggering query into predictions (`search('y')` after a historical `search('x') → details('x')` predicts `details('y')`). Off = predictions replay each template's last session binding verbatim. | ### Speculative prefetch | Variable | Default | Description | |---|---|---| | `RELATA_SPECULATE_ENABLED` | `true` | Kill switch for the speculative prefetch pipeline. `false` disables prediction submission and the drain worker entirely. | | `RELATA_SPECULATE_MAX_CONCURRENT` | `1` (adaptive) | Maximum speculative queries executing concurrently. When unset, derived from detected cores (`cores/4`, min 1). Waiting work back-pressures into the bounded submit queue (overflow drops are counted in `relata_speculative_dropped_total`). | | `RELATA_SPECULATE_HOURLY_CAP` | `1000` | Fixed-window cap on speculative executions per hour. Over-cap work is dropped and counted. `0` denies all speculation (equivalent to disabling). | Callers can scope the pattern model per workflow by sending an `X-Relata-Session` header; without it, one model is shared per principal+org. --- ### Graph index and algorithms | Variable | Default | Description | |---|---|---| | `RELATA_GRAPH_CACHE_BYTES` | `268435456` (256 MiB) | Total byte budget for the 16-shard `PagedGraphCache` (CSR adjacency cache, `crates/relata-query/src/graph_cache.rs`). Higher values keep more (object_type × tenant) adjacency graphs RAM-resident. | | `RELATA_PAGED_GRAPH_CACHE_CAP` | `32` | Max number of (object_type × tenant × orientation) CSR graphs held in RAM by the investigation-ops cache (`crates/relata-query/src/investigation_ops.rs`). Oldest is evicted on overflow. | | `RELATA_GRAPH_PREWARM` | `true` | Whether to speculatively build + cache the CSR graph after bulk ingest (`crates/relata-cli/src/serve/ingest.rs`). Any value other than `false`/`0`/`no` is treated as enabled. | | `RELATA_GRAPH_PREWARM_THRESHOLD` | `1000` | Minimum batch size (rows ingested) that triggers the background graph prewarm. Batches smaller than this skip the prewarm. | | `RELATA_BETWEENNESS_APPROX_THRESHOLD` | `10000` | Node count above which `betweenness_centrality_approx` switches from exact Brandes to sampling-based (`crates/relata-graph/src/gds.rs`). Raise for more exactness on larger graphs at CPU cost. | | `RELATA_SPECULATE_GRAPH` | `false` | Opts in to speculative prefetch / caching for `GRAPH_*` queries (`crates/relata-query/src/query_pattern.rs`). Off by default because graph results depend on mutable CSR state — enable only when graph mutation is low-frequency. | --- ### Vector index tuning | Variable | Default | Description | |---|---|---| | `RELATA_HNSW_AUTO_TUNE` | `true` | Kill-switch for automatic HNSW `ef_search` recall tuning (`crates/relata-storage/src/vector.rs`). Disabled only by `"false"`/`"0"`. | | `RELATA_HNSW_ADAPTIVE_DEFAULTS` | `true` | Whether new cold-path `DiskAnnIndex` buckets pick M/M0/ef_construction adaptively from the current corpus-size hint (`crates/relata-storage/src/vector.rs::recommended_hnsw_params`) instead of always building at the fixed large-corpus default. Set to `false` to pin every new index to the fixed `DEFAULT_M`/`DEFAULT_EF_CONSTRUCTION` tier. Strictly parsed via `relata_core::bool_env`: any set-but-unrecognised value is a fatal startup error. Never consulted by an explicit `HnswIndex::with_params` call. | | `RELATA_AUTOTUNE_INTERVAL` | `10000` | How often (every N searches) the HNSW `ef_search` auto-tuner re-evaluates recall. Higher values sample less frequently (lower overhead). | | `RELATA_IVF_THRESHOLD` | `1000000` | Vector count at which the cold tier switches from HNSW to IVF-PQ (`crates/relata-storage/src/ivf_builder.rs`). Below this, the hot HNSW tier serves all queries. | | `RELATA_IVF_N_LISTS` | `0` | Number of coarse centroids (lists) when building the IVF-PQ cold tier. `0` = auto `sqrt(N)`. | | `RELATA_IVF_N_PROBE` | `8` | Number of IVF centroids scanned per query (recall/latency tradeoff). Higher = better recall, more CPU. | | `RELATA_VECTOR_COLD_TIER` | `ivf-pq` | Cold-tier ANN algorithm to switch to above the IVF threshold. `ivf-pq` (default) or `hnsw` to keep the hot-tier algorithm in the cold tier. | ### Query parse cache and optimizer | Variable | Default | Description | |---|---|---| | `RELATA_PARSE_CACHE_MAX_ENTRIES` | `4096` | Max entries in the process-global parsed-SQL cache (`crates/relata-query/src/parser.rs`). LRU eviction above the cap. | | `RELATA_PARSE_CACHE_MAX_BYTES` | `67108864` (64 MiB) | Total byte budget for the parsed-SQL cache. | | `RELATA_JOIN_REORDER` | `true` | Enables the cost-based (DPhyp-style) join-reorder optimizer (`crates/relata-query/src/cost.rs`). When `false`/`0`, the input join order is preserved unchanged. | | `RELATA_CACHE_PER_TENANT_MAX_MB` | cache_bytes ÷ 4 | Per-tenant byte cap on the query-result cache (`crates/relata-query/src/result_cache.rs`). Prevents a noisy tenant from evicting everyone else's results. | --- ### gRPC server | Variable | Default | Description | |---|---|---| | `RELATA_GRPC_TIMEOUT_MS` | `30000` | Per-RPC deadline in milliseconds — prevents a slow client from holding a worker forever. `0` disables the timeout (air-gap / batch workloads). | | `RELATA_GRPC_TIMEOUT_SECS` | `30` | Wall-clock deadline in seconds for gRPC streaming scan operations (`spawn_blocking` paths that cannot be interrupted by an async timeout). Applied per-stream as an `Instant` deadline checked at each batch boundary. | ### Benchmark harness | Variable | Default | Description | |---|---|---| | `RELATA_BIN` | — | Path to a prebuilt `relata` binary for the `relata-bench` HTTP-door suite (`bench_http_door`). When unset, the suite looks for the release binary next to the bench binary (`cargo build -p relata-cli --release` first). | ## Wire protocol ports HTTP is always on. pgwire requires `RELATA_BEARER_TOKEN`. Every other door **auto-enables when `RELATA_BEARER_TOKEN` is set** (any profile) — set `<DOOR>_ENABLE=false` to force it off, or `<DOOR>_ENABLE=true` to force it on without a token (only on `free`; non-`free` is fail-closed without a token). All protocol doors bind to **loopback (`127.0.0.1`)** by default; HTTP/gRPC on the `server`/`cluster` profiles bind `0.0.0.0`. Every bind var is a plain override on every profile, not license-gated. | Protocol | Enable flag | Port var | Default port | Bind var | Default bind | |---|---|---|---|---|---| | HTTP API | always on | `RELATA_PORT` | `9090` | `RELATA_HTTP_BIND` | profile: `127.0.0.1` (free) / `0.0.0.0` (server/cluster) | | gRPC | always on | `RELATA_GRPC_PORT` | `50051` | `RELATA_GRPC_BIND` | profile: `127.0.0.1` (free) / `0.0.0.0` (server/cluster) | | PostgreSQL wire | token required | `RELATA_PG_PORT` | `5433` | `RELATA_PG_BIND` | `127.0.0.1` | | S3-compatible | token-gated (`RELATA_S3_ENABLE`) | `RELATA_S3_PORT` | `9191` | `RELATA_S3_BIND` | `127.0.0.1` | | Redis | token-gated (`RELATA_REDIS_ENABLE`) | `RELATA_REDIS_PORT` | `6379` | `RELATA_REDIS_BIND` | `127.0.0.1` | | MongoDB | token-gated (`RELATA_MONGO_ENABLE`) | `RELATA_MONGO_PORT` | `27017` | `RELATA_MONGO_BIND` | `127.0.0.1` | | Neo4j HTTP | token-gated (`RELATA_NEO4J_ENABLE`) | `RELATA_NEO4J_PORT` | `7474` | `RELATA_NEO4J_BIND` | `127.0.0.1` | | Bolt | token-gated (`RELATA_BOLT_ENABLE`) | `RELATA_BOLT_PORT` | `7687` | `RELATA_BOLT_BIND` | `127.0.0.1` | | ClickHouse HTTP | token-gated (`RELATA_CLICKHOUSE_ENABLE`) | `RELATA_CLICKHOUSE_PORT` | `8123` | `RELATA_CLICKHOUSE_BIND` | `127.0.0.1` | | ClickHouse native TCP | token-gated (`RELATA_CLICKHOUSE_NATIVE_ENABLE`) | `RELATA_CH_NATIVE_PORT` | `9000` | `RELATA_CH_NATIVE_BIND` | `127.0.0.1` | | Arrow Flight | token-gated (`RELATA_FLIGHT_ENABLE`) | `RELATA_FLIGHT_PORT` | `8815` | `RELATA_FLIGHT_BIND` | `127.0.0.1` | Additional door options: | Variable | Description | |---|---| | `RELATA_S3_SECRET_KEY` | SigV4 secret for the S3 door. Defaults to `RELATA_BEARER_TOKEN`. When set (the default when a token is present), the door REQUIRES verified SigV4 and rejects plaintext bearer / unsigned access-key auth. | | `RELATA_S3_ALLOW_PLAINTEXT` | Default `false`. Dev/legacy opt-out: set `true` to accept plaintext bearer / unsigned access-key auth on the S3 door even when a secret is configured (the credential travels in cleartext and is forgeable — do not use in production). | | `RELATA_MONGO_DEBUG` | Set `true` for verbose MongoDB wire debug logging. | | `RELATA_BOLT_DEBUG` | Set `true` for verbose Bolt wire debug logging. | | `RELATA_DOOR_READ_TIMEOUT_SECS` | Default `1800` (30 min). Idle-read timeout on the PostgreSQL wire door. Connections that receive no data for this many seconds are closed. Increase for long-running ETL sessions; decrease to reclaim idle connections faster. | | `RELATA_REDIS_READ_TIMEOUT_SECS` | Default `30`. Idle-read timeout on the Redis RESP door. Connections idle for this many seconds are closed. | | `RELATA_PGWIRE_STMT_TIMEOUT_MS` | Default `30000` (30 s). Per-statement timeout on the pgwire door. A query running longer than this is cancelled and the client receives an error. | | `RELATA_MONGO_MAX_CONNS` | Default `100`. Max concurrent connections on the MongoDB wire door. | | `RELATA_S3_BODY_LIMIT_MB` | Default `4` (4 MiB). Max inline S3 PutObject body size. Larger objects must use multipart upload. | | `RELATA_S3_MULTIPART_LIMIT_MB` | Default `10240` (10 GiB). Max total size for S3 multipart uploads. Individual parts default to 5 MiB. | | `RELATA_S3_BLOB_THRESHOLD_MB` | Default `4` (4 MiB). Objects larger than this are stored as content-addressed blobs in the object store; smaller objects are inlined. | | `RELATA_S3_BODY_TEXT_CAP` | `65536` | Max byte length of an S3 PutObject body that is decoded as UTF-8 and stored in the `body_text` column for identity enrichment. Bodies larger than this or non-UTF-8 bodies are stored in `body` only. | | `RELATA_S3_NOTIFY_URL` | — | HTTP endpoint for fire-and-forget S3 event notifications. When set, a JSON payload `{"event":"<put|delete>","bucket":"…","key":"…"}` is POSTed after each mutating S3 operation. Errors are logged and discarded (non-blocking). | --- ## TLS / mTLS | Variable | Default | Description | |---|---|---| | `RELATA_TLS_CERT` | — | Path to the TLS certificate (PEM) for in-process HTTP TLS termination. When set with `RELATA_TLS_KEY`, the HTTP listener binds with rustls TLS. Required on **every** profile — including `free` — unless `RELATA_PLAINTEXT_OK` resolves `true` (see that row for the loopback default). | | `RELATA_TLS_KEY` | — | Path to the TLS private key matching `RELATA_TLS_CERT`. | | `RELATA_GRPC_TLS_CERT` | — | gRPC TLS certificate path. | | `RELATA_GRPC_TLS_KEY` | — | gRPC TLS key path. | | `RELATA_GRPC_TLS_CA` | — | CA certificate for gRPC mTLS (mutual TLS client verification). | | `RELATA_GRPC_PLAINTEXT_OK` | `false` | Allow gRPC listener to boot without TLS (`true`/`1`/`yes`/`on`). **Production hazard.** Without it, missing TLS config is a hard startup error. | | `RELATA_HTTP_TIMEOUT_SECS` | `30` | Per-request HTTP timeout for the axum server's graceful-shutdown drain window and long-poll handlers. Distinct from the SDK-side `timeout=` constructor arg. | | `RELATA_GRPC_MAX_DECODE_BYTES` | `16777216` | Max inbound gRPC message size (bytes) before rejection. Raise for large-batch ingest. | | `RELATA_GRPC_CONCURRENCY_PER_CONN` | `256` | Max concurrent in-flight requests per gRPC connection. | | `RELATA_GRPC_MAX_CONCURRENT_STREAMS` | `256` | Max concurrent HTTP/2 streams per gRPC connection. | | `RELATA_JWKS_GRACE_SECS` | `600` | JWKS cache grace window (seconds) — a stale key is served this long past expiry while a refresh is attempted (`oidc-verify` mode). | | `RELATA_PGWIRE_ORG` | — | Organization/agency attribute stamped on the principal for the Postgres-wire door (psql has no native org concept). Unset = no org attribute. | | `RELATA_PG_TLS_CERT` | — | Path to the PEM certificate file for pgwire (Postgres-wire) TLS. When both `RELATA_PG_TLS_CERT` and `RELATA_PG_TLS_KEY` are set, the pgwire listener binds with TLS; when either is absent TLS is disabled and the listener operates in plaintext mode. | | `RELATA_PG_TLS_KEY` | — | Path to the PEM private-key file for pgwire TLS. Required alongside `RELATA_PG_TLS_CERT`; ignored when the cert is unset. | | `RELATA_PG_COPY_MAX_BYTES` | `1073741824` | Maximum in-memory buffer size for a `COPY <type> FROM STDIN` pgwire session (bytes). Payloads exceeding this limit are rejected with an error to prevent a single client from exhausting server memory. Default: 1 GiB. | | `RELATA_SCATTER_MAX_PARALLEL` | `64` | Max parallel fan-out requests per cluster scatter-gather read (clamped ≥ 1). | | `RELATA_SCATTER_PEER_TIMEOUT_MS` | `10000` | Per-peer request timeout (ms) for cluster scatter-gather fan-out. Clamped ≥ 1 ms. Raise on high-latency inter-node links; lower to fail fast on unreachable peers. | | `RELATA_SLOW_QUERY_MS` | `500` | Query wall-clock threshold (ms) above which a query is recorded in the `slow_queries` ring surfaced on the metrics dashboard. | | `RELATA_WAL_PUT_MAX_ATTEMPTS` | `3` | Max object-store PUT attempts per WAL segment upload (clamped ≥ 1). | | `RELATA_PARQUET_COMPRESSION` | `zstd` | Parquet segment compression codec. Accepted: `zstd` (default, level 3), `zstd:<level>` (level 1-22), `lz4`, `snappy`, `none`. ZSTD level 3 gives 2-4× size reduction vs Snappy at similar decode throughput. | | `RELATA_FLUSH_SEGMENT_MAX_ROWS` | `0` | Cap on rows per Parquet segment during flush to the object store. `0` = unbounded (legacy single-segment). Non-zero splits large flushes into multiple smaller segments for better S3 multipart upload behaviour and reduced p99 on very large commits. | | `RELATA_LAZY_RESTART` | `false` | When `true`, startup loads the manifest **catalog only** (O(manifest), not O(rows)) instead of eager row restoration. Faster cold starts; older segments hydrate on demand. | | `RELATA_HYDRATE_RECENT_SEGMENTS` | `0` | With lazy restart on, hydrate only the newest N segments into RAM at startup. `0` = fully lazy (all segments hydrate on demand). Set to a small number (e.g. 3) to keep recent data hot. | | `RELATA_MAX_AGENCY_INDEX_BUCKETS` | `200000` | Soft cap on total `(type, agency)` agency-index buckets before the memory-pressure spill path evicts the smallest-live-set buckets first. Bounds O(tenants × types) index growth; evicted buckets rebuild lazily and reads stay correct via the field-filtered fallback. | | `RELATA_EMBED_QUEUE_MAX` | `100000` | Cap on the embedding-backlog queue (`MediaWorker`). Tasks dropped beyond cap; `relata_embed_queue_dropped_total` counter increments. | | `RELATA_EMBED_QUEUE_HWM` | 90% of `RELATA_EMBED_QUEUE_MAX` | High-water mark for the embed queue. When queue depth ≥ HWM, all ingest endpoints return `429 Too Many Requests` with `Retry-After` so callers slow down before silent drops begin. Response body includes `embed_queue_depth`. Set equal to `RELATA_EMBED_QUEUE_MAX` to disable. | | `RELATA_EMBED_TIMEOUT_MS` | `30000` | Sidecar embedding HTTP call timeout in ms. On timeout, the drain worker logs + retries next cycle. | | `RELATA_EMBED_CIRCUIT_COOLDOWN_MS` | `60000` | Circuit-breaker cooldown for the embedder sidecar. After N consecutive failures, `enqueue` fast-fails until the cooldown elapses. | | `RELATA_EMBED_BATCH_SIZE` | `32` | Number of texts per sidecar embedding HTTP call. GPUs are designed for batched inference — batch_size=32 typically gives 5-10× throughput over batch_size=1. Range `[1, 1024]`. | | `RELATA_EMBED_CONCURRENCY` | `4` (adaptive) | Number of concurrent drain workers sharing the embedding queue. When unset, derived from detected cores (= cores). Multiple workers keep the GPU saturated while the CPU writes previous results back. Combined with `RELATA_EMBED_BATCH_SIZE`, gives ~15-50× throughput over the sequential per-row path. | | `RELATA_SEARCH_PRESET` | `balanced` | Search relevance preset: `strict` (exact-match preferred, no typo tolerance — governed/production), `balanced` (light typo tolerance, prefix on short tokens — general-purpose), or `lenient` (edit-distance 2, fuzzy on — demo/evaluation). See [Search presets](/docs/reference/search). | | `RELATA_SEARCH_LAST_TERM_PREFIX` | `true` | When `true`, the `/search` handler treats the last whitespace-delimited token as a prefix and unions BM25 results with `prefix_search` hits — enabling search-as-you-type without explicit wildcard syntax. Set to `false` to disable. Can also be overridden per-request via the `lastTermIsPrefix` body field. | | `RELATA_SEARCH_LANG` | `en` | ISO 639-1 language code for the built-in stemmer. **15 hand-rolled stemmers ship** (en, fr, de, es/pt (shared `stem_es`), it, nl (Dutch), sv (Swedish), no (Norwegian), da (Danish), fi (Finnish), hu (Hungarian), ro (Romanian), ru (Russian Cyrillic), tr (Turkish), ar (Arabic)) — see `crates/relata-storage/src/search.rs:3189-3745`. Unknown codes fall back to English. CJK text (Han/Kana/Hangul) is always bigram-segmented regardless of this setting. Read once at startup — restart required for changes to take effect. | | `RELATA_WAL_MIRROR_CHANNEL_CAP` | `4096` | Cap on the object-store WAL mirror channel. Prevents unbounded memory if the remote WAL drain falls behind. | | `RELATA_ALLOWED_ORIGINS` | — | **Removed** — use `RELATA_CORS_ALLOWED_ORIGINS`. Startup will FATAL if set. | | `RELATA_STOP_WORDS_LANG` | `en` | Stop-word language: `en` for built-in English list, `none` to disable. | | `RELATA_STOP_WORDS_FILE` | — | Path to a custom stop-word file (one word per line). Overrides the built-in list when set. | | `RELATA_SYNONYMS_FILE` | — | Path to a JSON synonym file for query-time expansion, e.g. `{"phone": ["telephone", "mobile"]}`. | | `RELATA_FACETS_<Type>` | — | Comma-separated list of facetable attributes per type, e.g. `RELATA_FACETS_Product=category,brand`. | | `RELATA_RANKING_<Type>` | — | Per-type custom ranking rules, e.g. `RELATA_RANKING_Article=recency:published_at:86400,popularity:weight:0.3`. | | `RELATA_REQUIRE_MTLS` | — | **Removed** — set `RELATA_AUTH_MODE=mtls` to enforce mutual TLS. Startup will FATAL if set. | | `RELATA_MTLS_CA_CERT_PATH` | — | CA certificate (PEM) used to verify client certs. Required for `RELATA_AUTH_MODE=mtls`. | | `RELATA_MTLS_REQUIRE_CLIENT_CERT` | — | Require client certificate on TLS handshake. | | `RELATA_MTLS_ALLOWED_DNS_SANS` | — | Comma-separated list of allowed DNS SANs in client certs. | --- ## Rate limiting | Variable | Default | Description | |---|---|---| | `RELATA_RATE_LIMIT_RPS` | free: `10000`, server/cluster: `100000` | Global per-IP request rate (requests/sec). Free defaults high (dev — false positive). Licensed tiers default 10x free (database-class throughput). Explicit env var overrides all profiles. | | `RELATA_RATE_LIMIT_BURST` | free: `10000`, server/cluster: `100000` | Burst capacity above `RELATA_RATE_LIMIT_RPS`. Matches the RPS default per profile. | | `RELATA_ACL_GRANT` | — | Comma-separated ACL grants for partner types, e.g. `CustomClaim:read+write,CustomDispute:read+write`. Grants **both** `api-user` and `mcp-client` principals ownership + ACL allow on the named types so ingest, query, and MCP tools all work without `403`. Bare type name (no `:perms`) defaults to `read`. | | `RELATA_DISKANN_DISK_RESIDENT` | — | When `true`, writes a `.rgph` sidecar alongside the HNSW graph for disk-resident ANN beam-search. Requires a backing object store. | | `RELATA_DISKANN_ALPHA` | `1.2` | Alpha selectivity for the Vamana RobustPrune step. Higher values (e.g. 1.4) prune more aggressively, reducing edge count at the cost of recall. Must be >= 1.0. | | `RELATA_DISKANN_MAX_DEGREE` | `32` | Maximum out-degree per node in the Vamana graph. Bounds memory used by the build-phase adjacency lists. | | `RELATA_DISKANN_L_BUILD` | `100` | Candidate list size `L` during the Vamana greedy-search build pass. Larger values improve graph quality at the cost of build time. | | `RELATA_COLD_TIER_FAIL` | `open` | DiskANN cold-tier page-in failure mode (`crates/relata-storage/src/paged_ann.rs`). `open` (default): a failed cold-tier read returns an empty result (observability-only — the query succeeds with fewer candidates). `closed`: a failed cold-tier read fails the query. Set to `closed` only when you need hard failure semantics on cold-tier I/O errors. | | `RELATA_COLD_ANN` | `rebuild` | Cold-tier IVF absorb strategy (`crates/relata-storage/src/paged_ann.rs` / `ivf_builder.rs`). `rebuild` (default): the legacy `ColdIvfBucket` behaviour — a full k-means rebuild of the resident overflow once it doubles. `incremental`: routes inserts into a centroid-incremental tier (`IncrementalCentroidIndex`) that appends to the nearest centroid's posting list and periodically re-centres only the centroids that changed, never rerunning k-means over the whole corpus. | | `RELATA_READ_RATE_LIMIT_RPS` | — | Read-path rate limit (separate bucket from write path). | | `RELATA_MEMORY_RATE_LIMIT_RPS` | — | In-memory scan rate limit. | | `RELATA_RATE_LIMIT_AUTH_FAIL_RPS` | free: `10000`, server/cluster: **`10`** | Brute-force throttle after auth failure. The server/cluster default is intentionally low (10 RPS) to slow credential stuffing; free uses the same value as `RELATA_RATE_LIMIT_RPS`. Always `≥1` (`.max(1)` guard). | | `RELATA_MAX_CONNS` | — | Max concurrent connections accepted. | | `RELATA_WEBHOOK_MAX_INFLIGHT` | `32` | Ceiling on concurrently in-flight alert-webhook deliveries (`spawn_alert_webhooks`). Deliveries beyond the cap are dropped with a `warn!`; deliveries under the cap get a bounded retry with exponential backoff on 5xx/timeout before giving up. Raise for high-volume alerting pipelines; lower to bound webhook concurrency. Zero/invalid falls back to `32`. | | `RELATA_QUERY_QUOTA` | — | Per-principal read-cost cap (cost units, not rows). A principal whose cumulative read cost exceeds this within `RELATA_QUERY_QUOTA_WINDOW_SECS` receives a 429. Distinct from `RELATA_MAX_RESULT_ROWS` (which caps result size). | | `RELATA_QUERY_TIMEOUT_SECS` | — | Per-query execution deadline (seconds); a query past it aborts with a timeout error so it can't pin a worker. Unset/0 = no limit. | | `RELATA_QUERY_MAX_INFLIGHT` | adaptive (detected cores, min 1) | CPU-query admission control: max `POST /query` executions allowed to run concurrently inside the offloaded execution core. A request that can't get a slot within `RELATA_QUERY_ADMISSION_WAIT_MS` is shed with `503` + `Retry-After` instead of queueing unboundedly. `/health`/`/health/ready` are exempt. | | `RELATA_QUERY_ADMISSION_WAIT_MS` | `50` | Max time (ms) `POST /query` waits for a CPU-query admission slot before shedding. | | `RELATA_QUERY_CPU_THREADS` | adaptive (detected cores × 4, min 4) | Max blocking-pool threads (`tokio::runtime::Builder::max_blocking_threads`) on the real server runtime — the pool an offloaded `/query` execution (`block_in_place`) runs on. Overrides tokio's flat built-in default (512). | | `RELATA_MAX_WATCH_SUBSCRIPTIONS` | `1024` | Max concurrent WATCH subscriptions; new ones past the cap are shed (each re-evaluates on every commit, so an unbounded count amplifies commit latency). | | `RELATA_MAX_RESULT_ROWS` | — | Hard ceiling on result set size. | | `RELATA_TRUSTED_PROXIES` | — | Comma-separated trusted proxy CIDRs for real-IP extraction. | | `RELATA_TRUST_UPSTREAM_PROXY` | — | Set `true` to trust `X-Forwarded-For` from the first proxy hop. | | `RELATA_CORS_ALLOWED_ORIGINS` | — | Comma-separated allowed CORS origins **and** CSRF guard allowlist. Both layers read this single var on every profile. When unset: `server`/`cluster` block all cross-origin requests and CSRF protection is disabled (warn); `free` defaults to a localhost-only allowlist (`localhost`/`127.0.0.1` on ports 3000/9090/5173) tuned for local frontend dev. E.g. `https://app.example.com,https://admin.example.com`. | --- ## Ingest and query | Variable | Default | Description | |---|---|---| | `RELATA_INGEST_PARTITIONS` | — | Number of parallel ingest write partitions. | | `RELATA_INGEST_QUEUE_MAX_BYTES` | `1073741824` (1 GiB) | Byte budget for queued (not-yet-drained) ingest batches. Backpressure (HTTP 429) fires when EITHER the batch count (10,000) OR this byte budget would be exceeded — so a handful of very large batches can't admit hundreds of GB before the store-side cap sees them. `0` disables the byte check (count-only backpressure). | | `RELATA_INGEST_QUEUE_LANES` | Detected CPU cores, clamped `1..=64` | Number of independent lock domains ("lanes") the ingest queue shards into (adaptive sizing). Each lane owns its own deque + content-hash dedup window, so producers routed to different lanes (by a stable hash of `object_type` + tenant) never contend on the same mutex. `capacity`/`RELATA_INGEST_QUEUE_MAX_BYTES` and per-tenant quota stay global regardless of lane count. Parsed strictly — a non-integer value fails startup. | | `RELATA_INGEST_SHED_PCT` | `80` | Queue-depth % at which the node sheds ingest: `/health/ready` returns 503 ("ingest queue backpressure") and the readiness gate fails. Clamped to 1..=100; out-of-range/unparseable falls back to 80. The last previously-hardcoded ingest-shed knob (complements the byte/count caps above). | | `RELATA_SEQUENCE_RULE` | — | Comma-separated spec for the sequence-correlation detection job's steps (e.g. `A→B,window=300`). Unset = default rule. | | `RELATA_AUTO_REGISTER_TYPES` | — | Set `true` to create new types on first ingest without prior DDL. | | `RELATA_GLOBAL_SCAN_ALLOWED` | — | Set `true` to allow full-table scans (expensive on large datasets). | | `RELATA_MV_MAX_ROWS` | — | Max rows per materialized view partition. | | `RELATA_FACET_SCAN_LIMIT` | `100000` | Row ceiling for `FACETS` aggregation. When a query requests facets, its LIMIT is bumped to at least this many rows so facet counts reflect the full matching set rather than just the top-k; the same value short-circuits the per-row facet-counting loop even if the caller's own LIMIT was larger. Lower it on smaller deployments to cut the bumped-scan cost; raise it for accurate facets over larger match sets. | | `RELATA_AUDIT_SHARDS` | `8` | Number of independent hash-chain shards for the audit log, clamped to `[1, 256]`. Each shard has its own lock, so `push` throughput scales with core count. Set to `1` to reproduce the legacy single-chain behavior exactly -- the setting to use when a compliance requirement demands one global total order. | | `RELATA_FTS_MAX_DOCS` | `5000000` on `server`/`cluster` profiles; unbounded on `free` | Max documents held in the full-text search index before the spill trigger fires. Set explicitly to override the profile default. | | `RELATA_FTS_MAX_TERMS` | `5000000` | Max unique terms in the full-text dictionary. Past this cap, new terms are silently dropped so the dictionary stays below ~200 MB even under high-cardinality OCR/log/social-media ingest. | | `RELATA_FTS_DOC_CACHE_MAX` | `100000` | Max snippet-text entries held in the in-memory `doc_text` cache. The oldest entry is evicted on overflow so the cache stays bounded; a cache miss yields an empty snippet (never a crash). Set to `0` to disable the cap entirely. | | `RELATA_FTS_RESIDENT_TEXT` | `false` | Whether the full-text index keeps a resident copy of every indexed document's text for snippet generation. Off by default: a search hit's snippet is instead resolved lazily from the row store, for the surviving `limit` hits only -- the row table already holds the text, so the default path never stores it twice. Set to `true` to restore the legacy resident-snippet behavior (marginally faster for small corpora that query heavily). Strict `bool_env` parse. | | `RELATA_FTS_SHARDS` | `1` | Number of independently-locked segments each per-type full-text index fans documents/queries out across. A document routes to shard `hash(id) % N`; a write only takes that one shard's lock, so a reader on any other shard is never blocked by concurrent indexing, and a query is scored across all `N` shards in parallel. BM25 IDF and length normalization stay corpus-global regardless of `N` (shared `n_docs`/`total_doc_len`/per-term `df`), so the default `1` (today's single-lock behavior, unchanged) and any `N > 1` return identical rankings -- only the RAM layout and concurrency characteristics differ. Clamped to at least 1; malformed/`0` values fall back to `1`. Strict `parse_env_u64_or`. | | `RELATA_WAND_OFFSET_THRESHOLD` | `1000` | Minimum `offset` value at which `preset_search_with_options` switches from standard WAND to Block-Max WAND (BMW) for deep-offset pagination. BMW uses per-block BM25 upper bounds to skip entire 64-doc blocks below the current score threshold, achieving O(log N) pruning instead of O(N) scanning. Results are identical to the standard path — BMW is a safe optimisation. Set to `0` to always use BMW; set to a very large value to disable it. | | `RELATA_PENDING_FTS_MAX` | `1000000` | Cap on the pending full-text-index backlog (documents awaiting async FTS indexing) before new entries are dropped and counted. `0` (or unset) uses the default. | | `RELATA_PENDING_INDEX_WORK_MAX` | `1000000` | Cap on the pending secondary/range-index + vector-embedding backlog (rows awaiting async indexing, deferred off the `insert`/`insert_with_prov` critical path). UNLIKE `RELATA_PENDING_FTS_MAX`, past this cap the insert is REJECTED (`StoreError::IndexQueueFull`) rather than silently dropped — a dropped secondary/vector entry would permanently hide a row from index-routed queries. Unset uses the default. | | `RELATA_INDEX_BACKPRESSURE_THRESHOLD` | `200000` | On a `RELATA_ROLE=query` node only: once the deferred index-work queue (spilled-or-unconfirmed rows, `ObjectStore::pending_secondary_work_queue_len`) reaches this depth, ingest doors (`/ingest`, OTLP traces/logs/metrics, schemaless) return HTTP 429 with a `Retry-After` hint instead of accepting more writes — protects against an unbounded backlog when the `RELATA_ROLE=indexer` node is slow or down. No-op on `RELATA_ROLE=both`/`indexer`. | | `RELATA_INDEX_BACKPRESSURE_DISABLED` | `false` | Set `true` to opt a `RELATA_ROLE=query` deployment out of the `RELATA_INDEX_BACKPRESSURE_THRESHOLD` 429 shedding above — writes are accepted regardless of backlog depth. | | `RELATA_ANN_EAGER` | — | Set `true` to eagerly load ANN index into RAM at startup. | | `RELATA_RULE_EVAL_INTERVAL_SECS` | — | How often the rules engine re-evaluates derived facts. | | `RELATA_DETECTION_JOBS_INTERVAL_SECS` | — | Detection job sweep interval. | | `RELATA_JOBS_MAX_PARALLEL` | `4` | How many `(job, tenant)` detection-scan units run concurrently on the blocking pool per scheduler tick. | | `RELATA_JOBS_TRIGGER_DEBOUNCE_MS` | `500` | Debounce window for an event-triggered detection job — how long to wait after the first commit to a subscribed object type before running, so a burst of commits coalesces into one run instead of one per commit. | | `RELATA_JOBS_TRIGGER_POLL_MS` | `200` | How often `detection_jobs_task` re-checks pending event-triggered jobs even with no fresh commit notification, bounding the worst-case delay between a debounce window elapsing and the job being observed due. Not the primary wake-up path — that is the storage layer's commit `Notify`. | | `RELATA_WORKFLOW_DRIVER_INTERVAL_SECS` | `1` | How often (seconds) the background workflow driver advances in-flight workflow executions. | | `RELATA_WORKFLOW_STEP_TIMEOUT_SECS` | — | Per-run wall-clock timeout (seconds). If set and a run exceeds it between step batches, remaining pending steps are marked failed and a dead-letter alert fires. Unset = no timeout. | | `RELATA_WRITE_CONCERN` | `one` | Replication write concern: `one` (fire-and-forget, default), `quorum` (wait for majority of peers), `all` (wait for all peers). Non-`one` values bound RPO under failure at the cost of write latency. **Startup fails on any other value** (a typo previously silently downgraded to `one`). | | `RELATA_DETECT_PACKS` | — | Comma-separated detector packs to activate. | | `RELATA_DETECT_BATCH_SIZE` | `256` | Chunk size for batched identity detection (`SmartIngest::detect_batch_with`). Inputs are detected in chunks of this many so the per-input setup and a reused hit buffer are amortised across the chunk instead of one synchronous `detect_with` per cell. Non-positive / unparseable values fall back to `256`. Detection results are identical regardless of batch size; this only tunes throughput. | | `RELATA_FK_EDGES` | — | Comma-separated FK-to-edge mappings: `TypeName.field=predicate,...` (e.g. `CdrRecord.tower_id=uses_tower,TransactionGraph.from_wallet=sends_to`). At ingest time each matching FK field is materialised as a `KnowledgeTriple` row so `PATHS_BETWEEN` can traverse typed-object→graph boundaries without schema changes. | | `RELATA_IDENTITY_LINK` | — | Comma-separated identity-link mappings: `TypeName.field=predicate,...` (e.g. `CdrRecord.caller_msisdn=has_msisdn`). At ingest time the field value is materialised as a `KnowledgeTriple` with the given predicate. Use identity-semantic predicates (`has_msisdn`, `has_email`, `identified_by`, `linked_to`) to make the link visible through `lookup_identity`. | --- ## Observability | Variable | Default | Description | |---|---|---| | `RELATA_LOG_LEVEL` | `info` | Log verbosity: `error` \| `warn` \| `info` \| `debug` \| `trace`. | | `RELATA_LOG_FORMAT` | TTY-detect | Log format: `pretty` (human-readable) \| `json` (structured). Defaults to `pretty` when stderr is a TTY, `json` otherwise. Note: the CLI binary defaults to `pretty` format; containers should explicitly set `RELATA_LOG_FORMAT=json` for structured log ingestion. | | `RELATA_OTLP_ENABLED` | `false` | Set `true` to enable native OTLP HTTP ingest endpoints (`POST /v1/traces`, `/v1/metrics`, `/v1/logs`). Disabled by default; off-the-shelf OTLP exporters (Jaeger, Tempo, Grafana Alloy) will receive 404 until enabled. | | `RELATA_OTLP_ENDPOINT` | — | OpenTelemetry OTLP/HTTP endpoint. Telemetry disabled when unset. | | `RELATA_OTLP_SAMPLE_RATIO` | `0.01` | Parent-based TraceID-ratio sampler for root traces (1% default). | | `RELATA_METRICS_PUBLIC` | — | Set `true` to serve `/metrics` without the bearer check (auth terminated at the network layer). Default fails closed. | | `RELATA_PROFILE_SAMPLE_RATE` | `0.01` | Fraction (`[0.0, 1.0]`) of production queries that get per-operator attribution (scan/filter/join/aggregate/sort/acl/serialize timing, feeding `/metrics` — see the [flamegraph guide](/docs/guides/observability)). `0.0` disables sampling entirely; `EXPLAIN ANALYZE` always instruments regardless of this setting. Malformed or out-of-range values fail startup. | | `RELATA_PPROF_ENABLE` | `false` | Enable `GET /debug/pprof/profile` (CPU pprof protobuf, `?seconds=1..30`) and `GET /debug/pprof/heap`. **Off by default** — a profiling endpoint is an attack surface (leaks workload shape + internal symbol names). When enabled, both routes additionally require `RELATA_ADMIN_TOKEN` to be set and a matching `Authorization: Bearer` on **every** profile, including the `free` profile where the general bearer check is otherwise optional — there is no dev bypass for this surface. See the [flamegraph guide](/docs/guides/observability). | --- ## Cluster | Variable | Default | Description | |---|---|---| | `NODE_ID` | `node-1` | Cluster node identifier. | | `NODE_ADDR` | — | This node's advertised address (host:port) for peer-to-peer communication. | | `NODE_REGION` | — | Region label for geo-aware routing. | | `CLUSTER_PEERS` | — | Comma-separated peer URLs. Empty = standalone mode. | | `CLUSTER_ROLE` | `coordinator` | Node role: `coordinator` \| `reader` \| `writer` \| `indexer` (routing/registry hint, not a hard access gate — see [Cluster Setup](/docs/deployment/cluster) for what each role means). An unrecognized value fails startup. | | `RELATA_ROLE` | `both` | Indexing-placement role: `query` \| `indexer` \| `both` (`crates/relata-cli/src/serve.rs`). Independent of `CLUSTER_ROLE` above (that one drives query fan-out routing; this one drives where deferred secondary/range/vector index-maintenance work — `relata-storage`'s `pending_index_work` queue — is applied). `both` (default) preserves the legacy behaviour: the index-work-drain background task applies deferred work inline, on this node. `query` spills that work to a `pending-index/` object-store prefix instead (falls back to inline drain with a loud warning if no remote store is configured) so a heavy bulk-index workload never touches this node's CPU. `indexer` runs a dedicated worker loop (`crates/relata-cli/src/serve/indexing_worker.rs`) that claims and applies spilled work items via an S3 conditional-put lease (`relata_cluster::WriteCoordinator::try_claim_index_work_item`); this task is otherwise idle. | | `CLUSTER_COORDINATOR` | — | Coordinator URL for leader election. | | `CLUSTER_DISCOVERY` | — | Discovery mechanism (`static` \| `dns` \| `k8s`). | | `CLUSTER_AUTH_TOKEN` | — | Shared token for inter-node gRPC authentication. | | `RELATA_GRPC_REQUEST_TIMEOUT_SECS` | `30` | Default per-RPC timeout (seconds) on the cluster gRPC client pool. Applies to every inter-node RPC unless overridden per-call; a stuck peer returns `DeadlineExceeded` instead of hanging the caller. | | `RELATA_CLUSTER_SHARDS` | `8` | Consistent-hash shard count. Must be identical on every node. | | `RELATA_CLUSTER_SEED` | — | Shared seed for partition key derivation (alternative to explicit K0/K1). | | `RELATA_CLUSTER_DEAD_AFTER_SECS` | `90` | Seconds without a heartbeat before a node is evicted. | | `RELATA_CLUSTER_REBALANCE_TIMEOUT_SECS` | — | Max time allowed for a rebalance operation. | | `RELATA_MAX_REPLICATION_LAG` | `10000` | Replica readiness gate: when this node is a replica (`reader` role) and its replication lag (`frontier − applied` WAL sequence units) exceeds this, `/health/ready` returns 503 so the replica is drained from the read path before it serves stale data. Current lag is also exported as the `relata_replication_lag` Prometheus gauge and in the readiness JSON. `0` disables the gate. Non-replica / single-node deployments are unaffected. Prometheus metrics: `relata_replication_lag` (this node, WAL sequence units), `relata_replication_lag_seconds` (slowest follower, wall-clock seconds), `relata_replication_lag_seconds_by_replica{replica_id}` (per-replica WAL sequence units). Alert `RelataReplicaLagHigh` fires when any replica exceeds 30 sequence units for 5 minutes. | | `RELATA_MULTI_REGION` | — | Set `true` to enable multi-region active-active mode. | | `RELATA_CLUSTER_BRANCH` | `main` | Branch this node stamps on fenced write-leases and `/internal/replicate` batches (cluster profile). | | `RELATA_LEASE_TTL_MS` | `30000` | Fenced write-lease TTL in milliseconds (cluster profile). Renewed every `ttl/3`; a missed renewal loses the lease. | | `RELATA_PARTITION_KEY_K0` | — | First u64 half of the 128-bit SipHash partition key. Both halves required when set. | | `RELATA_PARTITION_KEY_K1` | — | Second u64 half of the 128-bit SipHash partition key. | | `RELATA_GRPC_DIAL_MAX_ATTEMPTS` | `3` | Max inter-node gRPC dial attempts before giving up (clamped 1–10). | | `RELATA_GRPC_DIAL_BACKOFF_MS` | `50` | Base backoff between gRPC dial attempts (exponential, ms). | | `RELATA_GRPC_BREAKER_THRESHOLD` | `5` | Consecutive failures before the per-peer gRPC circuit breaker opens (clamped 1–100). | | `RELATA_GRPC_BREAKER_COOLDOWN_MS` | `5000` | Cooldown before a tripped gRPC circuit breaker probes the peer again. | | `RELATA_HEDGE_ENABLED` | `false` | Set `true`/`1`/`yes`/`on` to hedge scatter-gather reads (send a backup request after a delay). | | `RELATA_HEDGE_DELAY_MS` | `50` | Delay before issuing the hedged backup request when hedging is enabled; also the floor/fallback delay when a peer has no `LatencyTracker` samples yet. | | `RELATA_HEDGE_PERCENTILE` | `0.95` | When a `LatencyTracker` is wired, the hedge backup fires after this percentile of the target peer's own recently-observed latency instead of the fixed `RELATA_HEDGE_DELAY_MS`, floored at that delay. Clamped to `[0.0, 1.0]`. | | `RELATA_PARTITION_STRATEGY` | `hash` | Cluster row-partition strategy (`crates/relata-cluster/src/partition.rs`). `hash` (default, consistent-hash over the row key) \| `smart-graph` (co-locates graph-connected rows). Unknown values fall back to `hash` with a warning log. | | `RELATA_BRANCH_SHARD_REGION` | — | Opt-in: when set to a non-empty region name, constructs a `BranchShardCoordinator` that makes the `CROSS_SHARD_WRITE` guard load-bearing — a governed write whose `_target_branch` names a branch owned by a different shard is rejected instead of silently accepted. Unset (default) = no-op guard, every deployment unaffected. | | `RELATA_BRANCH_SHARD_CASE` | `main` | Case/branch-family name used to derive this shard's owned branch alongside `RELATA_BRANCH_SHARD_REGION` (`BranchShard::branch_name_for`). Only read when `RELATA_BRANCH_SHARD_REGION` is set. | | `RELATA_CROSS_REGION_PEERS` | — | Comma-separated `region=addr` pairs declaring cross-region merge peers, e.g. `eu=https://eu.example:9443,apac=https://apac.example:9443`. Only read when `RELATA_BRANCH_SHARD_REGION` is set; populates `BranchShardCoordinator.remote_shards` so the cross-region merge scheduler (below) has peers to fetch from. Malformed entries are skipped with a warning; unset/empty keeps the prior single-node default (no peers), and the merge scheduler no-ops every tick. | | `RELATA_CROSS_REGION_MERGE_INTERVAL_SECS` | `60` | Interval for the supervised cross-region merge background task that calls `run_merge_cycle_gated`. Strictly parsed — a malformed value fails startup. No-ops (logs at debug and skips the cycle) when `RELATA_BRANCH_SHARD_REGION`/`RELATA_CROSS_REGION_PEERS` are unset, or when `RELATA_MULTI_REGION` is off. | --- ## LLM and AI inference | Variable | Default | Description | |---|---|---| | `RELATA_LLM_URL` | — | LLM API base URL (OpenAI-compatible). Also accepted by `OPENAI_BASE_URL` for compatibility. | | `RELATA_LLM_BACKEND` | — | Native LLM backend: `bedrock` \| `gemini` \| `huggingface`. Leave unset for OpenAI-compatible HTTP (the default). | | `RELATA_LLM_API_KEY` | — | LLM API key. | | `RELATA_LLM_PROVIDER` | — | LLM provider hint. May be a name (`openai` \| `anthropic` \| `google` \| `hf` \| `bedrock`) or an endpoint URL. Read by the LLM dispatcher (`crates/relata-cli/src/serve/config.rs`) and the `config --migrate` map. | | `RELATA_LLM_MODEL` | — | Model name/ID to use. | | `RELATA_LLM_TIMEOUT_MS` | — | LLM request timeout in milliseconds. | | `RELATA_INFERENCE_BACKEND` | — | Inference accelerator backend (separate dispatch path from `RELATA_LLM_BACKEND`). | | `RELATA_NL_REQUIRE_LLM` | `false` | Set `true` to reject natural-language (`nl_query`) requests when no LLM is configured instead of falling back to the heuristic parser. **Startup fails on unrecognised values.** | | `RELATA_EMBED_URL` | — | Canonical embedding sidecar HTTP endpoint. Used by both the HTTP embedder (`RELATA_EMBEDDER=http`) and the media-worker learned-model path. E.g. `http://localhost:8080/embed`. **Since v1.1 the ingest hot path no longer embeds text rows** — set this so the media-worker drain cycle populates embeddings asynchronously after write returns (see [embedder sidecar](/docs/guides/llm-embedding)). Without it, the built-in CPU embedder (128-dim, deterministic) is used query-side only. Image/audio/video also need `RELATA_DECODER_ENDPOINT`. (`RELATA_ACCEL_ENDPOINT` is a deprecated alias — see the Deprecated section.) | | `RELATA_DECODER_ENDPOINT` | — | Media decoder sidecar HTTP endpoint. When unset, the media worker refuses to index media (no decode → no perceptual hashes → no embedding) rather than ship non-semantic fallback vectors. | | `RELATA_BEDROCK_URL` | — | AWS Bedrock endpoint URL. | | `RELATA_BEDROCK_API_KEY` | — | AWS Bedrock API key. | | `HF_ENDPOINT` | — | HuggingFace Inference Endpoints URL. **No `RELATA_` prefix** — read directly (upstream SDK convention; `crates/relata-intelligence/src/llm.rs`). | | `HUGGINGFACE_API_KEY` | — | HuggingFace API key. **No `RELATA_` prefix** — read directly (upstream SDK convention; `crates/relata-intelligence/src/llm_adapters.rs`). | | `GOOGLE_API_KEY` | — | Google Gemini API key. **No `RELATA_` prefix** — read directly (upstream SDK convention; `crates/relata-intelligence/src/llm_adapters.rs`). | | `RELATA_EMBEDDER` | — | Embedder type (`local` \| `openai` \| `hf` \| `http` \| `onnx`). | | `RELATA_EMBED_MODEL` | — | Embedding model name. | | `RELATA_EMBED_API_KEY` | — | Optional bearer token sent to the HTTP embedding endpoint when `RELATA_EMBEDDER=http` (`crates/relata-storage/src/embedder.rs`). Unset = no auth header. | --- ## Encryption and KMS | Variable | Default | Description | |---|---|---| | `RELATA_ENCRYPTION_AT_REST` | ON for `server`/`cluster`; OFF for `free` | Envelope-encrypts WAL + backups. **Default is profile-aware**: the production `server`/`cluster` profiles encrypt-at-rest by default (fail-closed) — set `false` to opt out (logged loudly at startup). `free` stays plaintext by default; set `true` to enable. | | `RELATA_KMS_LOCAL_DEV` | `false` | Set `true` to allow the `server` profile to fall back to the committed dev-secret KMS backend when `RELATA_KMS_KEY_ARN` is not set. For local Docker dev/CI only — **NEVER set in production**. Logs a loud warning at startup. | | `RELATA_KMS_PROVIDER` | — | KMS backend: `aws` \| `localstack` \| `vault`. | | `RELATA_KMS_KEY_ARN` | — | ARN of the KMS master key used for envelope encryption. | | `RELATA_KMS_REGION` | `RELATA_REGION` | KMS region. Defaults to `RELATA_REGION` when unset. | | `RELATA_KMS_PER_TENANT` | — | Set `true` to use distinct KMS keys per tenant/org. | | `RELATA_TOKENIZE_KEY` | — | 32-byte hex key for format-preserving tokenization of PII fields. | | `RELATA_ERASURE_SIGNING_KEY` | — | Key used to sign erasure proofs (GDPR right-to-erasure audit trail). | | `RELATA_AUDIT_HMAC_KEY` | — | HMAC-SHA256 signing key for audit-log entries (`crates/relata-cli/src/serve/cluster.rs`). When unset, the server falls back to the `RELATA_BEARER_TOKEN` bytes; on `server`/`cluster` with no token configured, startup **FATAL-exits** (no committed default — a baked-in key would allow audit-log forgery). On `free` an empty key is permitted with a loud startup warning. Also used by `relata audit verify` for offline chain verification (FATAL-exits if unset). | | `RELATA_TSA_URL` | — | RFC 3161 Time-Stamping Authority endpoint used by `CommitManifest::timestamp_head` to anchor the manifest chain head to an independent, third-party-verifiable time source. Additive to, and independent of, KMS `sign_head`. **Opt-in**: unset means no timestamp anchor and every other manifest code path is unchanged — not a startup failure (`crates/relata-prov/src/tsa.rs`, `HttpTsaClient::from_env`). | --- ## Governance and privacy | Variable | Default | Description | |---|---|---| | `RELATA_PURPOSE_MODE` | `open` | Purpose-check enforcement: `open` (default — any non-empty purpose token is accepted and recorded for audit) \| `strict` (only tokens pre-registered via `RELATA_PURPOSES` or the domain profile are accepted; others are rejected). **Startup fails on any other value.** | | `RELATA_PURPOSES` | — | Comma-separated allowed purpose strings for this deployment. | | `RELATA_DOMAIN_PROFILE` | `enterprise` | Domain preset that seeds default purposes and policies: `enterprise` \| `lea` \| `finint` \| `security` \| `custom`. **Startup fails on any other value.** | | `RELATA_TENANCY_MODE` | `single` | **Canonical tenancy switch.** `single` = single-tenant (default on every tier). `multi` is a **cluster-only** capability gated on the numeric `max_tenants` ceiling, not a license capability *string* — `free`/`server` FATAL unconditionally at startup (both are fixed at `max_tenants=1`); `cluster` with an effective `max_tenants` (license value, or the `RELATA_MAX_TENANTS` operator override) of `1` FATALs with guidance to raise the ceiling; `cluster` with `max_tenants == 0` (unlimited) or `> 1` is accepted and turns on strict isolation. Unattributed writes/reads return `400 MISSING_TENANT`-equivalent (`X-Organization-Id header is required …`) in `multi` mode. **Startup fails on unrecognised values.** ~~`RELATA_ORG_MODE`~~ and ~~`RELATA_REQUIRE_ORG`~~ are **removed** — startup FATAL if either is set. | | `RELATA_TRUST_ORG_HEADER` | `true` on free (no-auth only) | Whether to trust a client-supplied org header for tenant resolution. Unset: `true` on `free` **without** a bearer token configured (dev convenience); `false` when a token is set or on server/cluster. Set `true` explicitly to re-enable on a token-protected free deployment. | | `RELATA_AUTH_MODE` | `bearer` | Authentication mode: `bearer` \| `oidc` \| `oidc-verify` \| `saml` \| `mtls`. `none` is **removed** — startup FATAL if set. If unset and `RELATA_BEARER_TOKEN` is also unset, a random token is auto-generated and printed to stderr on first run. `oidc` is proxy-trust (an upstream gateway verifies the JWT and forwards `X-Verified-Principal`; requires `RELATA_TRUST_UPSTREAM_PROXY=true`). `oidc-verify` verifies the raw `Authorization: Bearer <jwt>` **in-process** (RS256/ES256 against the configured JWKS) — no upstream gateway required. | | `RELATA_OIDC_ISSUER` | — | Expected `iss` claim / issuer URL. Required for `oidc` and `oidc-verify`. | | `RELATA_OIDC_JWKS_URI` | — | JWKS URL used to fetch RS256/ES256 signing keys. In `oidc-verify` mode Relata fetches this itself (cached 5 min, 10 min grace window) and fails to start if the initial fetch fails. Required for `oidc` and `oidc-verify`. | | `RELATA_OIDC_AUDIENCE` | — | Expected `aud` claim value. Required for `oidc` and `oidc-verify`. | | `RELATA_OIDC_CLIENT_ID` | — | OAuth2 client id. Required for `oidc` (proxy-trust) only; not used by `oidc-verify`. | | `RELATA_SAML_IDP_ENTITY_ID` | — | SAML IdP entity id. Required for `RELATA_AUTH_MODE=saml`. | | `RELATA_SAML_IDP_SSO_URL` | — | SAML IdP single-sign-on URL. Required for `saml`. | | `RELATA_SAML_SP_ENTITY_ID` | — | SAML service-provider (Relata) entity id. Required for `saml`. | | `RELATA_SAML_ACS_URL` | — | SAML assertion-consumer-service URL. Required for `saml`. | | `RELATA_AUDIT_REDACT_PII` | — | Set `true` to redact sensitive field values from audit log entries. | | `RELATA_AUDIT_FAIL_CLOSED` | `false` | Opt-in: when `true`, a request whose audit entry could not be captured (bounded channel full AND the disk-backed spill also failed — a genuine drop, not the normal spill-absorbs-it case) is refused with `503 Service Unavailable` instead of being served unaudited. Default `false` keeps the historical fail-open behaviour (the governed action still succeeds even if its audit record could not be captured). | | `RELATA_CELL_POLICIES` | — | Path or inline JSON defining per-column cell-masking policies. | | `RELATA_TYPE_OWNERS` | — | JSON map of type → owner role for ownership enforcement. | | `RELATA_PRIVACY_DP_EPSILON` | — | Differential privacy epsilon (lower = more private, less accurate). | | `RELATA_PRIVACY_MIN_GROUP` | — | Minimum group size for DP aggregation suppression. | | `RELATA_REGION` | — | Deployment region tag (used for data-sovereignty routing and KMS). | | `RELATA_ATTESTATION_PLATFORM` | — | TEE attestation platform (`nitro` \| `sgx` \| `sev`). | | `RELATA_TENANT_QUOTAS` | — | Per-tenant cost-unit quota overrides. JSON object mapping tenant id → limit, e.g. `{"org-7": 100000, "org-9": 5000}`. Each configured tenant gets its own independent budget; unconfigured tenants fall through to the default limit. A malformed value is logged and ignored. | | `RELATA_QUERY_QUOTA_WINDOW_SECS` | — | Rolling-window length (seconds) for the per-principal read-cost quota. The quota now refills over this window instead of being a permanent cumulative cap that locks a principal out after 10k reads. | | `RELATA_PHOTODNA_HAMMING` | `10` | PhotoDNA/CSAM blocklist Hamming threshold for the ingest-side quarantine match. Default tolerates re-encode noise without over-flagging; lower = stricter. | | `RELATA_NEAR_DUP_HAMMING` | algo-keyed | Override the near-duplicate Hamming threshold. Unset keys off the algo tag (PDQ ≤ 31 / pHash ≤ 6). | --- ## Backup | Variable | Default | Description | |---|---|---| | `RELATA_BACKUP_DIR` | — | Local directory for backup snapshots. | | `RELATA_BACKUP_REPLICA_DIR` | — | Secondary backup replica directory (off-host). | | `RELATA_BACKUP_FULL_INTERVAL_SECS` | — | Interval between full backups. | | `RELATA_BACKUP_INCR_INTERVAL_SECS` | — | Interval between incremental backups. | | `RELATA_BACKUP_RETENTION_DAYS` | — | Days to retain old backups before deletion. | | `RELATA_BACKUP_TARGET` | — | Default object-store target URL (`s3://bucket/prefix`) for scheduled backups when the `--target` CLI argument is not passed (`crates/relata-cli/src/cmd_backup.rs`). Unset = no default target; the caller must supply one. | --- ## Streaming / Kafka | Variable | Default | Description | |---|---|---| | `RELATA_KAFKA_BROKERS` | — | Comma-separated Kafka broker addresses. | | `RELATA_KAFKA_TOPIC` | — | Kafka topic for ingest streaming. | | `RELATA_KAFKA_GROUP_ID` | — | Kafka consumer group ID. | | `RELATA_KAFKA_ORGANIZATION` | — | Tenant/agency tag applied to Kafka-ingested rows (mirrors the HTTP `X-Organization-Id` path). Unset/global = anonymous bucket. | | `RELATA_KAFKA_PURPOSE` | `operations` | Purpose token recorded in the audit entry for each Kafka-ingested row. | | `RELATA_KAFKA_MAX_FRAME_BYTES` | `67108864` (64 MiB) | Max byte size of a single Kafka fetch response frame. The default stays under the Kafka `recv_response` 128 MiB limit with framing overhead. Increase for high-throughput topics with large records; decrease on memory-constrained nodes. | --- ## Open Knowledge Framework (OKF) | Variable | Default | Description | |---|---|---| | `RELATA_OKF_SEED` | — | OKF seed file path for loading initial ontology entries. | | `RELATA_OKF_SOURCE` | — | OKF source identifier used in provenance tagging. | --- ## Miscellaneous | Variable | Default | Description | |---|---|---| | `RELATA_GRPC_STREAM_BATCH` | — | Number of rows per gRPC streaming batch. | | `RELATA_GRPC_TLS_CA` | — | CA path printed by `relata cluster-init` for the supervisor to forward to peers. | | `RELATA_ORPHAN_SWEEP_SECS` | `3600` | Interval for the background orphan-blob sweep. `0` disables the sweep. | | `RELATA_BLOB_REFCOUNT_PERSIST_INTERVAL_MS` | `1000` | Debounce interval for the blob-refcount snapshot. The **decrement** paths (decref / delete / sweep) write the snapshot at most once per this window instead of on every op, cutting a bulk erase/sweep from O(n) writes-per-op to O(1). **Increments** (`put_blob`) always persist synchronously and ignore this, so a still-referenced blob is never at risk. `0` disables debouncing (every decrement persists immediately — the legacy behaviour). A crash inside the window can only lose a decrement, leaving the on-disk refcount *higher* than reality (a leak the orphan sweeper reclaims), never lower. | | `RELATA_OBJECT_PUT_TIMEOUT_SECS` | `30` | Timeout for every remote object-store PUT (`store/remote_io.rs:1108`). Covers WAL segment flush, Parquet segment flush, manifest put, and blob put. Raise when running against a high-latency object store. | | `RELATA_DEDUP_TOKEN_MIN_AGE_SECS` | `86400` | Minimum age (seconds) before a dedup-token entry can be evicted by the TTL/LRU sweeper. Prevents replay-defence tokens from being reclaimed too soon (`serve.rs:1702`). | | `RELATA_LOG_TARGETS` | — | Per-module log-level override via `EnvFilter` directives, e.g. `relata_storage=debug,relata_query=warn`. Overrides `RELATA_LOG_LEVEL` for the named crates. | | `RELATA_MAX_TENANTS` | — | Overrides the license's `max_tenants` ceiling. Useful for testing, demos, or dev instances where the embedded license field is too small. Unset = use the license value. | | `RELATA_FANOUT_MAX_OFFSET` | `100000` | Largest `OFFSET` the cluster coordinator will inflate-and-slice for a fan-out query. Each shard is sent `LIMIT offset+limit` with `OFFSET` stripped; above this bound the query stays fail-closed (`503`) rather than pulling that many rows per shard across the network just to discard most of them. | | `RELATA_FANOUT_PARALLEL_MERGE_THRESHOLD` | `5000` | Row-count threshold above which the cluster coordinator's merge (GROUP BY re-grouping, global sort) switches to a rayon-parallel path. Also gated on `rayon::current_num_threads() >= 4`; below either gate the sequential path is used unchanged. | --- ## Deprecated / removed | Variable | Status | Replacement | |---|---|---| | `RELATA_ALLOWED_ORIGINS` | **Removed** — startup FATAL if set | Use `RELATA_CORS_ALLOWED_ORIGINS` | | `RELATA_REQUIRE_MTLS` | **Removed** — startup FATAL if set | Set `RELATA_AUTH_MODE=mtls` | | `RELATA_MAX_CONNECTIONS` | **Removed** — startup FATAL if set | Use `RELATA_HTTP_MAX_CONNS`. Previously silently stacked a second, lower-default `ConcurrencyLimitLayer` on top of `RELATA_HTTP_MAX_CONNS`'s — not just an unused alias, an active double-limiter bug. | | `RELATA_S3_BACKEND` | **Deprecated** — presence-only check | Use `AWS_ENDPOINT_URL` or `RELATA_OBJECT_STORE` to configure the object store backend. Setting this variable only triggers a warning when `RELATA_IN_MEMORY=true` is also set. ⚠ still read at `crates/relata-storage/src/remote.rs:236` — removal tracked separately. | | `RELATA_STATUS_URL` | **Deprecated alias** → `RELATA_URL` | ⚠ still read at `crates/relata-cli/src/main.rs:2097`, `crates/relata-cli/src/main.rs:2212` — removal tracked separately. | | `RELATA_ACCEL_ENDPOINT` | **Deprecated alias** → `RELATA_EMBED_URL` | ⚠ still read at `crates/relata-intelligence/src/accel.rs:361,476,598`, `crates/relata-cli/src/serve.rs:19790` (SSRF guard), `crates/relata-cli/src/serve.rs:20587` — removal tracked separately. | | `RELATA_SEARCH_SHORT_TERM_EXPANSION` | **Removed** | Sub-3-char query terms now always fall back to `prefix_search`; the expansion is no longer env-gated. | | `RELATA_SEARCH_SHORT_TERM_SCAN_CAP` | **Removed** | The prefix-scan path has no configurable scan cap. | | `RELATA_ALERT_MIN_SEVERITY` | `medium` | Minimum severity for webhook alert delivery (low, medium, high, critical). Alerts below this are stored but not pushed. | | `RELATA_ALERT_WEBHOOKS` | — | Comma-separated webhook URLs for alert delivery (PagerDuty/Slack/email gateway). System-level fallback; tenant-specific rules via NotificationRule. | | `RELATA_CLUSTER_ID` | — | UUID identifying this cluster. Generated at cluster init; shared by all nodes. Used for license binding + cross-cluster protection. | | `RELATA_CLUSTER_TIER` | — | Cluster licensing tier: small, medium, large, enterprise. Sets max_nodes ceiling. | | `RELATA_COLUMNAR_OVERRIDE` | — | Force-enable or force-disable columnar analytical reads regardless of profile. `true` or `false`. | | `RELATA_COORDINATOR_ADDR` | — | Address of the cluster coordinator for auto-join. Set on reader/writer/indexer nodes. | | `RELATA_MAX_NODES` | — | Maximum nodes this cluster supports (from license). Nodes beyond this are rejected at gossip join. | | `RELATA_CACHE_RAM_MB` | **Renamed** | Use `RELATA_STORE_MAX_RAM_MB` (row-store RAM budget). | | `RELATA_COORDINATOR` | **Output only** | Printed by `relata cluster-init`; not read as input. Use `RELATA_COORDINATOR_ADDR` on peer nodes. | | `RELATA_INGEST_QUEUE_CAPACITY` | **Renamed** | Use `RELATA_INGEST_QUEUE_MAX_BYTES`. The old name appears in diagnostic JSON only and is not read from the environment. | | `RELATA_NODE_ID` | — | Override the persistent deployment UUID shown in the startup banner. Set this in Docker/container environments where `$HOME` is read-only and the file-backed UUID cannot be persisted. | ============================================================================== # Error codes reference URL: https://relatadb.dev/docs/reference/error-codes ============================================================================== # Error codes reference Every query and HTTP error in Relata carries a structured RFC 7807 `application/problem+json` response. This page enumerates the error variants, their causes, and the suggested fix. > **Deep-linkable errors.** Every error response carries a `type` URI of the form > `https://relatadb.dev/errors/`, and each code resolves to a dedicated page > with the cause and remediation. Look up any code (in any form the server emits) > at **[relatadb.dev/errors](/errors)** — or click the code column below. ## RFC 7807 response shape ```http HTTP/1.1 400 Bad Request Content-Type: application/problem+json { "type": "https://relatadb.dev/errors/parse", "title": "Bad Request", "status": 400, "detail": "unexpected token 'SELEC' after query end", "instance": "/query", "correlation_id": "0192a4f8-7e1c-7c3a-9b02-2c4d5e6f7a8b" } ``` | Field | Meaning | |---|---| | `type` | URI identifying the error class — **stable across releases** | | `title` | HTTP status phrase (per RFC 9110) | | `status` | HTTP status code | | `detail` | Human-readable cause (English; may change between releases) | | `instance` | The route that produced the error | | `correlation_id` | UUIDv7 — cite this in ops tickets | ## HTTP status semantics | Status | Meaning | Retry? | |---|---|---| | 400 | Bad Request — client-side problem (parse error, bad argument) | No — fix the request | | 401 | Unauthorized — missing or invalid bearer token | No — supply a token | | 403 | Forbidden — ACL denied access (the principal lacks a permission) | No — request access | | 404 | Not Found — the type, route, or backup artefact does not exist | No | | 405 | Method Not Allowed — wrong HTTP verb on a known route | No | | 409 | Conflict — write lease held by another node; or row already exists | Yes, after lease TTL | | 413 | Payload Too Large — exceeded `DefaultBodyLimit` | No — chunk the request | | 422 | Unprocessable Entity — semantically invalid (e.g. backup SHA mismatch) | No | | 429 | Too Many Requests — admission control or rate limit hit | Yes, after `Retry-After` — see [Rate-limit headers](#rate-limit-headers) | | 500 | Internal Server Error — server-side bug or storage failure | Maybe, with backoff | | 501 | Not Implemented — recognised but unimplemented feature | No | | 503 | Service Unavailable — server is draining or shutting down | Yes, after restart | ## `QueryError` variants (canonical list) These are the variants of the query error type that surface as RFC 7807 errors. | Variant | `code` | HTTP | Cause | Suggested fix | |---|---|---|---|---| | `MissingPurpose` | [`REL_PURPOSE`](/errors/REL_PURPOSE) | 403 | Query lacks `PURPOSE` (only raised in strict mode — `RELATA_PURPOSE_MODE=strict`) | Prefix the query with `PURPOSE ''` or relax the mode | | `Timeout` | [`REL_TIMEOUT`](/errors/REL_TIMEOUT) | 408 / 504 | Execution exceeded `RELATA_QUERY_TIMEOUT_SECS` | Add `LIMIT`, add an index, or raise the timeout | | `WatchLimit` | [`REL_WATCH_LIMIT`](/errors/REL_WATCH_LIMIT) | 429 | `RELATA_MAX_WATCH_SUBSCRIPTIONS` reached | Close unused subscriptions or raise the cap | | `UnknownPurpose` | [`REL_PURPOSE_UNKNOWN`](/errors/REL_PURPOSE_UNKNOWN) | 403 | Purpose not registered in `PurposeRegistry` | Register it via the config, or use a known purpose | | `ParseError` | [`REL_PARSE`](/errors/REL_PARSE) | 400 | SQL / Cypher / GQL syntax error | Fix the syntax (v1.5.0 adds line:column) | | `UnknownType` | [`REL_UNKNOWN_TYPE`](/errors/REL_UNKNOWN_TYPE) | 404 | Object type not registered | `POST /types` to register it | | `UnknownColumn` | [`REL_UNKNOWN_COLUMN`](/errors/REL_UNKNOWN_COLUMN) | 400 | Column not in the type's declared contract | Check the type definition or remove the column | | `AccessDenied` | [`REL_ACL`](/errors/REL_ACL) | 403 | ACL denied the principal access | Grant the permission via the policy table | | `Storage` | [`REL_STORAGE`](/errors/REL_STORAGE) | 500 | Storage error (disk full, manifest corruption, etc.) | Check `/debug/stats`; contact ops | | `HumintProtectedType` | [`REL_HUMINT`](/errors/REL_HUMINT) | 403 | Tried to read a HUMINT-protected type without break-glass | Use the break-glass unmask flow | | `MissingAccessScope` | [`REL_ACCESS_SCOPE`](/errors/REL_ACCESS_SCOPE) | 403 | Type requires `access_scope_ref` | Pass the scope reference | | `CrossOrganizationAccessDenied` | [`REL_XORG`](/errors/REL_XORG) | 403 | No `SharingAgreement` covers this cross-org access | Establish a sharing agreement | | `QuotaExceeded` | [`REL_QUOTA`](/errors/REL_QUOTA) | 429 | Tenant admission quota exceeded | Wait, or raise the tenant quota | | `ResultCapExceeded` | [`REL_RESULT_CAP`](/errors/REL_RESULT_CAP) | 406 | Result exceeded `RELATA_MAX_RESULT_ROWS` | Add `LIMIT`, or raise the cap (carefully) | | `VectorBudgetExceeded` | [`REL_VECTOR_BUDGET`](/errors/REL_VECTOR_BUDGET) | 406 | Vector `k` exceeds `RELATA_MAX_VECTOR_K` | Lower the `LIMIT`, or raise/disable the budget via `RELATA_MAX_VECTOR_K` | | `MaskedColumnInAggregate` | [`REL_MASKED_COLUMN`](/errors/REL_MASKED_COLUMN) | 400 | ACL-masked column used as aggregate/sort/group target | Remove the masked column from aggregates | | `QueueFull` | [`REL_QUEUE_FULL`](/errors/REL_QUEUE_FULL) | 429 | All query priority queues are at capacity | Retry with backoff | | `ProtectedTypeColumn` | [`REL_PROTECTED_COLUMN`](/errors/REL_PROTECTED_COLUMN) | 403 | Column name references a protected type | Remove the protected column from the query | | `InternalError` | [`REL_INTERNAL`](/errors/REL_INTERNAL) | 500 | Bug; the `detail` is masked in `client_safe_msg` | File an issue with the `correlation_id` | ## Rate-limit headers All 429 responses from the per-IP token-bucket rate limiter include the following headers: | Header | Value | Description | |---|---|---| | `Retry-After` | seconds (integer) | Minimum seconds before retrying — always `1` for the normal rate-limit bucket | | `X-RateLimit-Limit` | integer | Requests-per-second quota that applies to this endpoint | | `X-RateLimit-Remaining` | integer | Tokens remaining in the current window — always `0` on a 429 | | `X-RateLimit-Reset` | Unix epoch seconds | When the next token becomes available | Example 429 response: ```http HTTP/1.1 429 Too Many Requests Content-Type: application/problem+json Retry-After: 1 X-RateLimit-Limit: 10000 X-RateLimit-Remaining: 0 X-RateLimit-Reset: 1753128001 { "type": "https://relatadb.dev/errors/REL_RATE_LIMITED", "title": "Too Many Requests", "status": 429, "detail": "Per-IP rate limit exceeded for /query.", "instance": "/query", "retryable": true } ``` SDK clients should inspect `Retry-After` and back off before retrying. Quota values are controlled by `RELATA_RATE_LIMIT_RPS`, `RELATA_READ_RATE_LIMIT_RPS`, and `RELATA_MEMORY_RATE_LIMIT_RPS` (see [Environment Variables](/docs/reference/env-vars)). ## SDK error mapping Each SDK maps the RFC 7807 response to a typed exception: ### Python ```python from relata.exceptions import ( RelataParseError, RelataAccessDenied, RelataQuotaExceeded, RelataTimeout, RelataResultCapExceeded, RelataError, ) try: client.query("SELEC * FROM Person") except RelataParseError as e: print(e.detail) # "unexpected token 'SELEC' after query end" print(e.correlation_id) # "0192a4f8-..." except RelataAccessDenied as e: print(f"missing permission: {e.detail}") except RelataError as e: # catch-all parent print(e.type_uri, e.status) ``` ### TypeScript ```typescript import { RelataParseError, RelataAccessDenied, RelataQuotaExceeded, RelataError, } from "@zysec-ai/relata-sdk"; try { await relata.query({ sql: "SELEC * FROM Person" }); } catch (e) { if (e instanceof RelataParseError) { console.error(e.detail); // string console.error(e.correlationId); // UUIDv7 } else if (e instanceof RelataError) { console.error(e.typeUri, e.status); } } ``` ### Go ```go import "github.com/relatadb/RelataDB/sdks/go/relata" var errRelata *relata.Error if errors.As(err, &errRelata) { log.Printf( "type=%s status=%d detail=%s correlation=%s", errRelata.Type, errRelata.Status, errRelata.Detail, errRelata.CorrelationID, ) } ``` ## Common error scenarios ### "no PURPOSE declared" You're in strict mode (`RELATA_PURPOSE_MODE=strict`). Either prefix your query with `PURPOSE ''` or set `RELATA_PURPOSE_MODE=open` for dev. ### "access denied: principal 'X' cannot perform 'read' on 'Person'" The principal's ACL role lacks `read` on `Person`. Add a policy: ```sql PURPOSE 'admin' INSERT INTO Policy (principal, permission, object_type) VALUES ('alice@example.com', 'read', 'Person') ``` ### "WATCH rejected: subscription limit reached" Each WATCH subscription re-evaluates on every commit, so the cap (`RELATA_MAX_WATCH_SUBSCRIPTIONS`) protects commit latency. Close unused subscriptions, or raise the cap on a write-light workload. ### "ResultCapExceeded" Your query result is larger than `RELATA_MAX_RESULT_ROWS` (default 1,000,000). Add `LIMIT N`, narrow the WHERE, or — for legitimate bulk exports — use `GET /export` instead of `/query`. ### "VectorBudgetExceeded" Vector search clamps the requested `k` to `RELATA_MAX_VECTOR_K` (default 1000). Lower the requested `LIMIT`, or raise/disable the cap via `RELATA_MAX_VECTOR_K=0`. ## Reporting errors When filing an issue, always include: 1. The `correlation_id` (UUIDv7) from the response. 2. The `type` URI. 3. The full `detail` string. 4. The server version (`/version`). 5. The route (`instance` field). 6. The query or request body that triggered it. Without the `correlation_id`, ops cannot trace the request through the logs. ## See also - [RFC 7807 — Problem Details for HTTP APIs](https://www.rfc-editor.org/rfc/rfc7807) - [HTTP API reference / SDK guide](/docs/sdks/overview) ============================================================================== # Graph analytics — one engine, ten+ algorithms, three query languages URL: https://relatadb.dev/docs/reference/graph-analytics ============================================================================== # Graph analytics — one engine, ten+ algorithms, three query languages RelataDB builds the graph **for you** out of standardized identities (any two records sharing a validated identifier are auto-linked), then lets you run production graph analytics over it without a separate Neo4j/TigerGraph/GDS product. The same governed plan that runs your SQL `SELECT` also runs PageRank, community detection, shortest path, and cycle detection — with ACL, org isolation, and provenance firing identically. You can query the graph three ways: **SQL TVFs** (first-class), **`CALL traverse.`** (native Cypher/GQL procedures), or **`CALL gds.`** (Neo4j GDS portability alias — your existing `gds.*` scripts port with minimal change). > **Why this matters:** in a polyglot stack you'd run graph analytics in a separate Neo4j + GDS instance, ETL the rows over, and lose governance + identity + temporal joins along the way. Here, graph analytics is a query against the same governed store — `PATHS_BETWEEN('alice','bob')` and `SELECT * FROM PAGERANK('Person','KNOWS')` compose with `AS OF`, `PURPOSE`, and cell-level ACL. ## The algorithm surface | Algorithm | SQL TVF | `gds.*` / `traverse.*` | What it answers | |---|---|---|---| | **PageRank** | `GRAPH_PAGERANK('Type','LABEL', DAMPING => 0.85, MAX_ITER => 20)` | `gds.pageRank.{stream\|stats\|write}` | Who are the influential nodes? | | **Degree centrality** | `DEGREE_CENTRALITY(...)` | `gds.degreeCentrality.*` | Who has the most connections? | | **Triangle count** | `TRIANGLE_COUNT(...)` | `gds.triangleCount.*` | How clustered is the network? | | **Connected components / WCC** | `CONNECTED_COMPONENTS(...)` | `gds.wcc.*` | Which nodes form one island? | | **Label propagation (community)** | `LABEL_PROPAGATION(...)` | `gds.labelPropagation.*` | What communities self-organize? | | **Louvain community** | community detection TVF | — | Higher-quality community detection | | **Strongly connected components (SCC)** | `SCC(...)` | — | Mutually-reachable clusters | | **Cycle detection** | `CYCLES(...)` | — | Where's the feedback loop? | | **Shortest path (SSSP)** | `PATHS_BETWEEN('a','b', max_hops)` + PLL index | — | How are A and B connected? | | **All-pairs shortest path (APSS)** | — | — | Distance matrix across the graph | | **Spanning tree / diameter** | — | — | Backbone + reachability radius | | **Node similarity** | `GRAPH_NODE_SIMILARITY('Type', node)` | — | Nodes structurally like X | | **Link prediction** | `LINK_PREDICT('Type')` | — | Likely missing edges | | **HubAuthority (HITS)** | — | via MCP `hub_authority` | Hubs vs authorities | > **Incremental warm-start:** PageRank / WCC / label-propagation variants avoid full recompute on a small edge delta — they pick up from the prior score vector. Cheap "add one edge, get new scores." ## Three ways to call them ### 1. SQL TVF (first-class — composes with everything) ```sql PURPOSE 'analytics' SELECT id, pagerank FROM GRAPH_PAGERANK('Person', 'KNOWS', DAMPING => 0.85, MAX_ITER => 20) ORDER BY pagerank DESC LIMIT 10; ``` ```sql -- Composes with temporal + identity predicates, same query PURPOSE 'investigation' SELECT id, pagerank FROM GRAPH_PAGERANK('Person', 'KNOWS') WHERE id IN ( SELECT object_id FROM IdentityIndex AS OF '2026-01-01T00:00:00' WHERE payload = '+44 7700 900123' ) ORDER BY pagerank DESC; ``` ### 2. `CALL traverse..` (native Cypher/GQL procedure) ```cypher // Over Bolt (port 7687) with the official Neo4j driver, or POST /query CALL traverse.pageRank.stream('Person', {maxIterations: 20, dampingFactor: 0.85}) YIELD nodeId, score RETURN nodeId, score ORDER BY score DESC LIMIT 10 ``` ### 3. `CALL gds..` (Neo4j GDS portability — port existing scripts) ```cypher // Identical shape to Neo4j GDS — minimal rewrite to port an existing pipeline CALL gds.pageRank.stream('myGraph', {maxIterations: 20, dampingFactor: 0.85}) YIELD nodeId, score RETURN nodeId, score ORDER BY score DESC LIMIT 10 ``` Supported `gds.*` procedures (mode defaults to `stream` when omitted): `gds.pageRank.{stream,stats,write}`, `gds.degreeCentrality.{stream,stats,write}`, `gds.triangleCount.*`, `gds.wcc.*`, `gds.labelPropagation.*`. An unrecognized `gds.` returns a typed error pointing at the equivalent SQL TVF to use directly (`use GRAPH_PAGERANK / DEGREE_CENTRALITY / TRIANGLE_COUNT / CONNECTED_COMPONENTS / LABEL_PROPAGATION SQL operators directly`). ## From the SDKs All three SDKs expose the graph operators as one-shot methods (they compile to the SQL TVFs server-side): ```python # Python — page rank over the Person/KNOWS graph pr = client.graph_pagerank("Person", damping=0.85, max_iter=20, purpose="analytics") # → [{"id": "p1", "score": 0.18}, ...] # Shortest path between two entities (PLL-indexed) path = client.graph_shortest_path("alice-id", "bob-id", purpose="investigation") # Communities comms = client.graph_community("Person", purpose="analytics") ``` ```typescript // TypeScript const pr = await relata.graphPageRank("Person", { damping: 0.85, maxIter: 20, purpose: "analytics" }); const path = await relata.graphShortestPath("alice-id", "bob-id"); ``` ```go // Go pr, _ := client.GraphPageRank(ctx, "analytics", "Person", &relata.GraphPageRankOptions{Damping: 0.85, MaxIter: 20}) path, _ := client.GraphShortestPath(ctx, "alice-id", "bob-id", &relata.GraphShortestPathOptions{}) ``` ## MCP tools (for agent-driven investigation) ```python # rank_key_nodes — "who are the influencers in this Person graph?" mcp.call_tool("rank_key_nodes", {"entity_type": "Person", "metric": "pagerank"}) # detect_communities — "show me the clusters" mcp.call_tool("detect_communities", {"entity_type": "Person", "algo": "louvain"}) # predict_links — "what edges are likely missing?" mcp.call_tool("predict_links", {"entity_type": "Person"}) # find_scc, hub_authority, paths_between, find_connections ... ``` The full MCP surface: `rank_key_nodes`, `detect_communities`, `predict_links`, `find_scc`, `hub_authority`, `paths_between`, `find_connections`, `get_relationships`. See [MCP Tools](/docs/reference/mcp-tools). ## GraphTrigger — edges from rows, automatically Don't want to manage edges at all? Declare a `graph_triggers` block on the type and the graph builds itself from the rows you were ingesting anyway: ```bash curl -X POST http://127.0.0.1:9090/types \ -d '{ "name": "CdrRecord", "graph_triggers": [ {"link_type": "CALLED", "src_field": "caller_id", "dst_field": "callee_id"} ] }' ``` Every `CdrRecord` insert now also creates a governed `CALLED` edge between the caller and callee — no separate edge-loading pipeline, no server restart. `PATHS_BETWEEN` and Cypher read the edge natively. ## Tips & takeaways - **PageRank's defaults are sane.** `DAMPING => 0.85, MAX_ITER => 20` is the textbook starting point; raise `MAX_ITER` only if scores haven't converged (the response tells you). - **Use `PATHS_BETWEEN` for "how are these two connected?"** It's PLL-indexed — sub-millisecond on typical graphs, no traversal cost. - **Community detection ≠ clustering.** Label propagation is fast and deterministic; Louvain finds higher-modularity communities but costs more. Try both. - **Graph + temporal is the killer combo.** `GRAPH_PAGERANK(...) AS OF ''` answers "who was influential at the time of the incident?" — impossible in a polyglot stack without snapshots. - **`gds.*` is for portability, `traverse.*` is native.** Both lower to the same governed plan. Use `gds.*` when porting existing Neo4j workloads; switch to SQL TVFs or `traverse.*` for new code (cleaner governance + composition). - **Link prediction surfaces likely missing edges** — great for "who probably knows whom" in OSINT / AML work, but treat the output as leads, not facts (no provenance on a predicted edge until you promote it to a real one). ## See also - [Cypher & SQL-PGQ](/docs/reference/cypher) — `MATCH` queries over the same graph - [SQL reference](/docs/reference/sql) — `PATHS_BETWEEN`, `LOOKUP_IDENTITY`, `RESOLVE_IDENTITY` - [Identity](/docs/concepts/identity) — how the graph forms itself from standardized identities - [Hybrid search](/docs/concepts/hybrid-search) — graph rank as the third RRF signal - [MCP tools](/docs/reference/mcp-tools) — the graph investigation verbs - [Use cases](/docs/use-cases/aml-sanctions-screening) — AML, LEA, maritime, OSINT ============================================================================== # GraphQL URL: https://relatadb.dev/docs/reference/graphql ============================================================================== # GraphQL Relata exposes a `POST /graphql` endpoint that translates a subset of GraphQL query syntax to Relata SQL and executes it through the same governed query path as `/query` (ACL, cell masking, tenant scoping, cluster fan-out). The translator is **hand-rolled with no external GraphQL dependency** — supply-chain safe by design. > **Conformance:** GraphQL query subset — field selection, `limit`, `where` (single equality), `__schema`/`__type` introspection. Mutations, subscriptions, fragments, unions, and interfaces are not supported. For the full query surface, use [SQL](/docs/reference/sql). ## Endpoint | Method | Path | Body | |---|---|---| | `POST` | `/graphql` | `application/json` — `{"query": "...", "variables": {...}}` | There is no `GET` form; `/graphql` accepts `POST` only. ## Authentication Same as `/query`. When `RELATA_BEARER_TOKEN` is set, include it: ```bash curl -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"query": "{ Person { id name } }"}' \ http://localhost:9090/graphql ``` Unauthenticated requests receive HTTP 401 (`application/problem+json`). ## Request body | Field | Required | Notes | |---|---|---| | `query` | yes | GraphQL query string. Empty → HTTP 400. | | `variables` | no | JSON object of `$var` bindings, bound server-side (see [Variables](#variables)). | `PURPOSE` is read from the `x-relata-purpose` header (defaults to `analytics`) and prepended to the translated SQL, so every `/graphql` read is governed and audited exactly like a purpose-tagged SQL query. ## Usage ### Field selection ```bash curl -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"query": "{ Person { id name email } }"}' \ http://localhost:9090/graphql ``` Translates to `SELECT id, name, email FROM Person LIMIT 100` (default limit 100). ### `limit` and `where` ```bash -d '{"query": "{ Person(limit: 5, where: {name: \"Alice\"}) { id name } }"}' ``` Translates to `SELECT id, name FROM Person WHERE name = 'Alice' LIMIT 5`. The `where` argument accepts a single `field = value` equality predicate. String values are escaped to prevent injection; field names are restricted to `[A-Za-z0-9_]`. ### Variables `$var` references in `limit` and `where` are bound from the `variables` object (#3260). A variable-definitions block (`query Q($n: String) { ... }`) is accepted and ignored — values always come from `variables`. ```bash -d '{ "query": "query ByName($n: String, $lim: Int) { Person(limit: $lim, where: {name: $n}) { id name } }", "variables": {"n": "Alice", "lim": 5} }' ``` Translates to `SELECT id, name FROM Person WHERE name = 'Alice' LIMIT 5`. The whole `where` filter can also be a variable: ```bash -d '{ "query": "{ Person(where: $f) { id name } }", "variables": {"f": {"name": "Alice", "age": 30}} }' ``` Object variables are conjoined with `AND` (`name = 'Alice' AND age = 30`). **Never silent.** A `$var` that is referenced but missing from `variables`, or whose value is null or wrong-typed (e.g. a string where `limit` needs an integer), returns an error envelope — the filter is never silently dropped. String variable values are bound as quoted literals, so variable content cannot inject SQL. ### Introspection ```bash -d '{"query": "{ __schema { types { name } } }"}' ``` `__schema` / `__type` queries return the ontology's object type names: ```json { "data": { "__schema": { "types": [ {"name": "Person"}, {"name": "Organization"} ] } }, "errors": [] } ``` ## Response format Every response uses the standard GraphQL envelope: ```json { "data": [ {"id": "...", "name": "Alice"} ], "errors": [] } ``` - On success, `data` is an array of row objects (capped by `RELATA_MAX_RESULT_ROWS`); `errors` is `[]`. - On a translator or policy error, HTTP 400/500 with `{"data": null, "errors": [{"message": "..."}]}`. - Malformed JSON or a missing `query` field returns an RFC 7807 `application/problem+json` error. ## Supported subset | Construct | Status | |---|---| | Field selection `{ Type { f1 f2 } }` | Supported | | `query { ... }` wrapper or bare `{ ... }` | Supported | | `limit: N` argument (default 100) | Supported | | `where: { field: "value" }` (single equality) | Supported | | `where: $objVar` (whole-filter object variable, AND-joined) | Supported | | `__schema` / `__type` introspection sentinel | Supported (type listing) | | `mutation { ... }` | 400 — "mutations not yet supported" | | Subscriptions, fragments, unions, interfaces | Not supported (parse error) | | Variable binding (`$var` in `limit` / `where`) | Supported (#3260) — missing/mistyped variable is a hard error | | Full GraphQL spec compliance | Deferred — use `/query` for the full SQL surface | ## Cluster fan-out When this node is a cluster coordinator with peers, the translated SQL is routed through the cluster-fan-out-aware path so results span all shards. A standalone node executes locally. The response envelope is identical either way. ## SDK usage The published SDKs expose the door directly — e.g. the Python SDK: ```python # Field selection. rows = client.graphql("{ Person(limit: 5) { id name } }") # Variables are bound server-side (#3260); a missing/mistyped variable raises. rows = client.graphql( "query ByName($n: String) { Person(where: {name: $n}) { id name } }", variables={"n": "Alice"}, ) ``` ## Configuration None required. `/graphql` is available on every deployment profile (`free`, `server`, `cluster`). Auth is mandatory on `server`/`cluster` and optional on `free`, identical to `/query`. ## See also - [SQL Reference](/docs/reference/sql) — the full dialect (use this beyond the subset above) - [SPARQL](/docs/reference/sparql) — the other query-language door - [Protocols](/docs/reference/protocols) — the full protocol-door catalogue ============================================================================== # Reference URL: https://relatadb.dev/docs/reference ============================================================================== # Reference Complete API and protocol reference for RelataDB. ## In this section **Query languages** - [SQL Reference](/docs/reference/sql) — full SQL grammar, operators, and temporal extensions - [Cypher & SQL-PGQ](/docs/reference/cypher) — graph pattern matching over the link store - [SPARQL](/docs/reference/sparql) — RDF-style triple queries over the knowledge graph - [GraphQL](/docs/reference/graphql) — GraphQL query subset with server-side variable binding - [Graph Analytics](/docs/reference/graph-analytics) — GDS-style algorithms (PageRank, communities, paths) - [Query Cookbook](/docs/reference/query-cookbook) — worked recipes across the query surface **Retrieval & time** - [Search & Retrieval](/docs/reference/search) — BM25 full-text, vector, and hybrid search - [Vector Index Parameters](/docs/reference/vector-params) — HNSW/DiskANN/IVF tuning knobs - [Bi-temporal Queries](/docs/reference/bitemporal) — AS OF valid-time and system-time queries - [Branching & Namespaces](/docs/reference/branching) — isolated query/write branches - [Agent Memory](/docs/reference/agent-memory) — memory verbs for AI agents **Platform** - [HTTP API](/docs/reference/api-reference) — the full REST route table - [MCP](/docs/reference/mcp-tools) — Model Context Protocol tools and memory verbs - [Protocols](/docs/reference/protocols) — Postgres wire, gRPC, Arrow Flight, S3, Redis, Mongo, ClickHouse, Neo4j - [Error Codes](/docs/reference/error-codes) — typed errors and RFC 9457 problem details - [Environment Variables](/docs/reference/env-vars) — all configuration knobs - [Limits](/docs/reference/limits) — the honest, code-verified picture of what works - [Performance Tuning](/docs/reference/performance) — caches, spill, and query budgets - [Licensing & Tiers](/docs/reference/licensing) — feature gating by license ============================================================================== # Licensing & Tiers URL: https://relatadb.dev/docs/reference/licensing ============================================================================== # Licensing & Tiers RelataDB ships as a single binary with three deployment profiles — `free`, `server`, and `cluster`. The engine is **identical on every profile**: every protocol door, at-rest encryption, disk-first paging, columnar Arrow, bi-temporal, identity, provenance, governance, MCP, and hybrid search are available everywhere. A license controls exactly **two numbers**: | Parameter | Type | Meaning | |---|---|---| | `storage_max_gb` | `u64` | Storage cap in GB. `0` = unlimited. | | `max_tenants` | `u32` | Maximum tenant count. `0` = unlimited. | | | **free** (no license) | **server** | **cluster** | |---|---|---|---| | Engine / doors / governance | full | full | full | | **Storage** (`storage_max_gb`) | 10 GB (fixed) | license value (`0` = unlimited) | license value (`0` = unlimited) | | **Tenants** (`max_tenants`) | 1 (`"default"`) | **1** (enforced) | license value (`0` = unlimited) | | **Tenancy mode** | single | single | single (1) or multi (>1) | | **Nodes** | 1 | 1 | many (multi-node) | **Multi-tenancy is a cluster-only capability.** `RELATA_TENANCY_MODE=multi` is rejected at startup on `free` / `server` (which are fixed at 1 tenant); it is accepted on `cluster` when `max_tenants > 1`. Bind address, TLS, and rate-limit defaults are profile-scoped **overridable defaults**, not license gates — every door honors `RELATA_*_BIND` on every profile. --- ## How the storage cap is enforced Every `POST /ingest` measures `total_stored_bytes` — the sum of RAM-resident row payload bytes **plus** disk-spilled Parquet segment bytes. On `free` the cap is 10 GB; on a licensed node it is the license's `storage_max_gb` (`0` = unlimited). Double-counting is not possible: rows move atomically from RAM to disk when the RAM cap is hit. | Threshold | Behaviour | |---|---| | < 90% of cap | Write accepted silently | | ≥ 90% | Write accepted; server logs a soft-warn | | ≥ 100% | Write rejected — **HTTP 402 Payment Required** | The Prometheus gauge `relata_store_total_stored_bytes` tracks current consumption so you can alert before hitting the wall. --- ## Activating a license When you purchase or renew a RelataDB license, you receive an activation blob from Relata. Install it on the node that will run the server: ```bash relata init --config-data ``` The binary verifies the license, writes it to `~/.relata/node.dat`, and reads it at startup. Run `relata init` before `relata serve` on a fresh node, or restart the server to pick up a renewed/changed license. You can inspect the active license at any time: ```bash relata config --print-template # shows the resolved NodeConfig curl localhost:9090/platform/license # the server's reported tier + caps ``` To obtain a license, renew, or change tier (storage, tenant count, region count), contact Relata support — licenses are issued and signed by Relata and cannot be self-generated. --- ## Grace, expiry, and tamper protection - **Expiry.** A license carries an expiry timestamp. Past expiry, the server continues running under a short grace window (loud startup warning), then refuses new writes while keeping reads available so you can export your data. - **Grace tokens.** Relata may issue short grace extensions for renewals in flight; the operator applies them with `relata init` exactly like a full license. - **Tamper protection.** Licenses are cryptographically signed by Relata and verified at startup. A modified, forged, or downgraded license is rejected; the engine never silently falls back to a higher tier than the signed license allows. Anti-escalation is enforced server-side — a trial/free-tier signature cannot unlock server or cluster caps. --- ## Capabilities by tier The engine is full on every tier. The license numbers (`storage_max_gb`, `max_tenants`) and the cluster-only multi-tenant capability are the only things that differ. | Capability | free | server | cluster | |---|:---:|:---:|:---:| | Relational · graph · vector · full-text · hybrid search | ✓ | ✓ | ✓ | | Bi-temporal · identity · provenance · governance · MCP | ✓ | ✓ | ✓ | | All protocol doors (S3, ClickHouse, Neo4j, Redis, Mongo, Bolt, Flight, pgwire) | ✓ | ✓ | ✓ | | At-rest encryption · disk-first paging · columnar Arrow | ✓ | ✓ | ✓ | | Agent memory · cognitive verbs · framework adapters | ✓ | ✓ | ✓ | | **Multi-tenant** (strict isolation) | — | — | ✓ (`max_tenants > 1`) | | Multi-region · sharding · hot-standby | — | — | ✓ | | **Storage cap** | 10 GB | `storage_max_gb` | `storage_max_gb` | | **Tenant cap** | 1 | 1 | `max_tenants` | --- ## Prometheus monitoring | Metric | Type | Description | |---|---|---| | `relata_store_total_stored_bytes` | gauge | Total stored bytes (hot + spilled) — metered against `storage_max_gb` | | `relata_store_disk_segment_bytes` | gauge | Bytes spilled to disk segments | | `relata_store_disk_segment_count` | gauge | Number of on-disk segments | ============================================================================== # Limits & Caveats URL: https://relatadb.dev/docs/reference/limits ============================================================================== # Limits & Caveats This is the code-verified status of every major feature — what works, what is partial, and what is explicitly deferred. > The governed core is real: bi-temporal store, planner with ACL and org isolation, provenance / audit hash chain, SmartIngest identity detection, and all protocol-compatibility doors are implemented and smoke-tested. ## SQL query surface | Feature | Status | Notes | |---|---|---| | `SELECT` + `WHERE` / `ORDER BY` / `LIMIT` / `JOIN` / `GROUP BY` | Working | `JOIN` = INNER hash join (O(n+m)); `GROUP BY` + `COUNT(*)` in executor | | `PURPOSE '…'` prefix | Working, optional | Recorded for audit when declared | | `AS OF ` / `AS OF SYSTEM TIME ` | Working | UTC ISO-8601 or `i64` ns UTC; non-UTC offsets rejected | | `LIMIT n AFTER 'cursor'` | Working | cursor = decimal `system_from` ns; cannot combine with `ORDER BY` | | `ORDER BY` | Working | Multi-column with `ASC`/`DESC` tiebreakers (`ast.rs:1316`, `executor.rs:2665`) | | `WITH PROVENANCE` | Working | Must be a trailing modifier (after `LIMIT`) | | `WHERE` expressions | Working | `col op literal`, arithmetic, `now()`, `now() - INTERVAL 'N …'` | | `MATCH(col, 'q'[, PHRASE\|FUZZY\|STEMMED])` | Working | Default from BM25 posting list; `PHRASE` from positional index; `FUZZY`/`STEMMED` fall back to substring scan | | `TUMBLE` / `HOP` / `SESSION` windows | Working | HOP = overlapping bucket fan-out; SESSION = per-key activity-gap grouping | | `EXPLAIN PATH` | Working | Returns `{strategy, graph_node_count, pll_warm}` JSON | | `EXPLAIN REPLAY('', SEQ => n)` | Working | Re-derives a logged exhibit link's seal byte-identically | | `EXPLAIN POLICY` | Partial | Prefix parsed but flag not consulted on execute path; use the `explain_policy` MCP tool | | CTE incl. `WITH RECURSIVE` | Working | `parse_ctes_opt` (`parser.rs:1119`); recursive iteration cap `WITH_RECURSIVE_MAX_ITER` (`executor.rs:5980`) | | `UNION` / `UNION ALL` / `INTERSECT` / `EXCEPT` | Working | `execute_set_ops` (`executor.rs:2985`); `SetOpKind` enum (`ast.rs:1675`) | ## Operators and TVFs **Working and SQL-reachable:** `LOOKUP_IDENTITY`, `RESOLVE_IDENTITIES`, `IDENTITY_CLUSTER`, `SAME_IDENTITY`, `PATHS_BETWEEN`, `NETWORK_EXPAND`, `PREGEL_BFS`, `DEGREE()`, `HYBRID_SEARCH`, `fts_search`, `SIMILAR TO`, `FACE_SEARCH`, all 10 graph TVFs (`GRAPH_DIJKSTRA`, `GRAPH_SCC`, `GRAPH_CYCLES`, `GRAPH_SSSP`, `GRAPH_SPANNING_TREE`, `GRAPH_APSP`, `GRAPH_DIAMETER`, `GRAPH_SIMILARITY`, `GRAPH_NODE_METRIC`, `GRAPH_LINK_PREDICT`), all 13 `ScorerOp` analytics operators, `BENEFICIAL_OWNERSHIP_CHAIN`, `SANCTIONS_SCREEN`, `CRYPTO_TRACE`, `WIRE_RECONSTRUCTION`, `HAWALA_TRACE`, `GRAPH_COMMUNITY`, `GEOFENCE`, `ANPR_TRACE`, `DISPATCH_PRIORITY`, `CRIME_PATTERN_CLUSTER`, `WATCH PURPOSE`. The DataFusion TVF form `SELECT * FROM (...)` works for all registered TVFs. Each TVF call is translated to its governed keyword form and runs under the same purpose + ACL + org-isolation path. ## DDL | Statement | Status | Notes | |---|---|---| | `CREATE EXTENSION vector` / `DROP EXTENSION vector` | Working (pgwire) | No-op OK tag | | `CREATE TABLE` (pgvector) | Working (pgwire) | Registers the type; column list is ignored | | `INSERT` / `UPDATE` / `DELETE` | Working (pgwire) | The only native SQL DML path | | `CREATE MATERIALIZED VIEW … REFRESH (INCREMENTAL\|FULL) EVERY ` | Working | Initial full refresh + background loop | | `ALTER TABLE … ADD/DROP COLUMN` | Parses, then 501s | Use the ontology API | | `CREATE INDEX` / `CREATE TYPE` / `CREATE SCHEMA` | Not implemented | — | ## Agent memory surface All 10 cognitive verbs are wired through to the runtime — 10 MCP tools and 10 HTTP `/memory/*` routes: | Claim | Reality | |---|---| | 10 cognitive verbs | 10 implemented: original 7 + `associate` / `resolve` / `summarise` | | `recall` = hybrid retrieval | BM25 + vector fused via RRF, re-scored by confidence × recency × forgetting curve | | `recognize` returns a memory | Returns a `MemoryItem` projection | | `consolidate` supersedes | Closes the superseded row's `valid_to` and inserts the new belief | | `forget` deletes | `RetentionMark` enforced by `ForgetScheduler` on a background cadence (not a hard delete) | | 5 memory canonical types | All producers wired: `MemoryItem`, `AgentSession`, `ToolCall`, `DecisionRecord`, `Episode` | ## Canonical types and SmartIngest - **76 canonical kinds ship.** The often-cited ~170 is the target catalogue, not the shipped count. The enum is `#[non_exhaustive]` with reserved headroom. - A few kinds (GSTIN, BTC address, FARA) have validators but **no SmartIngest detection gate** — they round-trip but are not auto-detected from free text. - OT/ICS kinds (Modbus, OPC UA, DNP3, S7, IEC 61850) are auto-detected via the opt-in `ics` detector pack; ambiguous forms require a label (`MODBUS:17`). - `relata detect ""` runs all detector packs regardless of `RELATA_DETECT_PACKS` (which governs HTTP `/ingest` only). - Default packs: `network,contact,crypto`. Opt-in: `financial,payment,social,transport,device,ics`. Or `all` / `none`. ## Protocol compatibility doors | Door | Writes? | Known limits | |---|---|---| | ClickHouse HTTP + native TCP | Read-only | Cannot create Relata types | | Neo4j HTTP + Bolt | Yes (`CREATE`/`MERGE`) | Governed write door: `run_protocol_cypher_write` routes through `governed_upsert` for nodes and `validate_link_write` for edges | | Postgres / pgvector | Yes | ANN index is cosine-only; `<->`/`<#>` correct via over-fetch + re-rank | | MongoDB | Yes | No transactions, change streams, `$push`/`$pull`/`$unset`; nested equality via flattened columns only | | Redis | Yes | No `MULTI/EXEC`, `BLPOP`, scripting, cluster; Pub/Sub is in-memory (zero persistence) | | S3 | Yes | Buckets must be empty to delete; ETag = SHA-256; large objects streamed from object store | All doors bind to `127.0.0.1` and share `RELATA_BEARER_TOKEN`. pgwire refuses to start without a token; others default to open dev mode when unset. ## Capacity and scaling The execution path no longer materialises results in memory. Streaming scan, spilling hash-join + aggregate, streaming results, and spill-to-disk that frees RAM have all shipped. | Structure | Paged backend | Default profile | |---|---|---| | Authoritative rows | Spill-to-disk | `server`/`cluster` | | Secondary / range indexes | `DiskIndexSource` | `server`/`cluster` | | Full-text postings | `DiskIndexSource` | `server`/`cluster` | | Vector index (HNSW) | `PagedAnnIndex` + IVF cold tier | `server`/`cluster` | | Graph adjacency (CSR) | `PagedCsrGraph` | `server`/`cluster` | | Identity index | Live-paged | `server`/`cluster` | ### Measured performance characteristics | Operation | Measured overhead | |---|---| | Conditional ACL | ~1.32× raw-scan p50. Budget gate: <2.5× | | Bitmap row filtering | ~1.0× (branch-predicted bitset) | | Cell masking | ~2.6× raw-scan p50 — avoid on hot scan paths in latency-sensitive deployments | | Cold-restart RTO at 10M rows | WAL+Parquet flush ~15s; cold-load from Parquet ~55s. Single-node RTO ≈ 1 minute | ### Aggregation note `GROUP BY` without a filter predicate uses the columnar path. Filtered aggregates fall back to per-row field access and scale super-linearly at 10M rows. ### Async embedding resilience | Knob | Default | Effect | |---|---|---| | `RELATA_EMBED_QUEUE_MAX` | `100000` | At cap, tasks are dropped (metric `relata_embed_queue_dropped_total`) | | `RELATA_EMBED_TIMEOUT_MS` | `30000` | Prevents a GPU hang from stalling the drain worker | | `RELATA_EMBED_CIRCUIT_COOLDOWN_MS` | `60000` | After 5 consecutive errors, circuit opens; `/health/ready` returns 503 with `reason: embedder_unhealthy`. Writes always succeed. | ## Explicitly deferred These capabilities are advertised as design targets but are not shipping: | Capability | Current state | |---|---| | Hardware attestation (SEV-SNP / TDX) | Software stub — synthetic report for dev/CI; no `/dev/sev-guest` / TDX-driver-backed report | | Real Cedar SDK integration | The `cedar-policy` crate (v4) **is** a workspace dependency, wired as a secondary policy filter via `AclEngine::set_cedar_policy` / `apply_cedar_policy`. A native ABAC engine (deny-wins, attribute conditions) owns the primary evaluation path; Cedar is consulted as a deny-wins overlay only when a policy text is configured | | Cypher `MATCH` patterns + `WITH WARRANT` | Parser only handles `WITH PROVENANCE`; these are target syntax only | | SQL:2011 period predicates | `FOR SYSTEM_TIME FROM/TO/BETWEEN`, `OVERLAPS`, `CONTAINS`, `PRECEDES`, auto period-splitting are not parsed; use explicit `valid_from`/`system_from` predicates (`AS OF` covers point-in-time) | | Biometric ACL | Deferred pending legal review | | Sub-tenant namespaces | `NamespacePath` on `Row` — accepted but deferred | ## Release-readiness rule A capability is release-ready only when it has: (1) committed implementation, (2) public-safe documentation if user-facing, (3) tests or validation output, (4) known limits documented here. Items listed above as "parses, then errors" or "not wired" are not release-ready. ============================================================================== # MCP tools reference URL: https://relatadb.dev/docs/reference/mcp-tools ============================================================================== # MCP tools reference Relata ships a built-in [Model Context Protocol](https://modelcontextprotocol.io/) server with 60+ tools that AI agents can invoke. MCP is the canonical agent surface for Relata — agents get scoped ACL principals, governed by the same Cedar-inspired ABAC model as human users. ## Endpoint ``` POST /mcp/initialize GET /mcp/tools POST /mcp/tools/call GET /mcp/sessions ``` `POST /mcp/tools/call` takes a flat body: `{"name": "", "arguments": {...}}`. Auth: bearer token. ## Generate a client config ```bash relata mcp config --client claude # prints a claude_desktop_config.json snippet relata mcp config --client cursor # Cursor IDE config relata mcp config --client cline # Cline / Roo relata mcp config --client stdio # raw stdio config ``` Each command prints a ready-to-paste JSON snippet pointing at your local Relata server. ## Tool catalogue (60+ tools) ### Query | Tool | Purpose | Notes | |---|---|---| | `query` | Execute governed SQL (SELECT / PATHS_BETWEEN / LOOKUP_IDENTITY / SIMILAR) | Mandatory `purpose`; rows capped at 10,000 | | `query_knowledge` | Execute SQL with automatic purpose injection | Default purpose `analytics` | | `explain_policy` | Static ACL/purpose/egress analysis without executing | Returns allowed/denied + reasons | | `suggest_extensions` | List active extension packs and relevance | No arguments | | `search_knowledge` | Free-text search over IntelChunk/entities/relationships | `type_filter`, `source_filter`, `min_confidence` | | `search_entities` | Full-text + identity search across entity types | Typo-tolerant (`fuzzy`) | | `hybrid_search` | Combined BM25 + vector + graph, RRF fusion | Per-query weights | | `find_in_social_corpus` | Unified social-media retrieval (BM25 + vector + identity filter) | | ### Entity & Identity | Tool | Purpose | Notes | |---|---|---| | `lookup_identity` | Resolve a raw identifier to canonical form + matching entities | Arg `raw` (phone/email/IP/IMEI/…) | | `list_entity_types` | List object types with row counts | No arguments | | `get_entities` | Paginated entity list for a type with filters | `entity_type`, `filters`, `limit`, `offset` | | `get_domain_summary` | Per-domain roll-up (financial/telco/cyber/sanctions/…) | Arg `domain` | | `resolve_entity_identity` | RESOLVE_IDENTITY canonical cluster for an entity | Arg `identity` | ### Ingest & RAG | Tool | Purpose | Notes | |---|---|---| | `ingest_document` | Store text + entities + relations with bi-temporal provenance | Routes to IntelChunk / canonical types / KnowledgeTriple | | `rag_store_answer` | Store a RAG answer (RagAnswer + RagSource rows) | Flat and dgrep-rag nested shapes | | `rag_store_elements` | Store dgrep-rag ExtractorElements | `elements`, `source_filename` | | `ingest_media` | Image/audio/video (base64) or text for embedding + perceptual-hash dedup | Returns a task id | ### Knowledge graph | Tool | Purpose | Notes | |---|---|---| | `get_relationships` | KnowledgeTriple records, filtered by subject/predicate/object/source | Returns triples + unique entities | | `paths_between` | Governed PATHS_BETWEEN walk | Args `from`, `to`, `max_hops` | | `list_link_types` | Governed edge types in the ontology | No arguments | ### Entity intelligence | Tool | Cost | Notes | |---|---|---| | `get_entity_profile` | 5 | 360° profile (identity, relationships, transactions, intel, sanctions) | | `get_timeline` | 3 | Chronological event timeline | | `find_connections` | 3 | Hidden network connections (relationship / transaction / shared attribute) | | `get_case_summary` | 5 | Per-purpose summary: inventory, graph, notes, RAG answers, next steps | | `add_case_note` | 1 | Analyst note stored as CaseAnnotation | | `get_audit_trail` | 1 | Tamper-evident audit log scoped to a purpose | ### Memory (agent cognitive verbs) | Tool | HTTP equivalent | Notes | |---|---|---| | `remember` | `POST /memory/remember` | Store a MemoryItem (`content`, `session_id`, `confidence`, `memory_class`) | | `remember_batch` | `POST /memory/remember/batch` | Bulk write; `items[]` + default `purpose` | | `recall` | `GET /memory/recall` | Hybrid BM25 + vector; `query`/`q`, `top_k`, `as_of`, `class_filter` | | `recognize` | `GET /memory/recognize/:id` | Fetch one MemoryItem by id | | `episodes_in` | `GET /memory/episodes` | List Episodes for a session | | `justify` | `GET /memory/justify/:id` | Provenance chain + audit trail | | `consolidate` | `POST /memory/consolidate` | Supersede a MemoryItem (`id`, `content`, `confidence`) | | `forget` | `DELETE /memory/forget/:id` | Retention-policy retract (`retain_days`; `-1` = legal hold) | | `associate` | `POST /memory/associate` | Link two items (`from_id`, `to_id`, `relation`) | | `resolve` | `GET /memory/resolve/:id` | Follow supersession chain to the canonical MemoryItem | | `summarise` | `POST /memory/summarise` | Governed summary of a session/topic | ### Governance, media & ops | Tool | Purpose | Notes | |---|---|---| | `erase_subject` | GDPR Art. 17 erasure (row + vector + blob) | Returns a signed certified receipt | | `similar_multimodal` | Governed cross-modal similarity (SIMILAR TO … LIMIT k) | ACL + cell masking apply | | `server_health` | Readiness snapshot | Mirrors `/health/ready` | | `job_status` / `list_jobs` | Continuous detection jobs | status, interval, last-run, alerts | | `schedule_job` | Trigger one run of a named job | Returns alert count | | `list_workflows` / `run_workflow` / `workflow_status` | Workflow definitions and executions | | | `metrics` | Operational counters | Mirrors `/metrics.json` | | `list_rules` / `create_rule` / `import_sigma` | Detection-rule management | Mirrors `/rules` | ### Investigation, graph analytics & finance | Tool | Purpose | Notes | |---|---|---| | `trace_crypto` | CRYPTO_TRACE hop-by-hop | `address`, `max_hops` | | `beneficial_ownership` | BENEFICIAL_OWNERSHIP_CHAIN | `party`, `max_depth` | | `reconstruct_wire` | WIRE_RECONSTRUCTION | `account`, `tolerance_pct` | | `trace_hawala` | HAWALA_TRACE informal value transfer | `seed`, `max_hops` | | `geofence` | GEOFENCE spatial query | `lat`, `lon`, `radius_m` | | `detect_communities` | Community detection (Louvain/Leiden) | `entity_type`, `algo` | | `rank_key_nodes` | PageRank / centrality | `metric`: pagerank/betweenness/closeness/eigenvector | | `hub_authority` | HITS hub + authority scores | | | `predict_links` | Link prediction (common_neighbors/jaccard/adamic_adar) | | | `find_scc` | Strongly connected components | Detects circular structures | | `screen_sanctions` | SANCTIONS_SCREEN | `name`, `threshold` | | `aggregate_stats` | Governed COUNT/SUM/AVG | `entity_type`, `agg`, `column` | | `investigate_entity` | Composite investigation profile | entity profile + timeline + connections + risk | | `find_threats` | Composite threat hunt | communities + top-risk nodes + active rules | ### Multimodal | Tool | Purpose | Notes | |---|---|---| | `search_video_frames` | Governed SIMILAR TO over VideoFrame entities | `query_id`, `top_k` | | `face_match` | Biometric face match (GATED — returns 403 until biometric ACL ships) | `probe_id`, `threshold` | ### Intelligence | Tool | Purpose | Notes | |---|---|---| | `nl_query` | Natural-language → governed SQL → execute | Deterministic fallback when `RELATA_LLM_URL` is unset; `interpret` for NL summary | ## Tool schema Every tool exposes a JSON Schema for its input parameters via `GET /mcp/tools`. Example (`search_knowledge`): ```json { "name": "search_knowledge", "description": "Search across ingested knowledge content — documents, entities, and relationships — using free-text query.", "inputSchema": { "type": "object", "required": ["query"], "properties": { "query": { "type": "string" }, "purpose": { "type": "string" }, "type_filter": { "type": "string", "description": "Restrict results to a content type: text, person, organization, location, relationship, answer." }, "source_filter": { "type": "string", "description": "Restrict to content from this document/source." }, "limit": { "type": "integer", "default": 20 }, "min_confidence": { "type": "number", "default": 0.0 }, "fuzzy": { "type": ["boolean", "object"], "description": "Bounded typo tolerance: true, or an object with edit_distance 1 or 2." } } } } ``` Agents that respect MCP (Claude, Cursor, Cline, LangChain, LlamaIndex) consume this schema and propose tool calls to the model. ## Response shape ```json { "content": [ { "type": "text", "text": "..." }, { "type": "json", "json": { } } ], "isError": false, "meta": { "processing_time_ms": 42 } } ``` The `meta.processing_time_ms` field lets agents measure their own tool-call overhead. ## Agent scoping (security) Agents authenticate via a bearer token like any other client. The token maps to a principal; the principal's ACL role determines what the agent can do. **Best practice**: give each agent a dedicated principal with the minimum required permissions. Don't reuse a human principal for an agent. ```bash # Register a scoped dynamic bearer token via the admin token surface # (POST /admin/tokens writes to the in-memory + on-disk token registry). # Give each agent a dedicated principal with the minimum required permissions. ``` ## Sessions `POST /mcp/initialize` is the MCP handshake — it returns server metadata and the tool catalogue so a client can discover capabilities before calling tools: ```bash curl -X POST http://localhost:9090/mcp/initialize \ -H "Authorization: Bearer $RELATA_TOKEN" # Response: # { # "protocol_version": "2024-11-05", # "server_info": { "name": "relata", "version": "...", "profile": "...", "node_id": "..." }, # "capabilities": { "tools": { "list_changed": false }, ... }, # "tools": [ { "name": "query", ... }, ... ] # } ``` `GET /mcp/sessions` lists `AgentSession` records with their ToolCall history (filter by `session_id`, cap with `limit`). AgentSessions are created on the first memory `remember` for a new `session_id` — not by `/mcp/initialize`. ## Multi-tenant agents ```bash curl -X POST http://localhost:9090/mcp/tools/call \ -H "Authorization: Bearer $RELATA_TOKEN" \ -H "X-Organization-Id: org-acme" \ -d '{"name": "query", "arguments": {"sql": "SELECT * FROM Person LIMIT 5", "purpose": "analytics"}}' ``` The principal's org scoping applies — agent queries never leak across tenants. ## Examples ### Claude Desktop `~/Library/Application Support/Claude/claude_desktop_config.json`: ```json { "mcpServers": { "relata": { "command": "npx", "args": ["-y", "relata-mcp-bridge"], "env": { "RELATA_URL": "http://localhost:9090", "RELATA_TOKEN": "relata-dev" } } } } ``` ### LangChain (Python) `relata_adapters.langchain.RelataMemory` is a governed `BaseMemory`-shaped adapter backed by Relata's `/memory/*` surface: ```python from relata_adapters.langchain import RelataMemory mem = RelataMemory(base_url="http://localhost:9090", purpose="research") # chain = ConversationChain(llm=..., memory=mem) # save_context stores each turn; load_memory_variables recalls the most # relevant prior memories for the incoming input. ``` ### Direct curl ```bash curl -X POST http://localhost:9090/mcp/tools/call \ -H "Authorization: Bearer $RELATA_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "search_knowledge", "arguments": { "query": "governance policy for cross-border data sharing", "purpose": "investigation", "limit": 5, "fuzzy": true } }' ``` ## Access control The same rules apply to every tool call: - **Auth:** Bearer token required (`Authorization: Bearer `). - **Purpose:** every call requires a registered purpose — absent purpose returns HTTP 400. - **ACL:** Cedar-inspired ABAC evaluated on every request; deny-wins. - **Egress filtering:** classified types (`SourceTrueIdentity`, `SigintIntercept`, `AccessScopedIntercept`, `LawfulInterceptRecord`) are blocked at egress regardless of query success. - **Audit:** every invocation recorded with principal, timestamp, purpose, cost units, and a tamper-evident hash chain. - **Quota:** most query tools cost 1 unit; multi-table / intelligence tools cost 3–5. Default 10,000 units/principal. Quota exhaustion returns HTTP 429. ### Purpose enforcement ```bash # Strict (default) — only registered purposes RELATA_PURPOSE_MODE=strict RELATA_PURPOSES=analytics,audit,compliance,operations # Open — any non-empty string (dev/test only) RELATA_PURPOSE_MODE=open ``` Hierarchical scoping with `:` separator is supported (`analytics:external`). ### Egress filtering These classified types never appear in tool results: | Type | What it contains | |---|---| | `SourceTrueIdentity` | HUMINT protected true identity | | `SigintIntercept` | Signal intelligence intercept records | | `AccessScopedIntercept` | Restricted access-scoped data | | `LawfulInterceptRecord` | Lawful intercept records | ## Errors ```json { "content": [{ "type": "text", "text": "error message" }], "isError": true } ``` | HTTP | Cause | |---|---| | 200 | Tool succeeded | | 400 | Missing / invalid parameters or missing purpose | | 401 | Bearer token missing or invalid | | 403 | ACL denied, purpose denied, egress blocked, or protected-type ingest | | 404 | Unknown tool | | 429 | Quota exhausted or ingest queue full | | 500 | Store unavailable or execution error | See [Error codes reference](/docs/reference/error-codes) for the RFC 7807 mapping. ## Not exposed over MCP `watch` / `subscribe` are not MCP tools. The `SubscriptionManager` + `/watch/stream` SSE endpoint produces a long-lived event stream; the MCP `tools/call` request-response envelope cannot carry an open stream. Use SSE directly for subscriptions. ## See also - [Agent memory surface](/docs/reference/agent-memory) - [Error codes reference](/docs/reference/error-codes) - [Search and retrieval](/docs/reference/search) - [MCP specification](https://modelcontextprotocol.io/) ============================================================================== # Performance tuning URL: https://relatadb.dev/docs/reference/performance ============================================================================== # Performance tuning How to tune a Relata deployment for throughput, latency, or cost. Numbers below are starting points measured on commodity hardware — always benchmark your own workload. ## Three levers | Lever | Knob class | Goal | |---|---|---| | **Memory** | `RELATA_*_RAM_MB`, `RELATA_*_MAX_BYTES` | Hot data resident | | **Parallelism** | rayon (graph), worker pools, scatter-gather | Use all cores | | **Disk I/O** | compaction strategy, WAL group-commit, spill format | Reduce fsync pressure | Memory is usually the highest-impact knob. Parallelism is free if your data fits in RAM. Disk I/O matters when it doesn't. ## Adaptive sizing (let Relata choose) If you're unsure, **unset the budget env vars**. Relata probes the hardware at startup and splits a single ~75%-of-RAM pool across consumers. The banner at startup logs the chosen split. Override only the bucket that matters for your workload: ```bash # Graph-heavy workload: give graph more RAM RELATA_GRAPH_RAM_BUDGET_MB=16384 # 16 GB # OLAP workload: grow the result cache RELATA_RESULT_CACHE_MAX_BYTES=1073741824 # 1 GB ``` ## Per-workload tuning ### OLTP (high QPS, small queries) **Goal**: sub-5 ms p99 reads, sub-10 ms p99 writes. ```bash # Tighten the query timeout (fail fast) RELATA_QUERY_TIMEOUT_SECS=5 # Cache results for repeat queries RELATA_RESULT_CACHE_ENABLED=true RELATA_RESULT_CACHE_TTL_SECS=60 # short TTL for OLTP freshness RELATA_RESULT_CACHE_MAX_BYTES=268435456 # 256 MB # Rate-limit aware admission RELATA_RATE_LIMIT_RPS=10000 # free; 100000 server ``` **Key perf wins**: 1. **Plan cache hit ratio > 90%** — same SQL templates reuse the verdict. 2. **Result cache hit ratio > 50%** for read-heavy patterns. 3. **PK lookups use the live-index** — O(1) always. 4. **Avoid `SELECT *`** — projection cost matters at high QPS. ### OLAP (large analytical scans) **Goal**: sub-second BI queries over 100 M+ rows. ```bash # Big exec budget for sorts, joins, aggregates RELATA_EXEC_RAM_BUDGET_MB=8192 # 8 GB # Big result cache for dashboard workloads RELATA_RESULT_CACHE_MAX_BYTES=2147483648 # 2 GB RELATA_RESULT_CACHE_MAX_ROWS=100000 ``` **Key perf wins**: 1. **Columnar aggregate path** (COUNT/SUM/MIN/MAX over a column) is 5–10× the row path. 2. **Per-column bloom filters** prune disk segments pre-decode. 3. **HLL + CMS sketches** accelerate `COUNT(DISTINCT)` and frequency estimates. 4. **Result cache + `WITH CACHE TTL 300`** for dashboard workloads. ### Time-series (high ingest) **Goal**: 100 K–1 M rows/sec sustained. ```bash # Batched fsync window (RELATA_WAL_SYNC=interval coalesces fsyncs ~every 10 ms) RELATA_WAL_SYNC=interval # Bigger segment flush threshold = fewer flushes RELATA_FLUSH_SEGMENT_MAX_ROWS=500000 # default 250 000 # Async media ingest for embedding-sidecar workloads RELATA_EMBED_BATCH_SIZE=64 RELATA_EMBED_CONCURRENCY=8 ``` **Key perf wins**: 1. **Group commit** — N concurrent writers coalesce into ≤1 fsync per ~10 ms window when `RELATA_WAL_SYNC=interval` (the default). 2. **Batch ingest via `/ingest` (not row-by-row INSERT)** — 10× throughput. ### Graph (multi-hop traversals) **Goal**: <100 ms p99 for 3-hop traversal over 100 M edges. ```bash # Give graph its own RAM budget (don't share with secondary indexes) RELATA_GRAPH_RAM_BUDGET_MB=16384 # 16 GB ``` **Key perf wins**: 1. **CSR cache default-on** — 5–50× on multi-op workflows. 2. **rayon parallelism** — 8–32× on multi-core. 3. **PLL hub labeling** — 100–1000× on distance queries. 4. **Bidirectional BFS** — √2× typical on point-to-point. ### Vector (ANN at scale) **Goal**: <10 ms p99 over 1 M vectors at 99% recall. ```bash # HNSW parameters (set at index creation, not at runtime) # M=128, ef_construction=350, ef_search=k.max(10) (defaults) # Cold tier for >RAM-scale RELATA_DISKANN_MAX_RESIDENT=1000000 # 1 M vectors RAM-resident RELATA_VECTOR_COLD_RESIDENT_MAX=200000 # IVF staging cap # Search preset RELATA_SEARCH_PRESET=balanced # strict | balanced | lenient ``` **Key perf wins**: 1. **int8 quantization** (default) — 4× memory savings, ±0.4% recall. 2. **Pre-filter vs post-filter adaptive threshold** at 25% selectivity. 3. **`RELATA_SEARCH_PRESET=strict`** for precision queries; `lenient` for recall. ## Query-level tuning ### Use `EXPLAIN` ```sql PURPOSE 'analytics' EXPLAIN SELECT * FROM Person WHERE name = 'Alice' ``` Output shows access path (Index vs Full), estimated rows, and selectivity. `EXPLAIN ANALYZE` shows per-operator actuals. ### Use indexes Equality on indexed columns is O(log n). Range on indexed columns walks the BTreeMap. CIDR match uses the prefix index. FTS uses BM25 with WAND pruning. ```sql -- Equality index used: SELECT * FROM Person WHERE email = 'alice@example.com' -- Range index used: SELECT * FROM Event WHERE ts > '2024-01-01' AND ts < '2024-02-01' -- BM25 index used: SELECT * FROM Document WHERE MATCH(body, 'governance') -- Vector ANN used: SELECT * FROM Vec WHERE SIMILAR TO '[0.1,0.2,...]' FIELD=_emb_text LIMIT 10 ``` ### Use `LIMIT` aggressively ```sql -- Good: ordered LIMIT walks the index, returns early SELECT * FROM Event ORDER BY ts DESC LIMIT 10 ``` ### Use cursor pagination ```sql -- Good: cursor pagination for infinite-scroll UIs SELECT * FROM Event ORDER BY ts DESC LIMIT 100 AFTER '' ``` ### Use `AS OF` for time travel ```sql -- Good: AS OF uses the bi-temporal index, not a full scan SELECT * FROM Person AS OF '2024-06-01T00:00:00Z' -- Bad: filtering on system_from manually SELECT * FROM Person WHERE system_from <1717200000000000000 ``` ## Cluster tuning ### Scatter-gather ```bash # Bound fan-out parallelism RELATA_SCATTER_MAX_PARALLEL=64 # Per-peer timeout (milliseconds) RELATA_SCATTER_PEER_TIMEOUT_MS=10000 ``` ### Cache coherence Gossip-based invalidations piggyback on heartbeats (10 s window). For read-your-writes across nodes, use session affinity (route the same principal to the same node). ## Common anti-patterns | Anti-pattern | Why it's slow | Fix | |---|---|---| | `SELECT *` on wide tables | Materialises every column | Project only what you need | | `COUNT(*)` on a type with no `SummaryStore` | Full scan | Use the `SummaryStore` fast path (auto for un-filtered) | | Recursive CTEs without `LIMIT` | Unbounded iteration | Always cap with `MAX_RECURSIVE_ITERS` | | `OFFSET` > 1000 | Linear skip | Use cursor pagination (`AFTER`) | | Inserting row-by-row | 1 fsync per row | Batch via `/ingest` | | Vector search without pre-filter | ANN over full index | Use `search_filtered` with an allowlist | | Graph algorithm on a cold CSR | Full rebuild per call | Enable paged-graph cache | ## Benchmarking ```bash # Quick gate (1 min) RELATA_GLOBAL_SCAN_ALLOWED=true cargo run -p relata-bench --release -- full --scale 100k --gate # Full bench suite RELATA_GLOBAL_SCAN_ALLOWED=true cargo run -p relata-bench --release -- full # 22-interface live-server bench ./scripts/bench.sh --full # Comparative vs other engines (Docker) ./scripts/bench.sh --docker --rust ``` Always benchmark on the target hardware with the target workload. ## See also - [Environment variables reference](/docs/reference/env-vars) - [SQL reference](/docs/reference/sql) - [Query cookbook](/docs/reference/query-cookbook) - [Vector index parameters](/docs/reference/vector-params) ============================================================================== # Protocol Compatibility URL: https://relatadb.dev/docs/reference/protocols ============================================================================== # Protocol Compatibility RelataDB speaks **8 compatibility doors** plus **5 native protocols** (13 wire surfaces total) from one binary and one governed store. Any existing client library that speaks one of the compat doors works without the Relata SDK. All compat doors bind to `127.0.0.1` and share one credential: `RELATA_BEARER_TOKEN`. Doors **auto-enable when `RELATA_BEARER_TOKEN` is set** (any tier); an explicit `RELATA__ENABLE=true|false` overrides. With no token and no explicit enable, doors stay off by default (an unauthenticated port is never auto-exposed). **pgwire refuses to start without a token.** On non-free profiles, an explicit `=true` without a token fails closed, and `RELATA_TENANCY_MODE=multi` refuses the tenant-less shared-token doors (use pgwire, which carries per-connection org). > **Cross-protocol consistency:** write via S3, Redis, or MongoDB and read the same data back over SQL, pgwire, or any other surface. ACL, org isolation, and the audit log apply on every door uniformly. ## Protocol matrix | Door | Port env var (default) | Writes to governed store? | |---|---|---| | S3 (AWS / boto3 / rclone) | `RELATA_S3_PORT` (9191) | Yes — `S3Object`, `S3Bucket` | | Postgres + pgvector | `RELATA_PG_PORT` (5433) | Yes — typed rows + `_emb_text` vectors | | ClickHouse HTTP | `RELATA_CLICKHOUSE_PORT` (8123) | Read-only (governed SELECT) | | ClickHouse native TCP | `RELATA_CH_NATIVE_PORT` (9000) | Read-only | | Neo4j HTTP Cypher | `RELATA_NEO4J_PORT` (7474) | Yes — `CREATE`/`MERGE` via governed write door | | Neo4j Bolt | `RELATA_BOLT_PORT` (7687) | Yes — `CREATE`/`MERGE` via governed write door | | Redis RESP | `RELATA_REDIS_PORT` (6379) | Yes — `KvEntry` | | MongoDB wire | `RELATA_MONGO_PORT` (27017) | Yes — `MongoDocument` | Native Relata protocols (always on): | Door | Default port | Notes | |---|---|---| | HTTP REST | 9090 | `/query`, `/ingest`, `/search`, `/memory/*`, `/mcp`, `/health`, `/status`, `/metrics`, `/types`, `/specs`, `/sparql`, `/watch/stream` | | gRPC | 50051 | gRPC door | | Arrow Flight | 8815 | Zero-copy columnar streaming; enable with `RELATA_FLIGHT_ENABLE=true` | | MCP | `/mcp` on HTTP | Model Context Protocol tools | | SPARQL | `/sparql` on HTTP | Single Basic Graph Pattern over `KnowledgeTriple` + optional `LIMIT` | ## Start the doors ```bash export RELATA_BEARER_TOKEN= RELATA_S3_PORT=9191 \ RELATA_PG_PORT=5433 \ RELATA_CLICKHOUSE_PORT=8123 \ RELATA_NEO4J_PORT=7474 \ RELATA_REDIS_PORT=6379 \ RELATA_MONGO_PORT=27017 \ relata serve ``` A single end-to-end smoke test for all six compat doors: `scripts/protocol_smoke_test.py`. ## S3 — boto3 / aws CLI / rclone / MinIO client Supported operations: ListBuckets, CreateBucket / HeadBucket / DeleteBucket, ListObjectsV2, GetBucketLocation, PutObject / GetObject / DeleteObject / HeadObject, and multipart upload. ```python import boto3 from botocore.config import Config s3 = boto3.client( "s3", endpoint_url="http://127.0.0.1:9191", aws_access_key_id="change-me", aws_secret_access_key="unused", config=Config(signature_version="s3v4", s3={"addressing_style": "path"}), ) s3.create_bucket(Bucket="cases") s3.put_object(Bucket="cases", Key="exhibit-1.txt", Body=b"hello world") body = s3.get_object(Bucket="cases", Key="exhibit-1.txt")["Body"].read() print(body) ``` When `RELATA_S3_SECRET_KEY` is set the door **requires verified SigV4** and rejects plaintext bearer auth. ### Read S3 objects over SQL ```sql SELECT key, size, content_hash FROM S3Object WHERE bucket = 'cases'; ``` ### Limits - Buckets must be empty to delete. - Multipart parts are in-memory only (lost on restart). - ETag is SHA-256. - Object bodies ≥ `RELATA_S3_BLOB_THRESHOLD_MB` (default 4 MiB) spill to the content-addressed blob store. ## Postgres + pgvector — psql / psycopg2 / LangChain PGVector ```bash psql -h 127.0.0.1 -p 5433 -U relata relata # password = RELATA_BEARER_TOKEN ``` ```sql CREATE EXTENSION vector; CREATE TABLE docs (id text PRIMARY KEY, embedding vector(3)); INSERT INTO docs VALUES ('a', '[1,0,0]'), ('b', '[0.9,0.1,0]'); -- Cosine KNN — auto-routed to Relata's HNSW index SELECT id FROM docs ORDER BY embedding <=> '[0.9,0.1,0]' LIMIT 2; ``` `INSERT` / `UPDATE` / `DELETE` and ordinary `SELECT` work. GUI clients (TablePlus, DBeaver, pgAdmin, DataGrip) connect and browse schema via the catalog intercept. | KNN operator | Metric | Note | |---|---|---| | `<=>` | cosine distance | Preferred — ANN index is cosine-only | | `<->` | L2 distance | Metric-correct via over-fetch + re-rank | | `<#>` | negative inner product | Metric-correct via over-fetch + re-rank | > **pgwire is fail-closed:** refuses to start when `RELATA_BEARER_TOKEN` is unset. ## ClickHouse — HTTP or native TCP ```bash # HTTP curl -X POST "http://127.0.0.1:8123/?query=SELECT+1" \ -H "X-ClickHouse-Key: change-me" curl -X POST http://127.0.0.1:8123/ \ -H "X-ClickHouse-Key: change-me" \ --data-binary "SELECT name FROM Person FORMAT JSONEachRow" ``` ```python from clickhouse_driver import Client ch = Client(host="127.0.0.1", port=9000, password="change-me") rows = ch.execute("SELECT name FROM Person LIMIT 5") print(rows) ``` Read-only. The door routes governed `SELECT`s through the planner; writes are not supported. ## Neo4j — HTTP Cypher or Bolt ```bash # HTTP Cypher curl -X POST http://neo4j:change-me@127.0.0.1:7474/db/neo4j/tx/commit \ -H "Content-Type: application/json" \ -d '{"statements":[{"statement":"MATCH (n) RETURN n LIMIT 5"}]}' ``` ```python from neo4j import GraphDatabase driver = GraphDatabase.driver("bolt://127.0.0.1:7687", auth=("neo4j", "change-me")) with driver.session() as s: print(s.run("MATCH (n) RETURN n LIMIT 5").data()) ``` Cypher subset supported: - Relationship path patterns including bounded `-[r*1..5]->` - Single-identifier `RETURN [AS alias]` - Whitelisted property predicates Typed labels `(n:Person)` and typed edges `[:KNOWS]` are parsed but ignored. **Read Cypher only.** ## Redis — any RESP client ```bash redis-cli -h 127.0.0.1 -p 6379 -a change-me SET foo bar redis-cli -h 127.0.0.1 -p 6379 -a change-me GET foo ``` Governed keys persist as `KvEntry` rows. Read back over SQL: ```sql SELECT key, value FROM KvEntry WHERE key = 'foo'; ``` Not supported: `MULTI/EXEC`, `BLPOP`, scripting, cluster commands. Pub/Sub is in-memory only (zero persistence). ## MongoDB — any Mongo wire client ```javascript const { MongoClient } = require("mongodb"); const c = new MongoClient("mongodb://localhost:27017", { auth: { username: "relata", password: "change-me" }, }); const db = c.db("cases"); await db.collection("exhibits").insertOne({ _id: "ex1", body: "hello" }); console.log(await db.collection("exhibits").findOne({ _id: "ex1" })); ``` Governed docs persist as `MongoDocument` rows. Read back over SQL: ```sql SELECT * FROM MongoDocument WHERE collection = 'exhibits'; ``` | Limit | Detail | |---|---| | Auth | SCRAM-SHA-256 only | | `maxWireVersion` | 17 | | Transactions / change streams | Not supported | | `$push` / `$pull` / `$unset` | Not supported | | Nested equality | Via flattened columns only | ## Arrow Flight — zero-copy streaming ```bash # Enable on the server RELATA_FLIGHT_ENABLE=true relata serve ``` Connect any Arrow Flight client (Python `pyarrow.flight`, etc.) to `grpc://localhost:8815`: ```python import pyarrow.flight as fl client = fl.connect("grpc://localhost:8815") reader = client.do_get(fl.FlightDescriptor.for_command( b"PURPOSE 'analytics' SELECT * FROM Person")) for batch in reader: print(batch.data.num_rows, "rows") ``` Arrow IPC format — no JSON intermediate, no string serialisation. Use for high-throughput columnar reads. ## SPARQL ```bash # GET curl "http://localhost:9090/sparql?query=SELECT+%3Fs+%3Fp+%3Fo+WHERE+%7B+%3Fs+%3Fp+%3Fo+%7D+LIMIT+10" \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" # POST curl -X POST http://localhost:9090/sparql \ -H "Content-Type: application/sparql-query" \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \ -d "SELECT ?s ?p ?o WHERE { ?s ?p ?o } LIMIT 10" ``` Single Basic Graph Pattern over `KnowledgeTriple`. Optional `LIMIT`. Both GET and POST verify bearer auth. ## Security notes - All doors bind to `127.0.0.1`. - All doors share `RELATA_BEARER_TOKEN`. - pgwire is fail-closed (refuses to start without a token). - All other doors default to open dev mode when the token is unset. - Egress filtering applies uniformly on every door. - Cross-protocol reads and writes go through the same planner with the same ACL, org isolation, and audit chain. ## See also - [SQL Reference](/docs/reference/sql) — the underlying query plane - [MCP Tools Reference](/docs/reference/mcp-tools) — agent-native surface - [Limits](/docs/reference/limits) — per-protocol status and known caveats ============================================================================== # Query cookbook URL: https://relatadb.dev/docs/reference/query-cookbook ============================================================================== # Query cookbook Reference examples for querying Relata from every supported surface. Each example is labelled **verified** (runs against the current binary) or **target syntax** (planned, not yet runnable). --- ## Part 1: SQL query forms ### 1. Basic purpose-scoped query Status: **verified** ```sql PURPOSE 'analytics' SELECT * FROM Person LIMIT 10 ``` CLI: ```bash relata query "PURPOSE 'analytics' SELECT * FROM Person LIMIT 10" # or from file: relata query --file queries/persons.sql ``` ### 2. Query without PURPOSE (optional) Status: **verified** ```bash relata query "SELECT * FROM Person LIMIT 5" ``` PURPOSE is optional; when provided it is recorded in the audit log. When omitted, the query still runs — purpose is not enforced on the read path. ### 3. Time-scoped query (valid-time) Status: **verified** ```sql SELECT * FROM Document AS OF '2026-01-01T00:00:00Z' LIMIT 10 ``` Returns rows whose `valid_from ≤ ts < valid_to` at the given timestamp. `AS OF SYSTEM TIME ''` uses system time instead of valid time. ### 4. Query with provenance Status: **verified** `WITH PROVENANCE` is a **trailing** modifier (after `LIMIT`): ```sql SELECT * FROM Event LIMIT 10 WITH PROVENANCE ``` Adds a parallel `provenance` array to the response — one entry per row with `source`, `method`, `confidence`, `recorded_at`, and `derived_from`. ### 5. Explain policy Status: **verified** (via HTTP `POST /query`) `EXPLAIN POLICY` is a **prefix** (before the optional `PURPOSE`): ```sql EXPLAIN POLICY PURPOSE 'analytics' SELECT * FROM Person LIMIT 10 ``` Returns the ACL decision tree and cell-mask plan for the requesting principal. ### 6. Filtered query ```sql SELECT * FROM Person WHERE name = 'Alice' LIMIT 5 ``` ### 7. Operator TVFs (verified) ```sql -- Graph: paths between two entity IDs, up to N hops. SELECT * FROM paths_between('alice', 'bob', 3); -- Identity lookup — same surface as LOOKUP_IDENTITY. SELECT * FROM lookup_identity('+919876543210'); -- Identity resolution modes: canonical / cluster / fuse. SELECT * FROM resolve_identity('alice@example.com'); SELECT * FROM resolve_identity('alice@example.com', 'cluster'); -- Finint — beneficial ownership chain. SELECT * FROM beneficial_ownership_chain('acme_inc', 5); -- Finint — sanctions screening (default Jaccard threshold 0.75). SELECT * FROM sanctions_screen('alice'); -- Crypto trace — BFS over TransactionGraph. SELECT * FROM crypto_trace('0xabc', 6, 1000.0); ``` TVFs are governed keyword operators: the parser accepts the keyword form (e.g. `CRYPTO_TRACE(...)`, `SANCTIONS_SCREEN(...)`) and the equivalent DataFusion TVF form (`SELECT * FROM crypto_trace(...)`), translating the latter back to the keyword form so purpose + ACL + org-isolation run identically. The translation requires `SELECT *` over a single TVF call — projections, JOINs, and subqueries over a TVF are rejected; run two queries and join in the client instead. ### 8. Custom ranking rules — `RANK BY` clause Blend time-decay, popularity, or exact-match signals into the result ordering: ```sql -- Rank articles by recency (half-life 24 h) and popularity field. SELECT * FROM Article WHERE MATCH(content, 'quantum computing') RANK BY RECENCY(published_at, 86400), CUSTOM(popularity, 0.3) LIMIT 20 -- Boost exact category match. SELECT * FROM Product WHERE MATCH(description, 'laptop') RANK BY EXACT(category, 2.0), RECENCY(updated_at, 3600) LIMIT 10 ``` **Rules:** - `RECENCY(field, half_life_secs)` — `exp(-elapsed / half_life)` using an integer timestamp field (ns UTC). - `CUSTOM(field, weight)` — multiplies a numeric field by the weight. - `EXACT(field, boost)` — adds `boost` when the field value case-insensitively matches the search term. Scores are additive; results are sorted descending. An explicit `ORDER BY` following `RANK BY` overrides the ranking order. **Per-type defaults via env:** ```bash # Apply recency + popularity ranking for all Article queries that omit RANK BY. export RELATA_RANKING_Article="recency:published_at:86400,custom:popularity:0.3" ``` Format: `rule:field:value[,rule:field:value,...]` ### 9. Suffix and infix search Search for words by their ending or middle substring using wildcard patterns: ```sql -- Suffix search: terms ending with 'son' (Johnson, Jackson, Thompson) SELECT * FROM Person WHERE MATCH(name, '*son') LIMIT 20 -- Infix search: terms containing 'iversi' (university, diversity) SELECT * FROM Person WHERE MATCH(name, '*iversi*') LIMIT 20 -- Explicit mode keyword (same as auto-detect from '*') SELECT * FROM Log WHERE MATCH(path, '*tion', SUFFIX) LIMIT 10 SELECT * FROM Log WHERE MATCH(message, '*ering*', INFIX) LIMIT 10 ``` **How it works:** - `'*suffix'` (leading `*`) → suffix search via reverse-trigram index - `'*infix*'` (both `*`) → infix search via forward trigram index with substring verification - Short patterns (<3 chars) fall back to a dictionary scan Wildcard detection is automatic — the `SUFFIX`/`INFIX` mode keyword is optional. ### 10. Degree queries — `DEGREE()` function `DEGREE(column, direction)` returns the O(1) in/out/both edge count for each node from the incremental degree index. No full graph rebuild is needed. ```sql -- Out-degree: how many calls did this phone make? SELECT number, DEGREE(number, 'out') AS out_degree FROM Phone WHERE DEGREE(number, 'out') > 5 ORDER BY out_degree DESC LIMIT 10; -- Combined in + out degree: SELECT name, DEGREE(id, 'both') AS degree FROM Person ORDER BY degree DESC LIMIT 5; ``` `direction` values: `'out'`, `'in'`, `'both'` (default `'both'`). Returns `0` when no link store is attached or the node has no edges. ### 11. Lookup tables — CSV enrichment at query time Register a lookup table from a CSV file, then enrich query results: ```sql -- Register once (survives until restart): REGISTER LOOKUP cmdb_assets FROM '/data/cmdb.csv' KEY (ip) FIELDS (owner, criticality, environment) REFRESH EVERY 5 MINUTES; -- Enrich query results with owner/criticality from the CMDB: SELECT src_ip, LOOKUP cmdb_assets(src_ip) -> owner AS src_owner, LOOKUP cmdb_assets(src_ip) -> criticality AS src_crit, bytes FROM NetworkFlow WHERE ts > now() - 1h LIMIT 100; ``` Or via REST: ```bash curl -X POST http://localhost:9090/lookup/register \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"name":"cmdb","path":"/data/cmdb.csv","key":"ip","fields":["owner","criticality"]}' ``` --- ## Part 2: Agent memory surface Relata is the **governed memory layer for AI agents**. Agents normally address it through the **memory verbs** (`remember`, `recall`, `recognize`, `justify`, `consolidate`, `forget`), not raw SQL. The verbs are exposed two ways: **MCP tools** (for tool-calling agents) and **`/memory/*` HTTP endpoints** (for direct clients). > The MCP verbs are governed: each requires a `purpose` token (recorded for audit), > even though SQL `PURPOSE` is optional. Results carry provenance and respect > ACL, org isolation, and bi-temporal history. ### Memory-verb lifecycle (MCP tools) Remember a belief, then recall, justify, consolidate, and forget it: ```json // 1. remember — store a governed memory item (bi-temporal + provenance) { "method": "tools/call", "params": { "name": "remember", "arguments": { "content": "Customer Alice plans to renew in Q3.", "session_id": "agent-session-42", "confidence": 0.9, "purpose": "account_management" } } } // 2. recall — hybrid BM25+vector retrieval of relevant memories (optionally AS OF) { "method": "tools/call", "params": { "name": "recall", "arguments": { "query": "What do we know about Alice's renewal?", "session_id": "agent-session-42", "top_k": 5, "as_of": "2026-06-01T00:00:00Z", "purpose": "account_management" } } } // 3. recognize — is this identity already known? { "method": "tools/call", "params": { "name": "recognize", "arguments": { "id": "alice@example.com", "purpose": "account_management" } } } // 4. justify — provenance chain + audit trail for a memory item { "method": "tools/call", "params": { "name": "justify", "arguments": { "id": "", "purpose": "compliance_review" } } } // 5. consolidate — supersede an old belief (keeps full history) { "method": "tools/call", "params": { "name": "consolidate", "arguments": { "id": "", "content": "Alice renewed in Q2.", "confidence": 0.95, "purpose": "account_management" } } } // 6. forget — schedule retention / legal-hold { "method": "tools/call", "params": { "name": "forget", "arguments": { "id": "", "retain_days": 90, "purpose": "data_retention" } } } ``` ### Same verbs over HTTP (`/memory/*`) ```bash curl -X POST http://localhost:9090/memory/remember \ -H "Content-Type: application/json" -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \ -d '{"content":"Alice plans to renew in Q3.","purpose":"account_management"}' curl -X POST http://localhost:9090/memory/recall \ -H "Content-Type: application/json" -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \ -d '{"query":"Alice renewal","top_k":5,"purpose":"account_management"}' ``` ### Data-plane query surfaces The same underlying engine is reachable directly via SQL across these surfaces. All examples use `SELECT * FROM Person LIMIT 5`. ### CLI (SQL string) ```bash relata query "SELECT * FROM Person LIMIT 5" ``` ### CLI (from file) ```bash # Create the query file cat > queries/persons.sql <<'EOF' SELECT * FROM Person LIMIT 5 EOF relata query --file queries/persons.sql relata query -f queries/persons.sql # shorthand ``` ### HTTP REST (JSON) ```bash curl -X POST http://localhost:9090/query \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \ -d '{"sql": "SELECT * FROM Person LIMIT 5"}' ``` Response: ```json { "rows": [ {"name": "Alice", "age": 30}, {"name": "Bob", "age": 25} ], "row_count": 2, "cost_units": 2 } ``` ### Postgres wire protocol (psql / any Postgres client) ```bash psql "host=localhost port=5432 user=relata dbname=relata" \ -c "SELECT * FROM Person LIMIT 5" ``` Any Postgres-compatible driver works: ```python # Python psycopg2 import psycopg2 conn = psycopg2.connect("host=localhost port=5432 dbname=relata user=relata") cur = conn.cursor() cur.execute("SELECT * FROM Person LIMIT 5") print(cur.fetchall()) ``` ### TypeScript / Node.js (REST) ```typescript import fetch from "node-fetch"; const res = await fetch("http://localhost:9090/query", { method: "POST", headers: { "Content-Type": "application/json", "Authorization": `Bearer ${process.env.RELATA_BEARER_TOKEN}`, }, body: JSON.stringify({ sql: "SELECT * FROM Person LIMIT 5" }), }); const data = await res.json(); console.log(data.rows); ``` ### Python (REST) ```python import requests, os r = requests.post( "http://localhost:9090/query", json={"sql": "SELECT * FROM Person LIMIT 5"}, headers={"Authorization": f"Bearer {os.environ.get('RELATA_BEARER_TOKEN', '')}"}, ) print(r.json()["rows"]) ``` ### Go (REST) ```go package main import ( "bytes" "encoding/json" "fmt" "net/http" ) func main() { body, _ := json.Marshal(map[string]string{ "sql": "SELECT * FROM Person LIMIT 5", }) resp, err := http.Post( "http://localhost:9090/query", "application/json", bytes.NewReader(body), ) if err != nil { panic(err) } defer resp.Body.Close() var result map[string]interface{} json.NewDecoder(resp.Body).Decode(&result) fmt.Println(result["rows"]) } ``` ### Agent-oriented: recall from any language Every SDK can drive the memory verbs over `/memory/*`. Example — recall from Python: ```python import requests, os r = requests.post( "http://localhost:9090/memory/recall", json={"query": "Alice renewal", "top_k": 5, "purpose": "account_management"}, headers={"Authorization": f"Bearer {os.environ.get('RELATA_BEARER_TOKEN', '')}"}, ) print(r.json()) # scored memory items with provenance ``` The same `POST /memory/recall` body works from TypeScript, Go, and Rust — swap the HTTP client, keep the JSON. --- ## Part 2b: Multi-tenant queries ### Scoping to an organization Pass `X-Organization-Id` on HTTP requests. Rows written with that header are only returned to principals presenting the same header (or principals with a `SharingAgreement`). Status: **verified** ```bash # Ingest into tenant "acme" curl -s -X POST "http://127.0.0.1:9090/ingest?object_type=Person&purpose=onboarding" \ -H "Content-Type: text/csv" \ -H "X-Organization-Id: acme" \ --data-binary $'name,email\nAlice,alice@acme.com' # Query scoped to "acme" curl -s -X POST http://127.0.0.1:9090/query \ -H "Content-Type: application/json" \ -H "X-Organization-Id: acme" \ -d '{"purpose":"analytics","sql":"SELECT * FROM Person LIMIT 10"}' ``` Without the header the query returns only unscoped rows (no `tenant_id` set). ### Sub-tenant namespaces Use `/`-separated paths to express hierarchy within an organization. A grant on a prefix covers all children. Status: **verified** ``` acme ← top-level tenant acme/eu ← regional sub-tenant acme/eu/hr ← team sub-tenant acme/us ← separate regional sub-tenant (does NOT see acme/eu) ``` ```bash # Write into a sub-tenant curl -s -X POST "http://127.0.0.1:9090/ingest?object_type=Employee&purpose=hr" \ -H "Content-Type: text/csv" \ -H "X-Organization-Id: acme/eu/hr" \ --data-binary $'name\nBob' # A query under acme/eu sees acme/eu AND acme/eu/hr rows curl -s -X POST http://127.0.0.1:9090/query \ -H "Content-Type: application/json" \ -H "X-Organization-Id: acme/eu" \ -d '{"sql":"SELECT * FROM Employee"}' ``` ### Per-tenant read quota defaults The server enforces these limits per agency per query (configurable via `relata.toml`): | Limit | Default | |---|---| | Max in-flight queries per tenant | 10 | | Max query duration | 30 s | | Max scanned rows per query | 10 000 000 | Exceeding any limit returns HTTP 429 with a `Retry-After` header. --- ## Part 2d: Adaptive caching and segment pruning The caching wave adds per-segment temperature tracking, pin-plan coordination, per-column bloom filters, and CMS-driven selectivity estimation. These work transparently — no query changes needed — but query patterns that *align* with the optimizations see the largest gains. ### Bloom-filter-friendly predicates Status: **verified** (bloom pruning is active when `RELATA_BLOOM_COLUMNS` includes the filtered column; segments are skipped at the manifest layer before any row reads). Equality predicates on bloom-indexed columns avoid reading segments that cannot match: ```sql -- tenant_id and object_type are bloom-indexed by default (RELATA_BLOOM_COLUMNS) SELECT * FROM Transaction WHERE tenant_id = 'acme' AND status = 'settled' LIMIT 1000 ``` ```sql -- point-in-time lookup: bloom prunes temporal segments before the AS-OF scan SELECT * FROM CaseRecord AS OF '2026-01-15T00:00:00Z' WHERE tenant_id = 'interpol' LIMIT 50 ``` ### CMS-informed selectivity Status: **verified** (CMS sketches are maintained per `RELATA_SKETCH_COLUMNS`; the cost-based optimizer uses them when deciding whether to apply a secondary index). The optimizer automatically routes high-selectivity predicates through the column index. No query hint is needed: ```sql -- When 'acme' appears in 95% of rows, the CBO skips the tenant_id index (not selective) -- When 'rare-org' appears in 0.1% of rows, the index is used automatically SELECT * FROM NetworkFlow WHERE tenant_id = 'rare-org' AND dst_port = 443 LIMIT 500 ``` ### WITH CACHE hint **verified** The `WITH CACHE` clause controls how the result cache treats a query: ```sql -- pin this result set for the next 10 minutes (TTL override) SELECT * FROM IncidentReport WHERE status = 'open' WITH CACHE TTL 600 LIMIT 5000 ``` ```sql -- honour staleness: treat entries older than 60s as a miss SELECT * FROM IncidentReport WHERE status = 'open' WITH CACHE STALENESS 60 LIMIT 5000 ``` ```sql -- skip the cache and force a fresh read from storage SELECT * FROM AuditLog WHERE event_time > '2026-07-10T00:00:00Z' WITH CACHE BYPASS LIMIT 100 ``` --- ## Part 2e: Search experience — **verified** ### Full-text search with typo tolerance **verified** ```sql -- Balanced mode (default): 1 edit distance, prefix on short tokens SELECT * FROM Person WHERE MATCH(name, 'alice') -- Fuzzy mode: 2 edit distances, catches more typos SELECT * FROM Person WHERE MATCH(name, 'alise', FUZZY) -- Phrase mode: exact phrase match SELECT * FROM Person WHERE MATCH(notes, 'senior analyst', PHRASE) ``` ### Search highlighting **verified** Highlighting is governed by the `/search` REST endpoint's `highlight` flag (and the `attributesToHighlight` array), not a SQL clause — snippets with ``-wrapped matched terms plus offsets are returned alongside each hit: ```bash curl -X POST http://localhost:9090/search \ -H "Authorization: Bearer $TOKEN" \ -d '{"query": "alice", "type": "Person", "highlight": true}' ``` ### Faceted search **verified** ```sql -- Returns facet value counts alongside hits for drill-down SELECT * FROM Product WHERE MATCH(name, 'laptop') FACETS category, brand LIMIT 20 ``` ### Custom ranking rules **verified** ```sql -- Blend BM25 with recency (half-life 1 day) and popularity (weight 0.3) SELECT * FROM Article WHERE MATCH(content, 'quantum') RANK BY recency(published_at, 86400), custom(popularity, 0.3) LIMIT 10 ``` ### Suffix / infix search **verified** ```sql -- Suffix: match names ending in 'son' SELECT * FROM Person WHERE MATCH(name, '*son') -- Infix: match names containing 'iversi' SELECT * FROM Person WHERE MATCH(name, '*iversi*') ``` ### Dedicated /search REST endpoint **verified** ```bash # Search-native JSON API — no SQL required curl -X POST http://localhost:9090/search \ -H "Authorization: Bearer $TOKEN" \ -d '{"query": "alice smith", "type": "Person", "limit": 10, "facets": ["tenant_id"], "highlight": true}' ``` ### Graph operators **verified** ```sql -- Weighted shortest path SELECT * FROM GRAPH_DIJKSTRA('Transaction', FROM => 'Person/A', TO => 'Person/B') -- Link prediction scores SELECT * FROM GRAPH_LINK_PREDICT('Person', FROM => 'Person/A', TO => 'Person/B', METHOD => 'adamic_adar') -- Degree query SELECT name, DEGREE(id, 'out') AS out_degree FROM Person WHERE DEGREE(id, 'out') > 5 -- Strongly connected components SELECT * FROM GRAPH_SCC('Transaction') ``` --- ## CDR analysis Status: **verified** (requires ingested `CdrRecord` rows via `relata cdr ingest `) ### Ingest CDRs from the terminal ```bash # Ingest a CSV file (columns: caller, callee, duration_secs, timestamp_utc) relata cdr ingest calls.csv [--purpose law_enforcement] ``` ### Common-contact hand-off analysis ```sql PURPOSE 'law_enforcement' SELECT callee, COUNT(*) AS call_count, SUM(duration_secs) AS total_secs FROM CdrRecord WHERE caller = '+919876543210' OR callee = '+919876543210' GROUP BY callee ORDER BY call_count DESC LIMIT 20 ``` CLI shorthand: ```bash relata cdr analyze +919876543210 ``` ### Timeline (most recent calls) ```sql PURPOSE 'law_enforcement' SELECT caller, callee, duration_secs, valid_from FROM CdrRecord WHERE caller = '+919876543210' OR callee = '+919876543210' ORDER BY valid_from DESC LIMIT 50 ``` CLI shorthand: ```bash relata cdr timeline +919876543210 ``` ### Identity resolution on CDR contacts ```sql -- Resolve which known person a CDR callee maps to PURPOSE 'law_enforcement' SELECT * FROM RESOLVE_IDENTITY('+447700900123') LIMIT 5 ``` --- ## Part 3: Cookbook rules - Do not add examples as **verified** until tested against the current binary. - If syntax is planned but not implemented, label it **target syntax**. - Purpose is optional; include it when your query has audit/governance requirements. ============================================================================== # Search and retrieval URL: https://relatadb.dev/docs/reference/search ============================================================================== # Search and retrieval RelataDB combines search with governance, time, identity, and provenance. ## Search surfaces | Surface | Purpose | |---|---| | BM25/full-text | keyword search and snippets | | Vector search | similarity over embeddings. Since v1.1 embeddings are caller-supplied — either pre-compute and send `_emb_text` in the row payload, or run the embedder sidecar so the media-worker drain cycle populates them post-ingest. `SIMILAR` ranks by **multi-vector** max-pool over all `_emb_*` slots and resolves the seed by `id` or `_pk`. | | Image near-dup | perceptual-hash (aHash+dHash) match. Images hash in-tree (pure-Rust decode, no sidecar), so a re-encoded/edited copy is found within a small Hamming distance out of the box; video/audio fingerprints still need the decoder sidecar. | | Hybrid retrieval | combine lexical, vector, and graph signals | | Identity lookup | retrieve by canonical identifiers and identity variants | | Graph traversal | relationship/path retrieval | | Temporal filters | retrieve as of valid-time or system-time | | Provenance filters | inspect where assertions came from | ## Ask in plain English (no LLM required) The `nl_query` MCP tool / REST endpoint translates a plain-English question to governed SQL and executes it. With **no `RELATA_LLM_URL` configured** a deterministic in-tree translator handles a documented set of intents — so analysts get structured answers with zero external dependencies. Set `RELATA_LLM_URL` (Ollama / vLLM / LM Studio) for richer free-form parsing. Supported deterministic phrasings: | Intent | Say | Runs | |---|---|---| | Graph path | "paths between `X` and `Y`", "path from `X` to `Y`" | `PATHS_BETWEEN('X','Y', MAX_HOPS => 5)` | | Ownership | "ownership chain of `C`", "who owns `C`", "beneficial owner of `C`" | `BENEFICIAL_OWNERSHIP_CHAIN('C', MAX_DEPTH => 5)` | | Sanctions | "sanctions screen `N`", "is `N` sanctioned" | `SANCTIONS_SCREEN('N', THRESHOLD => 0.85)` | | Browse | anything without an intent verb | `SELECT * FROM LIMIT 50` | A recognised intent whose entities can't be extracted returns a **clear "couldn't translate; try …" error** — it never guesses and runs the wrong query. The active translator (local vs LLM) is reported in the startup posture block. ## World-class search UX RelataDB's search engine includes features inspired by Meilisearch and Typesense. The `/search` REST endpoint accepts JSON parameters for fine-grained control. ### Subword tokenization camelCase, PascalCase, and letter↔digit transitions are split during tokenization so `"iPhone14Max"` produces tokens `["i", "phone", "14", "max"]`. This means `"phoneNumber"` is findable by searching `"phone"`. snake_case and kebab-case already split via the non-alphanumeric pass. Applied symmetrically at index and query time. ### Matching strategy Controls which query terms are *required* vs *optional* when scoring documents. All strategies still BM25-rank results; the difference is which documents are eligible. | Strategy | Behaviour | Best for | |---|---|---| | `any` (default) | Every term is optional (OR) — pre-default behaviour | High recall | | `all` | Every term must be present (AND) | Precision queries | | `last` | Only the last tokenised term is required; earlier terms optional | Search-as-you-type | | `frequency` | Rarest terms required; common terms optional | Mixed rare + common queries | | `boolean` | Query text carries explicit `AND`/`OR`/`NOT` operators | Exact-combination queries | ```bash curl -s -X POST http://127.0.0.1:9090/search \ -H 'Content-Type: application/json' \ -d '{"query":"alice wonderland","type":"Person","limit":10,"matching_strategy":"all"}' ``` **Boolean operators** — with `"matching_strategy": "boolean"`, uppercase `AND` / `OR` / `NOT` in the query text combine per-term BM25 matches as posting-list set operations (left-associative; a bare space means OR): ```bash curl -s -X POST http://127.0.0.1:9090/search \ -H 'Content-Type: application/json' \ -d '{"query":"sanction OR ofac NOT expired","type":"Alert","limit":10,"matching_strategy":"boolean"}' ``` The same grammar is reachable from SQL as `MATCH(col, 'q', BOOLEAN)` and from the typed `/search` door as `rank_by: ["boolean", "", ""]`. ### Per-query typo tolerance Override the global `RELATA_SEARCH_PRESET` per query. Gate fuzzy expansion by word length, disable specific words, or turn typos off entirely. ```json { "query": "john smith", "type": "Person", "limit": 10, "typo_tolerance": { "enabled": true, "min_word_size": 5, "disable_on_words": ["smith"], "disable_on_attributes": ["email"] } } ``` | Field | Default | Description | |---|---|---| | `enabled` | `true` | Master toggle. `false` = exact-match only. | | `min_word_size` | `0` | Words shorter than this stay exact (Meili default: 5 for 1-edit, 9 for 2-edit). | | `disable_on_words` | `[]` | Specific query terms that should not be fuzzy-expanded. | | `disable_on_attributes` | `[]` | Field names where typo tolerance is disabled. | #### Typo tolerance in SQL MATCH — `POST /query` The same `typo_tolerance` object is accepted by `POST /query` alongside the SQL text. When set, SQL `MATCH` / `PHRASE` / `SUFFIX` / `INFIX` conditions apply per-token levenshtein fuzzy matching: 1 edit for words ≥ `min_word_size` (default: any length), 2 edits for words ≥ 9 chars. ```json POST /query { "sql": "SELECT * FROM Person WHERE MATCH(name, 'jon')", "typo_tolerance": { "enabled": true, "min_word_size": 3 } } ``` This will also return rows where `name` contains `"john"` (1-edit match). ### BM25F per-field weights Boost results where a query term appears in high-priority fields (e.g. `title` outweighs `body`). Pass `fieldWeights` — a map of field names to `f32` multipliers. The BM25 score is scaled by the highest weight of any named field that contains a query term. Fields absent from the map default to `1.0`. ```json { "query": "database", "type": "Article", "limit": 10, "fieldWeights": {"title": 3.0, "body": 1.0, "tags": 5.0} } ``` Rows whose query terms appear only in `tags` are scaled ×5; rows where the term appears in `title` are scaled ×3; rows with the term only in `body` are unaffected. Omitting `fieldWeights` (or passing `{}`) leaves all scores unchanged. ### Faceted search The `FACETS` SQL clause returns per-attribute value counts. Facets are computed over the **full matching set** (up to 100 000 rows), not just the top-k returned rows — so browse-UI facet counts are accurate even when `LIMIT` is small. For numeric facet columns, the response also includes `facetStats` with `min`, `max`, `sum`, `avg`, and `count`: ```json { "facets": {"status": {"active": 42, "closed": 7}}, "facetStats": {"amount": {"min": 100, "max": 50000, "sum": 125000, "avg": 2314, "count": 54}} } ``` ### Estimated total hits Both `/query` and `/search` responses include `estimatedTotalHits` — the full matching-set size (accurate up to the 100k facet cap; a lower bound otherwise) — alongside `count`/`rows` (the actual returned rows). Use this for pagination UX. ### Multi-search / federated `POST /multi-search` runs N independent queries in one round-trip and returns combined results in input order — useful for dashboard widgets and federated type searches: ```bash curl -s -X POST http://127.0.0.1:9090/multi-search \ -H 'Content-Type: application/json' \ -d '{"queries":[{"query":"alice","type":"Person","limit":5},{"query":"transfer","type":"Transaction","limit":10}]}' ``` ## Typed query door — `namespace().query()` (no SQL) For the search / RAG persona, `/search` accepts a **second body shape** — a typed JSON query that compiles to a governed SQL plan server-side, so you never hand-build SQL. Governance (purpose, ACL, cell masking, audit) is identical to a hand-written query. The typed shape is detected by the **absence** of a `query` key (its presence keeps the legacy Meilisearch shape above on the same route). ```bash curl -X POST http://127.0.0.1:9090/search \ -H 'Content-Type: application/json' \ -d '{"from":"Document", "rank_by":["bm25","title","graph retrieval"], "filters":[{"field":"status","op":"eq","value":"published"}], "include_attributes":["id","title"],"limit":10}' # returns the governed /query envelope: {rows, columns, ...} ``` | Field | Description | |---|---| | `from` (alias `type`) | Object type to query. | | `rank_by` | `["bm25"\|"text","",""]` (BM25 full-text, `ORDER BY _score DESC`), or `["vector","ann",""]` (HYBRID_SEARCH — server embeds the text). | | `filters` | `[{field, op, value}]` (ops `eq\|ne\|gt\|gte\|lt\|lte\|like\|ilike\|in\|between`), a `{and:[...]}` / `{or:[...]}` object, or a single condition. | | `include_attributes` | Column projection (default `*`). | | `consistency` | `"strong"` (default) \| `"eventual"` (forward-compat). | | `compute_attributes` | `{label: rank_expr}` compiled to a trailing `COMPUTE` clause — see below. | ### Python SDK — flagship `namespace()` surface ```python from relata import RelataClient with RelataClient("http://localhost:9090", purpose="rag") as client: docs = client.namespace("Document") docs.write([{"id": "d1", "title": "Knowledge graphs 101", "body": "..."}]) # schemaless (POST /ingest/auto) res = docs.query( text="graph retrieval", match_column="title", filters=[{"field": "status", "op": "eq", "value": "published"}], limit=10, ) for row in res: print(row["title"]) ``` `.write()` is schemaless: `POST /ingest/auto` auto-creates the type on first write with field types inferred from the rows — no DDL. Async mirrors (`AsyncNamespace`, `await docs.query(...)`) and one shared connection pool across every namespace. ### Composable ranking + COMPUTE side outputs (SQL) In SQL, ranking is a full expression tree and per-hit signals are a `COMPUTE` clause (the typed door's `rank_by`/`compute_attributes` lower to these): ```sql -- Blend BM25 with a saturated popularity signal, a time decay, and geo distance. SELECT * FROM Article WHERE MATCH(content, 'ai') RANK BY SUM(BM25(), PRODUCT(0.5, SATURATE(popularity, 100, 2)), DECAY(published_at, 86400, 1), DIST(location, 1.0, 2.0)) LIMIT 10; -- Attach per-hit signals without changing matching or order. SELECT * FROM Doc WHERE MATCH(content, 'ai') RANK BY VECTOR() COMPUTE bm25_score AS BM25(), snippet AS HIGHLIGHT(content) LIMIT 10; ``` Ranking primitives: `SUM` / `MAX` / `PRODUCT` / `SATURATE` / `DECAY` / `ATTRIBUTE(name)` / `DIST(attr, x, y, …)` / `BM25()` / `VECTOR()`. `HIGHLIGHT(field)` is compute-only (renders a snippet). ### Namespace branching + pinning - **Branch** — constant-time COW: `BRANCH dev FROM main` (SQL) or `POST /v1/namespaces/dev/branch` with `{"branch_from":"main"}`. See [Branching & Namespaces](/docs/reference/branching) for the full surface (including how namespace branches differ from schema branches and sub-tenant namespaces). - **Pin** — `RELATA_PINNED_NAMESPACES=acme,contoso` reserves a dedicated NVMe cache slice per hot namespace so its index is evict-immune under pressure. ### Multi-query batch + reciprocal-rank fusion (RRF) One round-trip, up to **16 typed subqueries**, fused with rank-based RRF. This is the "build a search dashboard in one call" feature — multiple ranking strategies against one snapshot, merged into a single ranked list. `POST /search` with `{"queries": [...], "rerank_by": ["RRF", {"rank_constant": 60}], "limit": 20}`: ```bash curl -X POST http://127.0.0.1:9090/search \ -H 'Content-Type: application/json' \ -d '{ "queries": [ {"from":"Document", "rank_by":["bm25","title","graph retrieval"], "limit":50}, {"from":"Document", "rank_by":["field_weight",{"title":3.0,"body":1.0}, "graph retrieval"], "limit":50}, {"from":"Document", "rank_by":["vector","ann","graph retrieval"], "limit":50} ], "rerank_by": ["RRF", {"rank_constant": 60}], "limit": 20 }' ``` **What's happening:** - All subqueries are pinned to a single `AS OF SYSTEM TIME` snapshot, sampled once before fan-out — concurrent writes cannot split results. - Each subquery runs through the same governed `/query` path (PURPOSE, ACL, tenant, cluster fan-out identical to N independent calls). - `rerank_by: ["RRF", {rank_constant: K}]` fuses the per-subquery ranked lists using reciprocal-rank fusion (`score = Σ 1/(K + rank_i)`). Default `K = 60` (the standard RRF constant). Fused rows carry `_rrf_score`. - **Vector/ANN `rank_by` is rejected in batch mode** (the `AS OF` clause isn't in the `HYBRID_SEARCH` grammar today) — use BM25 / field-weight subqueries for the batch path. **Response shape:** ```json { "results": [ {/* per-subquery result set */}, {/* ... */} ], "as_of_system": 1735490000000000000, "processing_time_ms": 18, "fused": { "rank_constant": 60, "rows": [ /* merged, ranked, with _rrf_score */ ], "data": [ /* ... */ ] } } ``` ### Tips & takeaways - **RRF needs ≥ 2 subqueries.** A single subquery with `rerank_by: ["RRF"]` is rejected — there's nothing to fuse. - **Different `rank_by` strategies make RRF sing.** Combine BM25 (lexical) + `field_weight` (weighted BM25F) + filter variants; each contributes a different ranking signal, and RRF is robust to score-scale differences (it uses *ranks*, not scores). - **One snapshot = consistent dashboarding.** Because all subqueries share one `AS OF`, the facets/counts/hits in each pane are mutually consistent — no flicker from writes landing mid-search. - **`rank_constant` tuning:** lower `K` (e.g. 1) weights top ranks more heavily (good when you trust the top of each list); higher `K` (e.g. 100) smooths toward a popularity vote. 60 is the literature default and a safe start. - **Governance is per-subquery, not bypassed.** PURPOSE / ACL / cell masking apply to every subquery independently — a row you can't see in one subquery won't appear in the fused result either. Cross-ref: [Hybrid search (concepts)](/docs/concepts/hybrid-search) · [Vector params](/docs/reference/vector-params) · [SQL `RANK BY` expression](#composable-ranking--compute-side-outputs-sql) · [Search & Retrieval reference top](/docs/reference/search) ## Governance rules Search is not a policy bypass. Search and retrieval paths must preserve: - optional purpose (recorded for audit), - ACL pushdown, - cell masking, - provenance handling, - temporal semantics, - audit/cost accounting. ## Example topics - Search documents by text. - Search as of a system time. - Retrieve with provenance. - Combine identity lookup with document search. - Explain why a restricted result is masked or absent. ## Validation rule Any published latency/recall claim must cite a benchmark, conformance row, or reproducible command output. ============================================================================== # SPARQL URL: https://relatadb.dev/docs/reference/sparql ============================================================================== # SPARQL Relata exposes a **SPARQL 1.1 basic SELECT** read endpoint (`/sparql`) that translates queries to SQL over the provenance graph's `triples` view and executes them through the governed query path. Available since v0.9. > **Conformance:** SPARQL 1.1 basic SELECT — BGP triple patterns + `LIMIT`. This is a deliberate subset, not a full SPARQL 1.2 processor. Full SPARQL 1.2 processing requires the `sparql-rdf` feature flag (`spargebra`), not yet in workspace dependencies. For the full query surface, use [SQL](/docs/reference/sql). ## Endpoint | Method | Path | Body / Query | |---|---|---| | `GET` | `/sparql` | `?query=` | | `POST` | `/sparql` | `Content-Type: application/sparql-query` (body is the query) | Both return JSON. ### GET ```bash curl "http://localhost:9090/sparql?query=SELECT%20%3Fs%20%3Fp%20%3Fo%20WHERE%20%7B%20%3Fs%20%3Chttp%3A%2F%2Fwww.w3.org%2Fns%2Fprov%23wasAttributedTo%3E%20%3Fo%20%7D%20LIMIT%2010" ``` Equivalent unencoded query: ```sparql SELECT ?s ?p ?o WHERE { ?s ?o } LIMIT 10 ``` ### POST ```bash curl -X POST http://localhost:9090/sparql \ -H "Content-Type: application/sparql-query" \ -d "SELECT ?s ?o WHERE { ?s ?o } LIMIT 5" ``` ## Authentication Same as `/query`. When `RELATA_BEARER_TOKEN` is set, include it: ```bash curl -H "Authorization: Bearer $TOKEN" "http://localhost:9090/sparql?query=..." ``` ## Response format ```json { "sparql": "SELECT ?s WHERE { ?s ?p ?o } LIMIT 5", "sql": "SELECT * FROM triples LIMIT 5", "rows": 2, "columns": ["s", "p", "o"], "data": [ {"s": "urn:relata:row:...", "p": "...", "o": "..."} ], "processing_time_ms": 3 } ``` The translated SQL is echoed in the `sql` field, and `processing_time_ms` follows the Meilisearch/Typesense convention. ## Supported subset The translator (`relata_query`'s SPARQL→SQL translator) covers: | Construct | Status | |---|---| | `SELECT ?vars WHERE { ?s ?o }` | Supported | | Multiple BGP triple patterns (inner join) | Supported | | `LIMIT n` | Supported | | `OPTIONAL` | 400 — use `LEFT JOIN` in SQL | | `FILTER` | 400 — add the predicate to SQL `WHERE` | | `GRAPH`, `CONSTRUCT`, `ASK`, `DESCRIBE` | 400 | | `ORDER BY`, `OFFSET`, `DISTINCT`, `UNION` | 400 — use SQL equivalents | | Federated `SERVICE` | 400 — Relata is single-source | ## RDF / PROV-O predicates The provenance graph is exposed via the PROV-O vocabulary (`http://www.w3.org/ns/prov#`): | Predicate | Meaning | |---|---| | `prov:wasGeneratedBy` | Activity that produced the row | | `prov:wasDerivedFrom` | Prior assertion the row derives from | | `prov:wasAttributedTo` | Actor (human/agent/system/service) that asserted the row | | `prov:generatedAtTime` | `system_from` timestamp | RDF-star reified triples (per-triple provenance) require `spargebra` and are deferred. ## Configuration None required. `/sparql` is enabled on every deployment profile (`free`, `server`, `cluster`) when `RELATA_PROFILE` is set. Auth is mandatory on `server`/`cluster` and optional on `free`, identical to `/query`. ## See also - [SQL Reference](/docs/reference/sql) — the full dialect (use this for everything beyond basic SELECT) - [Provenance](/docs/concepts/provenance) — the PROV-O model behind the `triples` view - [GraphQL](/docs/reference/graphql) — the other query-language door ============================================================================== # SQL Reference URL: https://relatadb.dev/docs/reference/sql ============================================================================== # SQL Reference RelataDB extends ANSI SQL with bi-temporal reads, provenance, identity resolution, graph traversal, hybrid search, and agent memory verbs. The same SQL runs across every protocol door: `psql`, gRPC, HTTP `/query`, MCP `query`, Arrow Flight, and the SPARQL bridge. > **What runs today:** this page documents verified behaviour. Items marked *not implemented* or *partial* are not available. See [Limits](/docs/reference/limits) for the full status table. ## Statement shape ```sql [EXPLAIN POLICY] [PURPOSE ''] SELECT FROM [AS OF ''] [AS OF SYSTEM TIME ''] [WHERE ] [ORDER BY [ASC|DESC]] [LIMIT n [AFTER '']] [WITH PROVENANCE] ``` All clauses are order-sensitive. `WITH PROVENANCE` must be last. ## PURPOSE `PURPOSE ''` is an optional prefix. When omitted the query runs; when declared it is recorded in the audit log. Per-tenant ACL policy can require it. ```sql -- runs, no audit record SELECT * FROM Person LIMIT 10 -- runs, audited under purpose "analytics" PURPOSE 'analytics' SELECT * FROM Person LIMIT 10 -- returns the ACL decision tree without executing EXPLAIN POLICY PURPOSE 'analytics' SELECT * FROM Person LIMIT 10 ``` Common values: `analytics`, `audit`, `compliance_review`, `security_incident`, `research`, `operations`. ## AS OF — bi-temporal snapshots `AS OF` returns rows whose `valid_from <= ts < valid_to`. `AS OF SYSTEM TIME` uses system-recorded time instead. ```sql -- valid-time snapshot SELECT * FROM Document AS OF '2026-01-01T00:00:00Z' -- system-time snapshot SELECT * FROM Document AS OF SYSTEM TIME '2026-01-01T00:00:00Z' ``` Timestamp formats accepted: UTC ISO-8601 (`YYYY-MM-DD`, `YYYY-MM-DDTHH:MM:SS`, `YYYY-MM-DDTHH:MM:SS.fffZ`). Non-UTC offsets are rejected. Underlying storage is `i64` nanoseconds UTC. ## WITH PROVENANCE Trailing modifier — must come after `LIMIT`: ```sql SELECT * FROM Event LIMIT 10 WITH PROVENANCE ``` Adds a parallel `provenance` array to the response (one entry per row). Each entry exposes `source`, `method`, `confidence`, `recorded_at`, and `derived_from` (hex `ProvenanceRef` of the source row, or `null` for genesis). ## Keyset pagination: LIMIT n AFTER ```sql SELECT * FROM Person LIMIT 100 AFTER '1798765432100000000' ``` The cursor is the previous page's last `system_from` value (decimal nanoseconds). Cannot combine with `ORDER BY`. ## EXPLAIN ```sql EXPLAIN POLICY PURPOSE 'analytics' SELECT * FROM Person LIMIT 10 EXPLAIN_PATH('alice@example.com', 'bob@example.com') EXPLAIN_REPLAY('', SEQ => 5) ``` | Variant | Returns | |---|---| | `EXPLAIN POLICY` | ACL decision tree + cell-mask plan for the requesting principal | | `EXPLAIN_PATH` | `{strategy, graph_node_count, pll_warm}` for a graph path probe | | `EXPLAIN_REPLAY` | Re-derives a logged exhibit link's seal byte-identically | > `EXPLAIN POLICY` is parsed as a prefix but the flag is not consulted on the execute path. Use the `explain_policy` MCP tool for a live ACL decision. ## WHERE expressions | Form | Status | |---|---| | `col op literal` (`=`, `!=`, `<`, `<=`, `>`, `>=`, `LIKE`) | Working | | Arithmetic expressions | Working | | `now()` | Working — resolved at parse time | | `now() - INTERVAL 'N days\|hours\|minutes\|weeks'` | Working | | `MATCH(col, 'q'[, PHRASE\|FUZZY\|STEMMED\|BOOLEAN])` | Working | | `SOUNDEX(col) = 'name'` / `METAPHONE(col)` / `COLOGNE(col)` | Working — phonetic name-variant matching | `MATCH` modes: default resolves from the BM25 posting list; `PHRASE` uses the positional index; `FUZZY` and `STEMMED` fall back to substring scan; `BOOLEAN` evaluates uppercase `AND`/`OR`/`NOT` operators as posting-list set operations (a bare space means OR). Phonetic predicates (`SOUNDEX`/`METAPHONE`/`COLOGNE`) encode the right-hand literal to the phonetic key at parse time and support only `=` / `!=`. ## Aggregation and JOIN | Feature | Status | |---|---| | `JOIN` (INNER, hash join) | Working | | `GROUP BY` + `COUNT(*)` | Working | | `ORDER BY` (single column, ASC/DESC) | Working | | Multi-column `ORDER BY` (tiebreakers) | Working | | `UNION` / `UNION ALL` / `INTERSECT` / `EXCEPT` | Working | | CTE (`WITH [RECURSIVE] … AS (…)`) | Working | | `TUMBLE` / `HOP` / `SESSION` streaming windows | Working | ## Identity operators ```sql -- Universal lookup by value (phone, email, IMEI, IBAN, IP, BTC address, …) SELECT * FROM LOOKUP_IDENTITY('+919876543210') -- Resolution modes: cluster (default), canonical, fuse SELECT * FROM RESOLVE_IDENTITY('alice@example.com') SELECT * FROM RESOLVE_IDENTITY('alice@example.com', MODE => 'canonical') SELECT * FROM RESOLVE_IDENTITY('alice@example.com', MODE => 'fuse') ``` `RESOLVE_IDENTITY(..., MODE => 'fuse')` dispatches the EnrichmentRule chain. Returns an error when no rules are registered. > The SQL keyword is the singular `RESOLVE_IDENTITY`. The DataFusion TVF form `SELECT * FROM resolve_identity(...)` is also accepted and translated to the governed keyword form. ### Entity merge / dedup ```sql -- Ontological merge — link two identities as one person FUSE_IDENTITIES('alice@example.com', 'bob@example.com') -- Ontological unmerge — reverse a fuse SPLIT_IDENTITIES('alice@example.com', 'bob@example.com') -- SmartIngest auto-detect identities in free text DETECT_IDENTITIES('call bob at +971501234567') -- GDPR Art. 17 erasure — irreversible ERASE SUBJECT 'alice@example.com' REASON 'gdpr-art17' CERTIFY ``` ### Graph operators ```sql -- Shortest paths (Pregel BFS) SELECT * FROM PATHS_BETWEEN('alice', 'bob', 3) SELECT * FROM PATHS_BETWEEN('person-123', 'org-456', MAX_HOPS => 4) -- Network expansion SELECT * FROM NETWORK_EXPAND('entity-X', MAX_HOPS => 3) -- Per-node degree SELECT id, DEGREE(id) AS degree FROM Person ORDER BY degree DESC LIMIT 10 ``` > `NETWORK_EXPAND` takes the seed identifier as the first positional arg and the hop cap via the `MAX_HOPS => n` named arg (default `2`), with an optional `LINK_TYPES => 'a,b'` filter. Additional graph TVFs (all SQL-reachable): | TVF | Description | |---|---| | `GRAPH_DIJKSTRA('', FROM => 'a', TO => 'b')` | Weighted shortest path | | `GRAPH_SCC('')` | Strongly connected components | | `GRAPH_CYCLES('')` | Cycle detection | | `GRAPH_SSSP('', FROM => 'a')` | Single-source shortest paths | | `GRAPH_SPANNING_TREE('')` | Minimum spanning tree | | `GRAPH_APSP('')` | All-pairs shortest paths | | `GRAPH_DIAMETER('')` | Graph diameter | | `GRAPH_SIMILARITY('', TOP_K => n)` | Structural similarity | | `GRAPH_NODE_METRIC('', METRIC => 'kcore')` | Betweenness, PageRank, clustering coefficient | | `GRAPH_LINK_PREDICT('', FROM => 'a', TO => 'b', METHOD => 'adamic_adar')` | Link prediction score | ## Search operators ```sql -- Full-text search (BM25, custom engine — not Tantivy) SELECT * FROM Person WHERE MATCH(name, 'Ahmed Khalil') -- Hybrid BM25 + vector (reciprocal-rank fusion) HYBRID_SEARCH FROM Document QUERY 'terror finance' LIMIT 25 -- Embedding-vector similarity against a seed row (multi-vector max-pool) SIMILAR TO Document WHERE id = 'doc-42' LIMIT 10 -- pgvector-compatible KNN operators (pgwire door only) SELECT id FROM docs ORDER BY embedding <=> '[0.9,0.1,0]' LIMIT 5 ``` | Operator | Metric | Note | |---|---|---| | `<=>` | cosine distance | Preferred — ANN index is cosine-only | | `<->` | L2 distance | Metric-correct via over-fetch + re-rank | | `<#>` | negative inner product | Metric-correct via over-fetch + re-rank | ## Multimodal similarity ```sql -- Similar-to a reference (multi-vector max-pool cosine over _emb_* slots) SIMILAR TO Person WHERE id = 'person-42' SIMILAR TO MultimodalAsset WHERE id = 'asset-7' -- Near-duplicate / rough-visual-similarity image search (perceptual hash + Hamming index) SIMILAR_IMAGE('media-42') SIMILAR_IMAGE('media-42', THRESHOLD => 0.6, INDEX => 'ncmec') ``` `SIMILAR TO` falls back to Jaccard token similarity when the reference has no embedding slots. Face and voice search are exposed as the `FACE_SEARCH` / `VOICE_MATCH` operators and the MCP `face_match` / `voice_match` tools. `SIMILAR_IMAGE(media_ref, THRESHOLD => n, INDEX => corpus)` matches by perceptual hash rather than embedding — `THRESHOLD` defaults to `0.9`; `INDEX` optionally scopes the Hamming-index lookup to a named corpus (e.g. `'ncmec'` for CSAM near-dup triage). ## Agent memory verbs ```sql REMEMBER 'Alice plans to renew in Q3.' SESSION 'sess-1' CONFIDENCE 0.9 RECALL 'Alice renewal' TOP_K 5 AS OF '2026-01-01' REFLECT ON SESSION 'sess-1' CONSOLIDATE MEMORY 'mem-uuid-old' WITH 'Alice confirmed renewal for Q3.' FORGET MEMORY 'mem-uuid' RETAIN_DAYS 90 ASSOCIATE MEMORY 'mem-a' WITH 'mem-b' AS 'contradicts' CONFIDENCE 0.8 RESOLVE MEMORY 'mem-uuid' POLICY 'highest_confidence' ``` The same verbs are available via MCP tools and HTTP `/memory/*` endpoints. ## Domain TVFs All reachable from SQL under the governed keyword translation: | TVF | Domain | Description | |---|---|---| | `BENEFICIAL_OWNERSHIP_CHAIN(entity, max_depth)` | FinINT | Beneficial ownership chain | | `SANCTIONS_SCREEN(name[, THRESHOLD => 0.75])` | FinINT | Sanctions screening (default Jaccard threshold `0.75`) | | `CRYPTO_TRACE(wallet, max_hops, min_amount)` | FinINT | BFS over `TransactionGraph` | | `WIRE_RECONSTRUCTION(...)` | FinINT | Wire transfer reconstruction | | `HAWALA_TRACE(...)` | FinINT | Hawala network trace | | `GRAPH_COMMUNITY(...)` | Graph | Community detection | | `GEOFENCE(...)` | Geo | Geofence lookup | | `ANPR_TRACE(...)` | Intel | ANPR plate trace | | `DISPATCH_PRIORITY(...)` | Ops | Dispatch priority | | `CRIME_PATTERN_CLUSTER(...)` | Intel | Crime pattern clustering | ## Social-media analytics scorers All 13 `ScorerOp` operators are SQL-reachable: `SENTIMENT_SCORE`, `STANCE_SCORE`, `BIAS_SCORE`, `AUTHENTICITY_SCORE`, `POSTING_PATTERN`, `STYLE_FINGERPRINT`, `BOT_AMPLIFICATION_SCORE`, `NARRATIVE_TRACE`, `PERSONA_CLUSTER_DETECT`, `INAUTHENTIC_BEHAVIOUR_SCAN`, `COORDINATED_AMPLIFICATION`, `CROSS_PLATFORM_ACCOUNT`, `INFLUENCE_TIER`. ## Lookup tables ```sql -- Register once (survives until restart) REGISTER LOOKUP cmdb_assets FROM '/data/cmdb.csv' KEY (ip) FIELDS (owner, criticality, environment) REFRESH EVERY 5 MINUTES; -- Enrich query results at run time SELECT src_ip, LOOKUP cmdb_assets(src_ip) -> owner AS src_owner, LOOKUP cmdb_assets(src_ip) -> criticality AS src_crit, bytes FROM NetworkFlow WHERE ts > now() - INTERVAL '1 hour' LIMIT 100; ``` ## Materialized views ```sql CREATE MATERIALIZED VIEW active_persons AS SELECT * FROM Person WHERE status = 'active' REFRESH INCREMENTAL EVERY 60; ``` Background refresh runs every 60 seconds. `RELATA_MV_MAX_ROWS` (default 1,000,000) caps cached rows before eviction to base-table fallback. ## WATCH ```sql WATCH PURPOSE 'security_incident' SELECT * FROM Event WHERE severity = 'high' ``` Registers a subscription — pushes matching rows to the caller as they arrive. Available over the SSE `/watch/stream` endpoint. ## ERASE SUBJECT (GDPR Art. 17) ```sql ERASE SUBJECT 'person-42' REASON 'gdpr-art17' CERTIFY; ``` `REASON ''` is required; the trailing `CERTIFY` keyword acknowledges the operation is destructive and irreversible. Shreds rows, orphaned blobs, and the per-subject DEK (KMS-wired, fail-closed). Returns a signed Art. 17 receipt. ## DDL RelataDB is ontology-governed. Types are declared via the ontology, not `CREATE TABLE`. DDL is supported only via the pgwire door for pgvector compatibility. | Statement | Status | |---|---| | `CREATE EXTENSION vector` / `DROP EXTENSION vector` | Working (pgwire, no-op OK tag) | | `CREATE TABLE` | Working (pgwire) — registers the type; column list is ignored | | `INSERT` / `UPDATE` / `DELETE` | Working (pgwire) — the only native DML path | | `CREATE MATERIALIZED VIEW … REFRESH INCREMENTAL EVERY ` | Working | | `ALTER TABLE … ADD/DROP COLUMN` | Parses, then 501s — use the ontology API | | `CREATE INDEX` / `CREATE TYPE` / `CREATE SCHEMA` | Not implemented | ## Clause grammar & capability matrix Code-verified per-clause status. "Working" produces correct results under `cargo test`; "Partial" parses but output is incomplete; "Not implemented" returns a 400 or empty result. See [Limits](/docs/reference/limits) for deeper caveats. ### SELECT projection | Form | Status | Example | |---|---|---| | `SELECT *` / `SELECT col, col` | Working | `SELECT name, age FROM Person` | | `COUNT(*)`, `SUM/AVG/MIN/MAX(col)` | Working | `SELECT SUM(amount) FROM Transaction` | | Nested aggregates `COUNT(SUM(...))` | **Not implemented** | Use separate queries | | `COALESCE(col, default)` | **Not implemented** | Handle nulls in the application layer | | `CASE WHEN col = v THEN ... [ELSE ...] END` | Working | Equality-form `CASE WHEN` over a column | | `col AS alias` / `DISTINCT` | Working | `SELECT DISTINCT col FROM ...` | ### FROM, JOIN, subquery | Feature | Status | Note | |---|---|---| | Single type | Working | `FROM Person` | | `INNER JOIN` | Working | Hash join, O(n+m) | | `LEFT/RIGHT/FULL [OUTER] JOIN` | Working | Outer-join variants parse and execute | | Subquery in FROM | Working | Bounded by `MAX_PARSE_DEPTH = 40` | | CTE `WITH [RECURSIVE] ... AS (...)` | Working | Recursive CTEs split on `set_ops` | ### WHERE predicates | Predicate | Status | |---|---| | `col op literal` (`=`,`!=`,`<`,`<=`,`>`,`>=`,`LIKE`) | Working | | `IS NULL` / `IS NOT NULL` | Working | | `IN (a, b, c)` and `IN (SELECT ...)` | Working (uncorrelated; correlated resolved per outer row) | | `[NOT] BETWEEN a AND b` | Working (inclusive; numeric, temporal, string) | | `AND` / `OR` / `NOT`; `col op col`; arithmetic `col op (expr op literal)` | Working | | `now()` and `now() - INTERVAL 'N days\|hours\|minutes\|weeks\|months\|years'` | Working (months ≈ 30d, years ≈ 365d) | | `MATCH(col, 'q'[, PHRASE\|FUZZY\|STEMMED\|BOOLEAN])` | Working | | `SOUNDEX(col) = 'name'` / `METAPHONE(col)` / `COLOGNE(col)` | Working — phonetic name-variant matching | | SQL:2011 `FOR SYSTEM_TIME FROM/TO/BETWEEN`; `OVERLAPS`/`PRECEDES`/`SUCCEEDS` | **Not implemented** — use `AS OF` or explicit `valid_from`/`system_from` comparisons | ### ORDER BY, LIMIT, pagination | Feature | Status | Note | |---|---|---| | `ORDER BY col [ASC\|DESC]`; multi-column | Working | Secondary keys via tiebreakers | | `LIMIT n` | Working | | | `LIMIT n AFTER 'cursor'` | Working | Cursor = `system_from` ns (decimal); cannot combine with `ORDER BY` | | `OFFSET n` | **Not implemented** | Use cursor-based pagination | ### GROUP BY, windows, HAVING | Feature | Status | |---|---| | `GROUP BY col` | Working | | `GROUP BY TUMBLE/HOP/SESSION(ts, ...)` (streaming windows) | Working | | `HAVING` (post-aggregation predicate) | Working | | Window functions `ROW_NUMBER()`, `RANK()` | **Not implemented** | | `FILTER` within aggregate | **Not implemented** | ### EXPLAIN variants | Variant | Status | Returns | |---|---|---| | `EXPLAIN ` | Working | Physical plan (scan access, join order, spill decision); does not execute | | `EXPLAIN ANALYZE ` | Working | Runs the query, appends `Analyze` stage + `actual_rows` + `processing_time_ms` | | `EXPLAIN_PATH('a','b')` | Working | `{strategy, graph_node_count, pll_warm}` | | `EXPLAIN_REPLAY('', SEQ => n)` | Working | Re-derives an exhibit link's seal byte-identically | | `EXPLAIN POLICY` | Partial | Parses; flag not consulted on execute path — use the `explain_policy` MCP tool | ### Cluster fan-out aggregates On `cluster` profile with peers, fan-out merge supports `SUM(col)`/`SUM(*)` (merged across shards), `AVG(col)`/`AVG(*)` (weighted merge), and `COUNT(*)` (summed across shards). ### DDL (pgwire path only) These statements only work over the **pgwire listener** (port 5433 by default), not `/query`: | Statement | Status | |---|---| | `CREATE EXTENSION vector` / `CREATE TABLE` / `INSERT`/`UPDATE`/`DELETE` | Working (pgwire) | | `CREATE MATERIALIZED VIEW ... REFRESH [INCREMENTAL] EVERY ` | Working | | `ALTER TABLE` | Parses, then 501s — use `POST /ontology/migrate` or `POST /types/:name/schema` | | `CREATE INDEX` / `CREATE TYPE` / `CREATE SCHEMA` | Not implemented — use `POST /ontology/migrate` (SHACL) or `POST /types` (runtime) | ### Ingest body formats `POST /ingest?object_type=&purpose=

` auto-detects the body format (no `Content-Type` change required — the server inspects the first byte): | Format | Detection | Example | |---|---|---| | **CSV** | Default (no leading `{` or `[`) | `name,age\nAlice,30` | | **NDJSON** | Leading `{` (one JSON object per line) | `{"name":"Alice","age":30}\n{...}` | | **JSON Array** | Leading `[` | `[{"name":"Alice","age":30}]` | > The `purpose` parameter is validated against `^[a-zA-Z_][a-zA-Z0-9_]{0,63}$` — use underscores, not hyphens (e.g. `threat_intel`, not `threat-intel`); max 64 chars. A rejected purpose returns 400 with a hint showing the underscore-normalised form. ### Custom types | Operation | Status | Note | |---|---|---| | `POST /types` | Working | Register a custom type at runtime; persisted to `custom_types.jsonl` | | `DELETE /types/:name` | Working | Admin-only de-registration | | `RELATA_ACL_GRANT=MyType:read+write` | Working | Grants `api-user` + `mcp-client` read/write on named types | | Unknown type on ingest/read | 400 | Fail-closed with guidance to register via `POST /types` | ### Detection-rule SQL guard Detection-rule conditions are validated at create + eval time: rejects `;`, `--`, `/* */` markers; rejects `UNION`/`INTERSECT`/`EXCEPT` adjacent to punctuation; 2048-byte length cap on condition text. ## Not yet implemented | Feature | Notes | |---|---| | `OFFSET n` pagination | Use cursor-based `LIMIT n AFTER ''` | | Multi-column `DISTINCT` | — | | SQL:2011 period predicates | Use explicit `valid_from`/`system_from` predicates; `AS OF` covers point-in-time | | `EXPLAIN POLICY` live evaluation | Use the `explain_policy` MCP tool | | User-defined functions | Use the MCP tool surface | ============================================================================== # Vector index parameters reference URL: https://relatadb.dev/docs/reference/vector-params ============================================================================== # Vector index parameters reference How to tune Relata's HNSW + DiskANN + IVF vector indexes. Covers index creation, parameter selection, distance metrics, quantization, and search-time knobs. ## Index lifecycle A vector index lives on a typed table for every column whose name matches `_emb_*` (the convention for embedding slots). The index is keyed on `(object_type, modality, model_tag, tenant_id)` — multi-tenant by default. | Phase | Trigger | Effect | |---|---|---| | **Writing** | First insert of an `_emb_*` field | HNSW graph grows; tombstones accumulate | | **Compacting** | Tombstone ratio > 10% | Tier compaction merges; dead nodes reclaimed | | **Reading** | `SIMILAR TO` query, `/search`, or `HYBRID_SEARCH` | Beam search + filtered refine | | **Spilling** | `max_live` exceeded | Spill to DiskANN warm tier (object-store-backed flat segments) | By default, the index is **lazy** — the first query that touches an `_emb_*` field triggers build. Set `RELATA_ANN_EAGER=true` to build at insert time (higher ingest cost; lower first-query latency). ## HNSW parameters (index-time) These are set at index creation (first insert) and cannot be changed without a rebuild. | Parameter | Default | Range | Effect | |---|---|---|---| | `M` | 128 | 16–256 | Out-degree per node (layer > 0). Higher = better recall, more RAM. | | `M0` | 256 | 32–512 | Out-degree at layer 0 (the data layer). Typically 2× M. | | `ef_construction` | 350 | 100–1000 | Beam width during build. Higher = better recall, slower build. | | `ml` | `1/ln(M)` | (computed) | Level-decay factor for random level assignment. | **Memory cost**: ~`3 KB × vectors` at f32 / 1536-d (OpenAI). ~`1 KB × vectors` at int8 / 384-d (MiniLM). For 1 M vectors at f32/1536-d: ~3 GB. ### Choosing M and ef_construction | Workload | M | ef_construction | Notes | |---|---|---|---| | High recall, low QPS | 256 | 500 | Best recall@10; 2× RAM | | Balanced (default) | 128 | 350 | Good recall, fast search | | Low latency, high QPS | 64 | 200 | Lower recall; ~half RAM | | Binary embeddings | 32 | 100 | For 1-bit codes | ## HNSW parameters (search-time) | Parameter | Default | Range | Effect | |---|---|---|---| | `ef_search` | `max(k, 10)` | k–1000 | Beam width at query. Higher = better recall, slower. | | `limit` (k) | query-supplied | 1–1000 | Top-K returned | | `filter` | none | allowlist | Pre-filter or post-filter (adaptive at 25% selectivity) | ### Tuning `ef_search` ```sql -- Tighten ef_search for fast low-recall queries: SELECT * FROM Vec WHERE SIMILAR TO '[...]' FIELD=_emb_text LIMIT 10 EF_SEARCH=20 -- Loosen for high recall: SELECT * FROM Vec WHERE SIMILAR TO '[...]' FIELD=_emb_text LIMIT 10 EF_SEARCH=200 ``` The default `max(k, 10)` is fine for most workloads. For high-recall applications (face recognition, dedup), set `ef_search = 5 * k`. ## Distance metrics Today, **cosine** is the default and most-optimised metric. | Metric | Use case | Cost vs cosine | |---|---|---| | `Cosine` (default) | Semantic similarity (most embedding models) | 1× | | `L2Squared` | Face recognition, image search | ~1× | | `InnerProduct` (MIPS) | DPR-style retrievers | ~0.9× | ```sql SELECT * FROM Face WHERE SIMILAR TO '[...]' FIELD=_emb_face METRIC=L2 LIMIT 10 ``` ## Quantization | Tier | Bytes per dim | Recall loss | Use when | |---|---|---|---| | f32 / `full` (default) | 4 | 0% | Default (`RELATA_VECTOR_QUANT=full`); small indexes | | FP16 | 2 | <0.5% | 2× memory savings | | int8 | 1 | ±0.4% | 4× memory savings; opt in via `RELATA_VECTOR_QUANT=int8` | | PQ | ~0.1 | 2–5% | Billion-scale cold tier | | Binary hash | 1/8 | 5–15% | Deep-1-bit first-pass filter | The HNSW hot tier defaults to `full` (raw f32 — `RELATA_VECTOR_QUANT=full`). Set `RELATA_VECTOR_QUANT=int8` for a 4× memory reduction at ±0.4% recall. The IVF cold tier uses int8 with optional PQ (32–64× compression). ## Cold tier (DiskANN + IVF) When the index exceeds `RELATA_DISKANN_MAX_RESIDENT`, the cold tier engages: | Component | Purpose | Trigger | |---|---|---| | IVF bucket | Cluster by k-means++ centroids | Always on (cold tier) | | Posting list | Per-centroid vector list | Paged from object store via LRU | | `nprobe` | Number of centroids to search | Default 32 | | DiskANN graph | Object-store-backed HNSW for SSD traversal | Cold tier | ```bash # Bound RAM-resident vectors in the cold tier staging area RELATA_VECTOR_COLD_RESIDENT_MAX=200000 # default 100 000 # Bound total HNSW-resident vectors (warns past this) RELATA_DISKANN_MAX_RESIDENT=1000000 # 0 / unset = unbounded ``` ## Search-time knobs ### `/search` body ```json { "query": "alice", "vector": [0.1, 0.2, ...], "vector_field": "_emb_text", "metric": "cosine", "limit": 10, "ef_search": 100, "filter": {"tenant_id": "org-acme"} } ``` ### SQL `SIMILAR TO` ```sql SELECT * FROM Vec WHERE SIMILAR TO '[0.1, 0.2, ...]' FIELD=_emb_text METRIC=COSINE LIMIT 10 EF_SEARCH=100 ``` ### Pre-filter vs post-filter For selective predicates (e.g. "only vectors where `tenant_id = 'org-acme'`"), Relata chooses automatically based on selectivity: | Selectivity | Strategy | Why | |---|---|---| | > 25% | Post-filter | Most candidates survive; ANN is faster | | <25% | Pre-filter (allowlist) | Few candidates; skip ANN entirely | ## Performance characteristics | Operation | Latency (1 M vectors, int8, 384-d) | |---|---| | Build (parallel bulk insert) | ~3 min on 16 cores | | Point search (k=10) | <5 ms p99 | | Filtered search (1% selectivity) | <8 ms p99 | | Insert (single vector) | <100 µs | | Soft delete (tombstone) | <10 µs | | Compaction (10% tombstones) | ~5 s for 1 M vectors | ## Common pitfalls | Pitfall | Symptom | Fix | |---|---|---| | ef_search too low | Recall@10 <0.9 | Raise `EF_SEARCH=100` or higher | | M too low for high-dimensional embeddings | Recall plateaus | Rebuild with `M=256` | | Filtered search recall cliff | Recall@10 drops sharply at <5% selectivity | Use pre-filter explicitly | | Index RAM exceeds budget | OOM warning at startup | Reduce `max_resident` or shard the type | | Stale tombstone accumulation | Search latency drifts upward | Trigger compaction (`relata compact --type `) | ## See also - [Search and retrieval](/docs/reference/search) - [Performance tuning](/docs/reference/performance) ============================================================================== # Go SDK URL: https://relatadb.dev/docs/sdks/go ============================================================================== # Go SDK `go get github.com/relatadb/sdk-go/v2` — Go 1.25+. The core is **stdlib-only** (`net/http`, `encoding/json`, `crypto/rand`, `context`, `time`); `google.golang.org/grpc` + `github.com/apache/arrow/go/v15` are pulled only for Arrow Flight. Every method takes `ctx context.Context` first and returns `(..., error)`. > See the [SDK overview](/docs/sdks/overview) for the cross-language parity matrix. This page is the Go capability catalog. ## Quickstart ```bash go get github.com/relatadb/sdk-go/v2 ``` ```go package main import ( "context" "fmt" "log" "github.com/relatadb/sdk-go/v2/relata" ) func main() { ctx := context.Background() client := relata.New("http://localhost:9090", &relata.ClientOptions{ BearerToken: "relata-dev", DefaultPurpose: "analytics", }) _, err := client.Query(ctx, "INSERT INTO Person (_pk, name, email) VALUES ('p1', 'Alice', 'a@x.com')") if err != nil { log.Fatal(err) } res, _ := client.Query(ctx, "SELECT * FROM Person LIMIT 5") for _, row := range res.Rows { fmt.Println(row["name"], row["email"]) } } ``` Idiomatic `context.Context` everywhere — cancel long queries via `context.WithTimeout` / `context.WithCancel`. ## The query surface | Path | Method | Returns | |---|---|---| | SQL | `Query(ctx, sql, opts ...QueryOption)` | `*QueryResult` — `WithPurpose`, `WithTimeout`, `WithDialect` | | Parameterized | `QueryWithParams(ctx, sql, params, opts...)` | `*QueryResult` | | Arrow Flight (gRPC zero-copy) | `QueryFlight(ctx, sql, flightEndpoint, purpose, bearer)` | `arrow.Table` | | GraphQL | `GraphQL(ctx, query, variables, operationName)` | `any` — `variables` bound server-side (#3260) | | SPARQL | `Sparql(ctx, query)` | `map[string]any` | | Cypher | any `MATCH`-prefixed string via `Query()` | auto-routed, governed | | GQL (ISO 39075) | `Query(ctx, stmt, relata.WithDialect("gql"))` | header-selected, governed (#3265) | | Fluent builder | `relata.NewQuery(sql).Purpose(p).Where(...).Limit(10).Execute(ctx, client)` | `*QueryResult` | Free constructors: `relata.PathsBetween(from, to, maxHops)`, `relata.MatchFace(img, topK)`, `relata.LookupIdentity(id)`, `relata.HybridSearch(type, query, topK)` — for typed-query IR compilation. ## Typed domain clients 18 typed sub-clients (68 MCP wrappers, at parity with Python/TypeScript). Each takes the parent `*Client`: ```go ingest := relata.NewIngestClient(client) objs := relata.NewObjectClient(client) gov := relata.NewGovernanceClient(client) mcp := relata.NewMcpClient(client) vc := relata.NewVectorClient(client) // + IdentityClient, SearchClient, StreamingClient, AuditClient, // TenantAdminClient, BackupClient, TokenClient, LogClient, // SystemClient, A2AClient, S3Client, Namespace, Memory ``` | Client | Key methods | Cross-ref | |---|---|---| | `GovernanceClient` | rules CRUD, `ImportSigma`, retention/WORM/legal-holds, breakglass, alerts, DSAR | [Detection Rules](/docs/guides/detection-rules) | | `IdentityClient` | `Label`, `RecordUncertainty`, `RegisterLookup`/`ListLookups`/`InvokeLookup`, `EraseSubject` | [Identity](/docs/concepts/identity) | | `ObjectClient` | `Upsert`, `BatchUpsert`, `UpsertTyped`, `Get`, `Delete` | — | | `IngestClient` | `Bulk`, `BulkCSV`, `IngestAuto`, `IngestCDR`, `OTLPTraces/Logs/Metrics`, `IngestIter` (channel) | [Ingestion](/docs/guides/ingestion) | | `VectorClient` | `KNNSearch`, `HybridSearch`, `SimilarTo`, `Embed`/`EmbedBatch` + `EmbedImage/Face/Audio/Video` | [Hybrid Search](/docs/concepts/hybrid-search) | | `SearchClient` | typed `/search`: `Query(ctx, SearchRequest)` | [Search reference](/docs/reference/search) | | `StreamingClient` | `QueryRows` (*RowIterator), `QueryArrowRaw` (*BytesIterator, `io.Reader`), `Watch`/`Alerts` (*SSEIterator) | — | | `AuditClient` | `Count`, `Entries`, `FindByRequestID`, `SignReceipt`, `ExportPDF` → `[]byte` | — | | `TenantAdminClient` | tenant CRUD, quota, sharing, platform usage/license | [Multi-Tenancy](/docs/guides/multi-tenancy) | | `BackupClient` | `Create`, `List`, `Restore`, `RestoreStatus`, `Compact`, `WaitForRestore` | [Backup & Restore](/docs/guides/backup-restore) | | `TokenClient` | `Remember`, `Check`, `Revoke`, `Stats` | — | | `LogClient` | `Append`, `Head`, `LoadLeaves` | — | | `SystemClient` | LLM config/test, jobs/workflows, feeds, notifications, pipelines | — | | `A2AClient` | `SubmitTask`, `GetTask`, checkpoints, `AgentCard` | — | | `McpClient` | `Initialize`, `ListTools`, `CallTool` + **68 typed wrappers** | [MCP Tools](/docs/reference/mcp-tools) | | `S3Client` | `HTTP` (returns configured `*http.Client` with bearer+tenant transport), `BaseURL` | [S3 door](/docs/guides/s3-door) | | `Namespace` | `client.Namespace("Document")` → `Query/Write/Get/DeleteAll/BranchFrom` | [Search reference](/docs/reference/search) | ## Functional options — every per-call override Go-idiomatic variadic options for every knob (no builder boilerplate): ```go hits, _ := client.Search(ctx, "alice", "Person", relata.WithSearchLimit(50), relata.WithSearchFacets("status", "city"), relata.WithHighlight(), relata.WithMatchingStrategy("all"), relata.WithTypoTolerance(map[string]any{"enabled": true, "minWordSize": 4}), relata.WithWeights(0.2, 0.5, 0.3), // graph, bm25, vector ) pr, _ := client.GraphPageRank(ctx, "analytics", "Person", &relata.GraphPageRankOptions{Damping: 0.85, MaxIter: 20}) ``` ## Vectors & embeddings ```go vc := relata.NewVectorClient(client) knn, _ := vc.KNNSearch(ctx, "Document", "embedding", []float64{0.1,...}, 10, &relata.KNNOptions{EFSearch: 200}) hybrid, _ := vc.HybridSearch(ctx, "Document", "graph retrieval", &relata.HybridSearchOptions{K: 10, Rerank: true, Weights: &[3]float64{0.2,0.5,0.3}}) e, _ := vc.Embed(ctx, "Alice Smith", "") // → *EmbedResponse eimg, _ := vc.EmbedImage(ctx, b64, "") // CLIP vc.EmbedFace(ctx, b64, "") // ArcFace vc.EmbedAudio(ctx, b64, "") // CLAP vc.EmbedVideo(ctx, b64, "") // CLIP keyframe ``` ## Graph & intelligence operators All on `*Client` — 10 graph algorithms + 10 AML/financial + 3 maritime: ```go client.GraphPageRank(ctx, "analytics", "Person", &relata.GraphPageRankOptions{...}) client.GraphShortestPath(ctx, "alice-id", "bob-id", &relata.GraphShortestPathOptions{MaxHops: 5}) client.GraphCommunity(ctx, "analytics", "Person", nil) client.SanctionsScreen(ctx, "compliance", "Acme Holdings", &relata.SanctionsScreenOptions{Threshold: 0.85}) client.BeneficialOwnershipChain(ctx, "compliance", "Acme Holdings", 6) client.CryptoTrace(ctx, "compliance", "0xabc...") client.VesselTrack(ctx, "analytics", 123456789, 86400) client.DarkFleetDetect(ctx, "analytics", 48) ``` See [Graph Analytics](/docs/reference/graph-analytics). ## Agent memory — 10 cognitive verbs + recall-quality knobs `Memory` is a standalone client (owns its own `*Client` via `New(...)`): ```go mem, _ := relata.NewMemory("http://localhost:9090", "agent", &relata.MemoryOptions{BearerToken: "relata-dev"}) id, _ := mem.Add(ctx, "Alice prefers dark mode") // retrieval-quality operators (functional options) results, _ := mem.Search(ctx, "ui preferences", relata.WithTopK(10), relata.WithMinConfidence(0.6), relata.WithRecencyHalfLife(259200), relata.WithBudgetTokens(1500), relata.WithCancelThreshold(0.92), ) detail, _ := mem.SearchDetailed(ctx, "ui preferences", /* same opts */) // detail.RecallCostTokens + detail.Cancelled — observe the knobs' effect ``` Full verb set: `Add`, `AddBatch`, `Search`, `SearchDetailed`, `Get`, `Update`, `Forget`, `Associate`, `Episodes`, `Justify`, `Resolve`, `Summarise`. See [Agent memory reference](/docs/reference/agent-memory). ## Authentication & multi-tenant ```go client := relata.New("http://localhost:9090", &relata.ClientOptions{ BearerToken: "relata-dev", DefaultPurpose: "analytics", Tenant: "org-acme", // X-Relata-Tenant-Id on every request ActingAs: "user-42", // X-Acting-As (delegation) DelegatedBy: "admin-1", // X-Delegated-By Timeout: 15 * time.Second, MaxRetries: 3, RetryBackoff: 500 * time.Millisecond, AdminBaseURL: "http://admin.internal:9090", // /admin/* + /platform/* split }) ``` ## Examples ~25 runnable examples in `sdks/go/examples/`. Each is a self-contained `main` package — `go run ./examples/`: ```bash go run ./examples/basic -url http://localhost:9090 -token $RELATA_TOKEN go run ./examples/ingest -url http://localhost:9090 -token $RELATA_TOKEN go run ./examples/memory_quickstart -url http://localhost:9090 -token $RELATA_TOKEN go run ./examples/intelligence -url http://localhost:9090 -token $RELATA_TOKEN go run ./examples/graph_algorithms -url http://localhost:9090 -token $RELATA_TOKEN go run ./examples/streaming -url http://localhost:9090 -token $RELATA_TOKEN go run ./examples/face_search -url http://localhost:9090 -token $RELATA_TOKEN ``` Full set: [`sdks/go/examples/`](https://github.com/relatadb/tree/main/sdks/go/examples). ## Next steps - [Search and retrieval](/docs/reference/search) — typed `/search`, multi-query batch + RRF - [Agent memory reference](/docs/reference/agent-memory) — 10 verbs + recall-quality knobs - [Graph analytics](/docs/reference/graph-analytics) — 10+ algorithms, `gds.*` portability - [Query cookbook](/docs/reference/query-cookbook) - [Full Go SDK source](https://github.com/relatadb/tree/main/sdks/go) ============================================================================== # SDKs URL: https://relatadb.dev/docs/sdks ============================================================================== # SDKs First-class client SDKs for RelataDB. All three speak the same HTTP API and expose the same shape: query, ingest, search, graph, identity, and the cognitive-memory verbs. Pick the one for your stack. ## In this section - [SDK Overview](/docs/sdks/overview) — capability matrix, install, and the shared request/response model - [Python](/docs/sdks/python) — `pip install relata-sdk`; `RelataClient`, framework adapters - [TypeScript](/docs/sdks/typescript) — `npm install @zysec-ai/relata-sdk`; `createClient(...)` - [Go](/docs/sdks/go) — `github.com/relatadb/sdk-go` - [Method Reference](/docs/sdks/methods) — every public method, grouped by domain, with signatures, tunable parameter defaults, *when-to-use* notes, and runnable examples for the flagship surface You don't need an SDK to use RelataDB. Point an existing Postgres, MongoDB, Redis, S3, ClickHouse, or Neo4j client at the matching protocol door. See [Compatibility & Doors](/docs/compatibility). See also: [Quickstart](/docs/quickstart) for the first-5-minutes path and the [HTTP API reference](/docs/reference/api-reference) for the underlying endpoints. ============================================================================== # Admin, Ops & Multi-Tenancy URL: https://relatadb.dev/docs/sdks/methods/admin ============================================================================== # Admin, Ops & Multi-Tenancy Operators run backups, restores, and compaction via `BackupClient`; manage pipelines, feeds, notification rules, jobs, and LLM config via `SystemClient`; manage tenants, tiers, sharing, and platform usage via `TenantAdminClient`; and mint/revoke bearer tokens via `TokenClient`. {/* BEGIN GENERATED — overwritten by scripts/gen_sdk_methods_docs.py */} > **46 methods** across this domain. Signatures, parameters (incl. keyword-only tunable defaults), and the three SDK spellings are ported from the live SDK source; flagship methods carry a hand-written *When to use* + example. ## `BackupClient` ### `restore(snapshot_id, tenant=None, point_in_time_ns=None)` **When to use.** Restore from a snapshot; poll with `wait_for_restore` / `restore_status`. - **Python** — `client.backupclient.restore(snapshot_id, tenant=None, point_in_time_ns=None)` - **TypeScript** — `restore(snapshotId, opts)` - **Go** — `Restore(snapshotID)` **Parameters:** `snapshot_id`, `tenant=None`, `point_in_time_ns=None` **Example** ```python rid = BackupClient.from_client(relata).restore(snapshot_id="snap-9") ``` ### `compact()` compact()` - **Python** — `client.backupclient.compact()` - **TypeScript** — `compact()` - **Go** — `Compact()` ### `create(kind='full', tenant=None)` create(kind='full', tenant=None)` - **Python** — `client.backupclient.create(kind='full', tenant=None)` - **TypeScript** — `create(opts)` - **Go** — `Create()` **Parameters:** `kind='full'`, `tenant=None` ### `list()` list()` - **Python** — `client.backupclient.list()` - **TypeScript** — `list()` - **Go** — `List()` ### `restore_status(restore_id)` restore_status(restore_id)` - **Python** — `client.backupclient.restore_status(restore_id)` - **TypeScript** — `restoreStatus(restoreId)` - **Go** — `RestoreStatus(restoreID)` **Parameters:** `restore_id` ### `wait_for_restore(restore_id, timeout_secs=300.0, poll_interval_secs=2.0)` wait_for_restore(restore_id, timeout_secs=300.0, poll_interval_secs=2.0)` - **Python** — `client.backupclient.wait_for_restore(restore_id, timeout_secs=300.0, poll_interval_secs=2.0)` - **TypeScript** — `waitForRestore(restoreId, opts)` - **Go** — `WaitForRestore(restoreID)` **Parameters:** `restore_id`, `timeout_secs=300.0`, `poll_interval_secs=2.0` ## `RelataClient` ### `cluster_drain(node_id)` cluster_drain(node_id)` - **Python** — `client.relataclient.cluster_drain(node_id)` - **TypeScript** — `clusterDrain(nodeId)` - **Go** — `ClusterDrain(nodeID)` **Parameters:** `node_id` ### `cluster_nodes()` cluster_nodes()` - **Python** — `client.relataclient.cluster_nodes()` - **TypeScript** — `clusterNodes()` - **Go** — `ClusterNodes()` ### `cluster_rebalance()` cluster_rebalance()` - **Python** — `client.relataclient.cluster_rebalance()` - **TypeScript** — `clusterRebalance()` - **Go** — `ClusterRebalance()` ### `cluster_topology()` cluster_topology()` - **Python** — `client.relataclient.cluster_topology()` - **TypeScript** — `clusterTopology()` - **Go** — `ClusterTopology()` ## `SystemClient` ### `create_notification_rule(rule)` create_notification_rule(rule)` - **Python** — `client.systemclient.create_notification_rule(rule)` - **TypeScript** — `createNotificationRule(rule)` - **Go** — `CreateNotificationRule(rule)` **Parameters:** `rule` ### `define_pipeline(definition)` define_pipeline(definition)` - **Python** — `client.systemclient.define_pipeline(definition)` - **TypeScript** — `definePipeline(definition)` - **Go** — `DefinePipeline(definition)` **Parameters:** `definition` ### `delete_notification_rule(rule_id)` delete_notification_rule(rule_id)` - **Python** — `client.systemclient.delete_notification_rule(rule_id)` - **TypeScript** — `deleteNotificationRule(id)` - **Go** — `DeleteNotificationRule(id)` **Parameters:** `rule_id` ### `feed_channels()` feed_channels()` - **Python** — `client.systemclient.feed_channels()` - **TypeScript** — `feedChannels()` - **Go** — `FeedChannels()` ### `feed_health()` feed_health()` - **Python** — `client.systemclient.feed_health()` - **TypeScript** — `feedHealth()` - **Go** — `FeedHealth()` ### `feed_publish(channel, payload)` feed_publish(channel, payload)` - **Python** — `client.systemclient.feed_publish(channel, payload)` - **TypeScript** — `feedPublish(channel, payload)` - **Go** — `FeedPublish(channel, payload)` **Parameters:** `channel`, `payload` ### `get_workflow(name)` get_workflow(name)` - **Python** — `client.systemclient.get_workflow(name)` - **TypeScript** — `getWorkflow(name)` - **Go** — `GetWorkflow(name)` **Parameters:** `name` ### `job_status(name)` job_status(name)` - **Python** — `client.systemclient.job_status(name)` - **TypeScript** — `jobStatus(name)` - **Go** — `JobStatus(name)` **Parameters:** `name` ### `jobs_status()` jobs_status()` - **Python** — `client.systemclient.jobs_status()` - **TypeScript** — `jobsStatus()` - **Go** — `JobsStatus()` ### `list_pipelines()` list_pipelines()` - **Python** — `client.systemclient.list_pipelines()` - **TypeScript** — `listPipelines()` - **Go** — `ListPipelines()` ### `list_workflows()` list_workflows()` - **Python** — `client.systemclient.list_workflows()` - **TypeScript** — `listWorkflows()` - **Go** — `ListWorkflows()` ### `llm_config()` llm_config()` - **Python** — `client.systemclient.llm_config()` - **TypeScript** — `llmConfig()` - **Go** — `LLMConfig()` ### `notification_rules()` notification_rules()` - **Python** — `client.systemclient.notification_rules()` - **TypeScript** — `notificationRules()` - **Go** — `NotificationRules()` ### `register_workflow(name, steps)` register_workflow(name, steps)` - **Python** — `client.systemclient.register_workflow(name, steps)` - **TypeScript** — `registerWorkflow(name, steps)` - **Go** — `RegisterWorkflow(name, steps)` **Parameters:** `name`, `steps` ### `run_workflow(name)` run_workflow(name)` - **Python** — `client.systemclient.run_workflow(name)` - **TypeScript** — `runWorkflow(name)` - **Go** — `RunWorkflow(name)` **Parameters:** `name` ### `test_llm(prompt, model=None)` test_llm(prompt, model=None)` - **Python** — `client.systemclient.test_llm(prompt, model=None)` - **TypeScript** — `testLlm(prompt, opts)` - **Go** — `TestLLM(prompt)` **Parameters:** `prompt`, `model=None` ### `workflow_run(run_id)` workflow_run(run_id)` - **Python** — `client.systemclient.workflow_run(run_id)` - **TypeScript** — `workflowRun(runId)` - **Go** — `WorkflowRun(runID)` **Parameters:** `run_id` ### `workflow_status(name)` workflow_status(name)` - **Python** — `client.systemclient.workflow_status(name)` - **TypeScript** — `workflowStatus(name)` - **Go** — `WorkflowStatus(name)` **Parameters:** `name` ## `TenantAdminClient` ### `create_sharing(tenant_id, partner_tenant, object_type, action='read', expires_ns=None)` **When to use.** Establish a governed cross-tenant data-sharing agreement. - **Python** — `client.tenantadminclient.create_sharing(tenant_id, partner_tenant, object_type, action='read', expires_ns=None)` - **TypeScript** — `createSharing(tenantId, partnerTenant, opts)` - **Go** — `CreateSharing(tenantID, partnerTenant, objectType)` **Parameters:** `tenant_id`, `partner_tenant`, `object_type`, `action='read'`, `expires_ns=None` **Example** ```python TenantAdminClient.from_client(relata).create_sharing("acme", partner_tenant="beta") ``` ### `set_tier(tenant_id, tier)` **When to use.** Move a tenant between tiers (operator surface, cluster-only multi-tenant). - **Python** — `client.tenantadminclient.set_tier(tenant_id, tier)` - **TypeScript** — `setTier(tenantId, tier)` - **Go** — `SetTier(tenantID, tier)` **Parameters:** `tenant_id`, `tier` **Example** ```python TenantAdminClient.from_client(relata).set_tier("acme", tier="enterprise") ``` ### `create(tenant_id, tier='standard', **fields)` create(tenant_id, tier='standard', **fields)` - **Python** — `client.tenantadminclient.create(tenant_id, tier='standard', **fields)` - **TypeScript** — `create(tenantId, opts)` - **Go** — `Create(tenantID)` **Parameters:** `tenant_id`, `tier='standard'`, `**fields` ### `delete(tenant_id)` delete(tenant_id)` - **Python** — `client.tenantadminclient.delete(tenant_id)` - **Go** — `Delete(tenantID)` **Parameters:** `tenant_id` ### `get(tenant_id)` get(tenant_id)` - **Python** — `client.tenantadminclient.get(tenant_id)` - **TypeScript** — `get(tenantId)` - **Go** — `Get(tenantID)` **Parameters:** `tenant_id` ### `list_sharing(tenant_id)` list_sharing(tenant_id)` - **Python** — `client.tenantadminclient.list_sharing(tenant_id)` - **TypeScript** — `listSharing(tenantId)` - **Go** — `ListSharing(tenantID)` **Parameters:** `tenant_id` ### `me()` me()` - **Python** — `client.tenantadminclient.me()` - **TypeScript** — `me()` - **Go** — `Me()` ### `platform_license()` platform_license()` - **Python** — `client.tenantadminclient.platform_license()` - **TypeScript** — `platformLicense()` - **Go** — `PlatformLicense()` ### `platform_usage()` platform_usage()` - **Python** — `client.tenantadminclient.platform_usage()` - **TypeScript** — `platformUsage()` - **Go** — `PlatformUsage()` ### `reactivate(tenant_id)` reactivate(tenant_id)` - **Python** — `client.tenantadminclient.reactivate(tenant_id)` - **TypeScript** — `reactivate(tenantId)` - **Go** — `Reactivate(tenantID)` **Parameters:** `tenant_id` ### `set_quota(tenant_id, quota)` set_quota(tenant_id, quota)` - **Python** — `client.tenantadminclient.set_quota(tenant_id, quota)` - **TypeScript** — `setQuota(tenantId, quota)` - **Go** — `SetQuota(tenantID, quota)` **Parameters:** `tenant_id`, `quota` ### `suspend(tenant_id)` suspend(tenant_id)` - **Python** — `client.tenantadminclient.suspend(tenant_id)` - **TypeScript** — `suspend(tenantId)` - **Go** — `Suspend(tenantID)` **Parameters:** `tenant_id` ### `tenant_usage(tenant_id)` tenant_usage(tenant_id)` - **Python** — `client.tenantadminclient.tenant_usage(tenant_id)` - **TypeScript** — `tenantUsage(tenantId)` - **Go** — `TenantUsage(tenantID)` **Parameters:** `tenant_id` ### `usage()` usage()` - **Python** — `client.tenantadminclient.usage()` - **TypeScript** — `usage()` - **Go** — `Usage()` ## `TokenClient` ### `check(token_id)` check(token_id)` - **Python** — `client.tokenclient.check(token_id)` - **TypeScript** — `check(tokenId)` - **Go** — `Check(tokenID)` **Parameters:** `token_id` ### `remember(token_id, ttl_secs=None)` remember(token_id, ttl_secs=None)` - **Python** — `client.tokenclient.remember(token_id, ttl_secs=None)` - **TypeScript** — `remember(tokenId, opts)` - **Go** — `Remember(tokenID)` **Parameters:** `token_id`, `ttl_secs=None` ### `revoke(token_id)` revoke(token_id)` - **Python** — `client.tokenclient.revoke(token_id)` - **TypeScript** — `revoke(tokenId)` - **Go** — `Revoke(tokenID)` **Parameters:** `token_id` ### `stats()` stats()` - **Python** — `client.tokenclient.stats()` - **TypeScript** — `stats()` - **Go** — `Stats()` {/* END GENERATED */} ============================================================================== # MCP & Agent-to-Agent URL: https://relatadb.dev/docs/sdks/methods/agents ============================================================================== # MCP & Agent-to-Agent `McpClient` exposes Relata as a governed MCP tool source: `list_tools` + `call_tool`, plus 72 typed tool wrappers. `A2AClient` is the agent-to-agent door: discover the agent card, submit tasks, persist checkpoints across runs. {/* BEGIN GENERATED — overwritten by scripts/gen_sdk_methods_docs.py */} > **71 methods** across this domain. Signatures, parameters (incl. keyword-only tunable defaults), and the three SDK spellings are ported from the live SDK source; flagship methods carry a hand-written *When to use* + example. ## `McpClient` ### `call_tool(name, arguments=None)` **When to use.** Invoke any of Relata's governed MCP tools by name with a JSON arg map. - **Python** — `client.mcpclient.call_tool(name, arguments=None)` - **TypeScript** — `callTool(name, args)` - **Go** — `CallTool(name, arguments)` **Parameters:** `name`, `arguments=None` **Example** ```python mcp = McpClient.from_client(relata) print(mcp.call_tool("memory_search", {"query": "alice", "top_k": 3})) ``` ### `add_case_note(case_id, note, author=None)` add_case_note(case_id, note, author=None)` - **Python** — `client.mcpclient.add_case_note(case_id, note, author=None)` - **TypeScript** — `addCaseNote(caseId, note, opts)` - **Go** — `AddCaseNote(caseID, note)` **Parameters:** `case_id`, `note`, `author=None` ### `aggregate_stats(entity_type, agg='COUNT', column='*', purpose='analytics')` aggregate_stats(entity_type, agg='COUNT', column='*', purpose='analytics')` - **Python** — `client.mcpclient.aggregate_stats(entity_type, agg='COUNT', column='*', purpose='analytics')` - **TypeScript** — `aggregateStats(entityType, opts)` - **Go** — `AggregateStats(entityType)` **Parameters:** `entity_type`, `agg='COUNT'`, `column='*'`, `purpose='analytics'` ### `associate(from_id, to_id, relation='related_to', purpose=None)` associate(from_id, to_id, relation='related_to', purpose=None)` - **Python** — `client.mcpclient.associate(from_id, to_id, relation='related_to', purpose=None)` - **TypeScript** — `associate(fromId, toId, opts)` - **Go** — `Associate(fromID, toID)` **Parameters:** `from_id`, `to_id`, `relation='related_to'`, `purpose=None` ### `beneficial_ownership(party, max_depth=6, purpose='analytics')` beneficial_ownership(party, max_depth=6, purpose='analytics')` - **Python** — `client.mcpclient.beneficial_ownership(party, max_depth=6, purpose='analytics')` - **TypeScript** — `beneficialOwnership(party, opts)` - **Go** — `BeneficialOwnership(party)` **Parameters:** `party`, `max_depth=6`, `purpose='analytics'` ### `consolidate(memory_id, content, confidence=1.0, purpose=None)` consolidate(memory_id, content, confidence=1.0, purpose=None)` - **Python** — `client.mcpclient.consolidate(memory_id, content, confidence=1.0, purpose=None)` - **TypeScript** — `consolidate(memoryId, content, opts)` - **Go** — `Consolidate(memoryID, content)` **Parameters:** `memory_id`, `content`, `confidence=1.0`, `purpose=None` ### `create_rule(name, condition_sql, severity=None, description=None, purpose='security')` create_rule(name, condition_sql, severity=None, description=None, purpose='security')` - **Python** — `client.mcpclient.create_rule(name, condition_sql, severity=None, description=None, purpose='security')` - **TypeScript** — `createRule(name, conditionSql, opts)` - **Go** — `CreateRule(name, conditionSQL)` **Parameters:** `name`, `condition_sql`, `severity=None`, `description=None`, `purpose='security'` ### `detect_communities(entity_type, algo='louvain', purpose='analytics')` detect_communities(entity_type, algo='louvain', purpose='analytics')` - **Python** — `client.mcpclient.detect_communities(entity_type, algo='louvain', purpose='analytics')` - **TypeScript** — `detectCommunities(entityType, opts)` - **Go** — `DetectCommunities(entityType)` **Parameters:** `entity_type`, `algo='louvain'`, `purpose='analytics'` ### `episodes_in(session_id, limit=20, purpose=None)` episodes_in(session_id, limit=20, purpose=None)` - **Python** — `client.mcpclient.episodes_in(session_id, limit=20, purpose=None)` - **TypeScript** — `episodesIn(sessionId, opts)` - **Go** — `EpisodesIn(sessionID)` **Parameters:** `session_id`, `limit=20`, `purpose=None` ### `erase_subject(subject, reason='gdpr-art17-request')` erase_subject(subject, reason='gdpr-art17-request')` - **Python** — `client.mcpclient.erase_subject(subject, reason='gdpr-art17-request')` - **TypeScript** — `eraseSubject(subject, opts)` - **Go** — `EraseSubject(subject)` **Parameters:** `subject`, `reason='gdpr-art17-request'` ### `explain_policy(sql, purpose)` explain_policy(sql, purpose)` - **Python** — `client.mcpclient.explain_policy(sql, purpose)` - **TypeScript** — `explainPolicy(sql, purpose)` - **Go** — `ExplainPolicy(sql, purpose)` **Parameters:** `sql`, `purpose` ### `face_match(probe_id, threshold=0.8, top_k=10, purpose='security_incident')` face_match(probe_id, threshold=0.8, top_k=10, purpose='security_incident')` - **Python** — `client.mcpclient.face_match(probe_id, threshold=0.8, top_k=10, purpose='security_incident')` - **TypeScript** — `faceMatch(probeId, opts)` - **Go** — `FaceMatch(probeID)` **Parameters:** `probe_id`, `threshold=0.8`, `top_k=10`, `purpose='security_incident'` ### `find_connections(entity, purpose, limit=50)` find_connections(entity, purpose, limit=50)` - **Python** — `client.mcpclient.find_connections(entity, purpose, limit=50)` - **TypeScript** — `findConnections(entity, purpose, opts)` - **Go** — `FindConnections(entity, purpose)` **Parameters:** `entity`, `purpose`, `limit=50` ### `find_in_social_corpus(object_type, text_query=None, user=None, top_k=20)` find_in_social_corpus(object_type, text_query=None, user=None, top_k=20)` - **Python** — `client.mcpclient.find_in_social_corpus(object_type, text_query=None, user=None, top_k=20)` - **TypeScript** — `findInSocialCorpus(query, opts)` - **Go** — `FindInSocialCorpus(query)` **Parameters:** `object_type`, `text_query=None`, `user=None`, `top_k=20` ### `find_scc(entity_type, purpose='analytics')` find_scc(entity_type, purpose='analytics')` - **Python** — `client.mcpclient.find_scc(entity_type, purpose='analytics')` - **TypeScript** — `findScc(entityType, opts)` - **Go** — `FindScc(entityType, purpose)` **Parameters:** `entity_type`, `purpose='analytics'` ### `find_threats(entity_type, purpose='security_incident')` find_threats(entity_type, purpose='security_incident')` - **Python** — `client.mcpclient.find_threats(entity_type, purpose='security_incident')` - **TypeScript** — `findThreats(entityType, opts)` - **Go** — `FindThreats(entityType, purpose)` **Parameters:** `entity_type`, `purpose='security_incident'` ### `forget(memory_id, retain_days=0, purpose=None)` forget(memory_id, retain_days=0, purpose=None)` - **Python** — `client.mcpclient.forget(memory_id, retain_days=0, purpose=None)` - **TypeScript** — `forget(memoryId, opts)` - **Go** — `Forget(memoryID)` **Parameters:** `memory_id`, `retain_days=0`, `purpose=None` ### `geofence(lat, lon, radius_m=1000, target_type='MovementEvent', purpose='analytics')` geofence(lat, lon, radius_m=1000, target_type='MovementEvent', purpose='analytics')` - **Python** — `client.mcpclient.geofence(lat, lon, radius_m=1000, target_type='MovementEvent', purpose='analytics')` - **TypeScript** — `geofence(lat, lon, opts)` - **Go** — `Geofence(lat, lon)` **Parameters:** `lat`, `lon`, `radius_m=1000`, `target_type='MovementEvent'`, `purpose='analytics'` ### `get_audit_trail(purpose=None, principal_filter=None, limit=100)` get_audit_trail(purpose=None, principal_filter=None, limit=100)` - **Python** — `client.mcpclient.get_audit_trail(purpose=None, principal_filter=None, limit=100)` - **TypeScript** — `getAuditTrail(opts)` - **Go** — `GetAuditTrail()` **Parameters:** `purpose=None`, `principal_filter=None`, `limit=100` ### `get_case_summary(case_id, purpose)` get_case_summary(case_id, purpose)` - **Python** — `client.mcpclient.get_case_summary(case_id, purpose)` - **TypeScript** — `getCaseSummary(caseId, purpose)` - **Go** — `GetCaseSummary(caseID, purpose)` **Parameters:** `case_id`, `purpose` ### `get_domain_summary(domain)` get_domain_summary(domain)` - **Python** — `client.mcpclient.get_domain_summary(domain)` - **TypeScript** — `getDomainSummary(opts)` - **Go** — `GetDomainSummary(domain)` **Parameters:** `domain` ### `get_entities(object_type, filters=None, limit=50)` get_entities(object_type, filters=None, limit=50)` - **Python** — `client.mcpclient.get_entities(object_type, filters=None, limit=50)` - **TypeScript** — `getEntities(objectType, opts)` - **Go** — `GetEntities(objectType)` **Parameters:** `object_type`, `filters=None`, `limit=50` ### `get_entity_profile(entity_id, purpose)` get_entity_profile(entity_id, purpose)` - **Python** — `client.mcpclient.get_entity_profile(entity_id, purpose)` - **TypeScript** — `getEntityProfile(entityId, purpose)` - **Go** — `GetEntityProfile(entityID, purpose)` **Parameters:** `entity_id`, `purpose` ### `get_relationships(subject=None, purpose, predicate=None, object=None, source=None, limit=50)` get_relationships(subject=None, purpose, predicate=None, object=None, source=None, limit=50)` - **Python** — `client.mcpclient.get_relationships(subject=None, purpose, predicate=None, object=None, source=None, limit=50)` - **TypeScript** — `getRelationships(entityId, purpose, opts)` - **Go** — `GetRelationships(entityID, purpose)` **Parameters:** `subject=None`, `purpose`, `predicate=None`, `object=None`, `source=None`, `limit=50` ### `get_timeline(entity_id, purpose, since_ns=None, until_ns=None)` get_timeline(entity_id, purpose, since_ns=None, until_ns=None)` - **Python** — `client.mcpclient.get_timeline(entity_id, purpose, since_ns=None, until_ns=None)` - **TypeScript** — `getTimeline(entityId, purpose, opts)` - **Go** — `GetTimeline(entityID, purpose)` **Parameters:** `entity_id`, `purpose`, `since_ns=None`, `until_ns=None` ### `hub_authority(entity_type, max_iter=20, purpose='analytics')` hub_authority(entity_type, max_iter=20, purpose='analytics')` - **Python** — `client.mcpclient.hub_authority(entity_type, max_iter=20, purpose='analytics')` - **TypeScript** — `hubAuthority(entityType, opts)` - **Go** — `HubAuthority(entityType)` **Parameters:** `entity_type`, `max_iter=20`, `purpose='analytics'` ### `hybrid_search(entity_type, query, top_k=10, purpose=None)` hybrid_search(entity_type, query, top_k=10, purpose=None)` - **Python** — `client.mcpclient.hybrid_search(entity_type, query, top_k=10, purpose=None)` - **TypeScript** — `hybridSearch(entityType, query, opts)` - **Go** — `HybridSearch(entityType, query)` **Parameters:** `entity_type`, `query`, `top_k=10`, `purpose=None` ### `import_sigma(sigma_yaml, purpose='security')` import_sigma(sigma_yaml, purpose='security')` - **Python** — `client.mcpclient.import_sigma(sigma_yaml, purpose='security')` - **TypeScript** — `importSigma(sigmaYaml, opts)` - **Go** — `ImportSigma(sigmaYAML, purpose)` **Parameters:** `sigma_yaml`, `purpose='security'` ### `ingest_document(chunks_jsonl, manifest_json, purpose)` ingest_document(chunks_jsonl, manifest_json, purpose)` - **Python** — `client.mcpclient.ingest_document(chunks_jsonl, manifest_json, purpose)` - **TypeScript** — `ingestDocument(chunksJsonl, manifestJson, purpose)` - **Go** — `IngestDocument(chunksJSONL, manifestJSON, purpose)` **Parameters:** `chunks_jsonl`, `manifest_json`, `purpose` ### `ingest_media(object_type, modality='image', codec=None, bytes_b64=None, text=None, tenant_id=None, partition_key=None)` ingest_media(object_type, modality='image', codec=None, bytes_b64=None, text=None, tenant_id=None, partition_key=None)` - **Python** — `client.mcpclient.ingest_media(object_type, modality='image', codec=None, bytes_b64=None, text=None, tenant_id=None, partition_key=None)` - **TypeScript** — `ingestMedia(objectType, opts)` - **Go** — `IngestMedia(objectType)` **Parameters:** `object_type`, `modality='image'`, `codec=None`, `bytes_b64=None`, `text=None`, `tenant_id=None`, `partition_key=None` ### `initialize(client_id='relata-python-sdk', version='1.0')` initialize(client_id='relata-python-sdk', version='1.0')` - **Python** — `client.mcpclient.initialize(client_id='relata-python-sdk', version='1.0')` - **TypeScript** — `initialize(opts)` - **Go** — `Initialize()` **Parameters:** `client_id='relata-python-sdk'`, `version='1.0'` ### `investigate_entity(entity_type, entity_id, purpose='security_incident')` investigate_entity(entity_type, entity_id, purpose='security_incident')` - **Python** — `client.mcpclient.investigate_entity(entity_type, entity_id, purpose='security_incident')` - **TypeScript** — `investigateEntity(entityType, entityId, opts)` - **Go** — `InvestigateEntity(entityType, entityID, purpose)` **Parameters:** `entity_type`, `entity_id`, `purpose='security_incident'` ### `job_status()` job_status()` - **Python** — `client.mcpclient.job_status()` - **TypeScript** — `jobStatus()` - **Go** — `JobStatus()` ### `justify(memory_id, purpose=None)` justify(memory_id, purpose=None)` - **Python** — `client.mcpclient.justify(memory_id, purpose=None)` - **TypeScript** — `justify(memoryId, opts)` - **Go** — `Justify(memoryID, purpose)` **Parameters:** `memory_id`, `purpose=None` ### `list_entity_types()` list_entity_types()` - **Python** — `client.mcpclient.list_entity_types()` - **TypeScript** — `listEntityTypes()` - **Go** — `ListEntityTypes()` ### `list_jobs()` list_jobs()` - **Python** — `client.mcpclient.list_jobs()` - **TypeScript** — `listJobs()` - **Go** — `ListJobs()` ### `list_link_types()` list_link_types()` - **Python** — `client.mcpclient.list_link_types()` - **TypeScript** — `listLinkTypes()` - **Go** — `ListLinkTypes()` ### `list_rules()` list_rules()` - **Python** — `client.mcpclient.list_rules()` - **TypeScript** — `listRules()` - **Go** — `ListRules()` ### `list_tools()` list_tools()` - **Python** — `client.mcpclient.list_tools()` - **TypeScript** — `listTools()` - **Go** — `ListTools()` ### `list_workflows()` list_workflows()` - **Python** — `client.mcpclient.list_workflows()` - **TypeScript** — `listWorkflows()` - **Go** — `ListWorkflows()` ### `lookup_identity(value, purpose='analytics')` lookup_identity(value, purpose='analytics')` - **Python** — `client.mcpclient.lookup_identity(value, purpose='analytics')` - **TypeScript** — `lookupIdentity(value, opts)` - **Go** — `LookupIdentity(value)` **Parameters:** `value`, `purpose='analytics'` ### `metrics()` metrics()` - **Python** — `client.mcpclient.metrics()` - **TypeScript** — `metrics()` - **Go** — `Metrics()` ### `nl_query(query, purpose=None, interpret=False)` nl_query(query, purpose=None, interpret=False)` - **Python** — `client.mcpclient.nl_query(query, purpose=None, interpret=False)` - **TypeScript** — `nlQuery(query, opts)` - **Go** — `NlQuery(query)` **Parameters:** `query`, `purpose=None`, `interpret=False` ### `paths_between(from_id, to_id, max_hops=4, purpose=None)` paths_between(from_id, to_id, max_hops=4, purpose=None)` - **Python** — `client.mcpclient.paths_between(from_id, to_id, max_hops=4, purpose=None)` - **TypeScript** — `pathsBetween(fromId, toId, opts)` - **Go** — `PathsBetween(fromID, toID)` **Parameters:** `from_id`, `to_id`, `max_hops=4`, `purpose=None` ### `predict_links(entity_type, from_id=None, to_id=None, method='common_neighbors', purpose='analytics')` predict_links(entity_type, from_id=None, to_id=None, method='common_neighbors', purpose='analytics')` - **Python** — `client.mcpclient.predict_links(entity_type, from_id=None, to_id=None, method='common_neighbors', purpose='analytics')` - **TypeScript** — `predictLinks(entityType, opts)` - **Go** — `PredictLinks(entityType)` **Parameters:** `entity_type`, `from_id=None`, `to_id=None`, `method='common_neighbors'`, `purpose='analytics'` ### `query_knowledge(sql, purpose)` query_knowledge(sql, purpose)` - **Python** — `client.mcpclient.query_knowledge(sql, purpose)` - **TypeScript** — `queryKnowledge(sql, purpose)` - **Go** — `QueryKnowledge(sql, purpose)` **Parameters:** `sql`, `purpose` ### `rag_store_answer(question, answer, source_ids=None, purpose)` rag_store_answer(question, answer, source_ids=None, purpose)` - **Python** — `client.mcpclient.rag_store_answer(question, answer, source_ids=None, purpose)` - **TypeScript** — `ragStoreAnswer(question, answer, purpose, opts)` - **Go** — `RagStoreAnswer(question, answer, purpose)` **Parameters:** `question`, `answer`, `source_ids=None`, `purpose` ### `rag_store_elements(elements, purpose)` rag_store_elements(elements, purpose)` - **Python** — `client.mcpclient.rag_store_elements(elements, purpose)` - **TypeScript** — `ragStoreElements(elements, purpose)` - **Go** — `RagStoreElements(elements, sourceFilename, purpose)` **Parameters:** `elements`, `purpose` ### `rank_key_nodes(entity_type, metric='pagerank', damping=0.85, max_iter=20, purpose='analytics')` rank_key_nodes(entity_type, metric='pagerank', damping=0.85, max_iter=20, purpose='analytics')` - **Python** — `client.mcpclient.rank_key_nodes(entity_type, metric='pagerank', damping=0.85, max_iter=20, purpose='analytics')` - **TypeScript** — `rankKeyNodes(entityType, opts)` - **Go** — `RankKeyNodes(entityType)` **Parameters:** `entity_type`, `metric='pagerank'`, `damping=0.85`, `max_iter=20`, `purpose='analytics'` ### `recall(query, purpose, top_k=5, min_confidence=None, recency_half_life_secs=None, budget_tokens=None, stability_days=None, cancel_threshold=None)` recall(query, purpose, top_k=5, min_confidence=None, recency_half_life_secs=None, budget_tokens=None, stability_days=None, cancel_threshold=None)` - **Python** — `client.mcpclient.recall(query, purpose, top_k=5, min_confidence=None, recency_half_life_secs=None, budget_tokens=None, stability_days=None, cancel_threshold=None)` - **TypeScript** — `recall(query, purpose, opts)` - **Go** — `Recall(query, purpose)` **Parameters:** `query`, `purpose`, `top_k=5`, `min_confidence=None`, `recency_half_life_secs=None`, `budget_tokens=None`, `stability_days=None`, `cancel_threshold=None` ### `recall_procedure(agent_id, name=None, all_versions=False, limit=20, purpose=None)` recall_procedure(agent_id, name=None, all_versions=False, limit=20, purpose=None)` - **Python** — `client.mcpclient.recall_procedure(agent_id, name=None, all_versions=False, limit=20, purpose=None)` - **TypeScript** — `recallProcedure(agentId, opts)` - **Go** — `RecallProcedure(agentID)` **Parameters:** `agent_id`, `name=None`, `all_versions=False`, `limit=20`, `purpose=None` ### `recognize(memory_id, purpose=None)` recognize(memory_id, purpose=None)` - **Python** — `client.mcpclient.recognize(memory_id, purpose=None)` - **TypeScript** — `recognize(memoryId, opts)` - **Go** — `Recognize(memoryID, purpose)` **Parameters:** `memory_id`, `purpose=None` ### `reconstruct_wire(account, tolerance_pct=5.0, purpose='analytics')` reconstruct_wire(account, tolerance_pct=5.0, purpose='analytics')` - **Python** — `client.mcpclient.reconstruct_wire(account, tolerance_pct=5.0, purpose='analytics')` - **TypeScript** — `reconstructWire(account, opts)` - **Go** — `ReconstructWire(account)` **Parameters:** `account`, `tolerance_pct=5.0`, `purpose='analytics'` ### `remember(content, purpose, confidence=1.0, memory_class='semantic')` remember(content, purpose, confidence=1.0, memory_class='semantic')` - **Python** — `client.mcpclient.remember(content, purpose, confidence=1.0, memory_class='semantic')` - **TypeScript** — `remember(content, purpose, opts)` - **Go** — `Remember(content, purpose)` **Parameters:** `content`, `purpose`, `confidence=1.0`, `memory_class='semantic'` ### `remember_batch(items, purpose=None)` remember_batch(items, purpose=None)` - **Python** — `client.mcpclient.remember_batch(items, purpose=None)` - **TypeScript** — `rememberBatch(items, opts)` - **Go** — `RememberBatch(items)` **Parameters:** `items`, `purpose=None` ### `remember_procedure(agent_id, name, instruction_text, purpose=None)` remember_procedure(agent_id, name, instruction_text, purpose=None)` - **Python** — `client.mcpclient.remember_procedure(agent_id, name, instruction_text, purpose=None)` - **TypeScript** — `rememberProcedure(agentId, name, instructionText, opts)` - **Go** — `RememberProcedure(agentID, name, instructionText, purpose)` **Parameters:** `agent_id`, `name`, `instruction_text`, `purpose=None` ### `resolve(memory_id, purpose=None)` resolve(memory_id, purpose=None)` - **Python** — `client.mcpclient.resolve(memory_id, purpose=None)` - **TypeScript** — `resolve(memoryId, opts)` - **Go** — `Resolve(memoryID, purpose)` **Parameters:** `memory_id`, `purpose=None` ### `resolve_entity_identity(identity, purpose='analytics')` resolve_entity_identity(identity, purpose='analytics')` - **Python** — `client.mcpclient.resolve_entity_identity(identity, purpose='analytics')` - **TypeScript** — `resolveEntityIdentity(identity, opts)` - **Go** — `ResolveEntityIdentity(identity, purpose)` **Parameters:** `identity`, `purpose='analytics'` ### `run_workflow(name)` run_workflow(name)` - **Python** — `client.mcpclient.run_workflow(name)` - **TypeScript** — `runWorkflow(name)` - **Go** — `RunWorkflow(name)` **Parameters:** `name` ### `schedule_job(name)` schedule_job(name)` - **Python** — `client.mcpclient.schedule_job(name)` - **TypeScript** — `scheduleJob(name)` - **Go** — `ScheduleJob(name)` **Parameters:** `name` ### `screen_sanctions(name, threshold=None, purpose='compliance_review')` screen_sanctions(name, threshold=None, purpose='compliance_review')` - **Python** — `client.mcpclient.screen_sanctions(name, threshold=None, purpose='compliance_review')` - **TypeScript** — `screenSanctions(name, opts)` - **Go** — `ScreenSanctions(name)` **Parameters:** `name`, `threshold=None`, `purpose='compliance_review'` ### `search_entities(query, entity_types=None)` search_entities(query, entity_types=None)` - **Python** — `client.mcpclient.search_entities(query, entity_types=None)` - **TypeScript** — `searchEntities(query, opts)` - **Go** — `SearchEntities(query)` **Parameters:** `query`, `entity_types=None` ### `search_knowledge(query, purpose, top_k=10)` search_knowledge(query, purpose, top_k=10)` - **Python** — `client.mcpclient.search_knowledge(query, purpose, top_k=10)` - **TypeScript** — `searchKnowledge(query, purpose, opts)` - **Go** — `SearchKnowledge(query, purpose, topK)` **Parameters:** `query`, `purpose`, `top_k=10` ### `search_video_frames(query_id, media_type='VideoFrame', top_k=20, purpose='security_incident')` search_video_frames(query_id, media_type='VideoFrame', top_k=20, purpose='security_incident')` - **Python** — `client.mcpclient.search_video_frames(query_id, media_type='VideoFrame', top_k=20, purpose='security_incident')` - **TypeScript** — `searchVideoFrames(queryId, opts)` - **Go** — `SearchVideoFrames(queryID)` **Parameters:** `query_id`, `media_type='VideoFrame'`, `top_k=20`, `purpose='security_incident'` ### `server_health()` server_health()` - **Python** — `client.mcpclient.server_health()` - **TypeScript** — `serverHealth()` - **Go** — `ServerHealth()` ### `similar_multimodal(entity_type, entity_id, top_k=10, modality='text', purpose=None)` similar_multimodal(entity_type, entity_id, top_k=10, modality='text', purpose=None)` - **Python** — `client.mcpclient.similar_multimodal(entity_type, entity_id, top_k=10, modality='text', purpose=None)` - **TypeScript** — `similarMultimodal(entityType, entityId, opts)` - **Go** — `SimilarMultimodal(entityType, entityID)` **Parameters:** `entity_type`, `entity_id`, `top_k=10`, `modality='text'`, `purpose=None` ### `suggest_extensions(prefix)` suggest_extensions(prefix)` - **Python** — `client.mcpclient.suggest_extensions()` - **TypeScript** — `suggestExtensions(prefix)` - **Go** — `SuggestExtensions(prefix)` **Parameters:** `prefix` ### `summarise(ids=None, session_id=None, scope=None, contents=None, purpose=None)` summarise(ids=None, session_id=None, scope=None, contents=None, purpose=None)` - **Python** — `client.mcpclient.summarise(ids=None, session_id=None, scope=None, contents=None, purpose=None)` - **TypeScript** — `summarise(opts)` - **Go** — `Summarise()` **Parameters:** `ids=None`, `session_id=None`, `scope=None`, `contents=None`, `purpose=None` ### `trace_crypto(address, max_hops=5, min_amount=0, purpose='analytics')` trace_crypto(address, max_hops=5, min_amount=0, purpose='analytics')` - **Python** — `client.mcpclient.trace_crypto(address, max_hops=5, min_amount=0, purpose='analytics')` - **TypeScript** — `traceCrypto(address, opts)` - **Go** — `TraceCrypto(address)` **Parameters:** `address`, `max_hops=5`, `min_amount=0`, `purpose='analytics'` ### `trace_hawala(seed, max_hops=5, purpose='analytics')` trace_hawala(seed, max_hops=5, purpose='analytics')` - **Python** — `client.mcpclient.trace_hawala(seed, max_hops=5, purpose='analytics')` - **TypeScript** — `traceHawala(seed, opts)` - **Go** — `TraceHawala(seed)` **Parameters:** `seed`, `max_hops=5`, `purpose='analytics'` ### `workflow_status(run_id)` workflow_status(run_id)` - **Python** — `client.mcpclient.workflow_status(run_id)` - **TypeScript** — `workflowStatus(runId)` - **Go** — `WorkflowStatus(runID)` **Parameters:** `run_id` {/* END GENERATED */} ============================================================================== # Audit & Reporting URL: https://relatadb.dev/docs/sdks/methods/audit ============================================================================== # Audit & Reporting `AuditClient` reads the governed audit chain: `count` returns chain-validity + entry count, `find_by_request_id` traces a single request end-to-end, `export_pdf` produces a signed, exportable report. {/* BEGIN GENERATED — overwritten by scripts/gen_sdk_methods_docs.py */} > **10 methods** across this domain. Signatures, parameters (incl. keyword-only tunable defaults), and the three SDK spellings are ported from the live SDK source; flagship methods carry a hand-written *When to use* + example. ## `AuditClient` ### `count()` **When to use.** Quick audit-chain health: validity + entry count. - **Python** — `client.auditclient.count()` - **TypeScript** — `count()` - **Go** — `Count()` **Example** ```python from relata import AuditClient print(AuditClient.from_client(relata).count()) ``` ### `export_pdf(case_id, template='default')` **When to use.** Produce a signed PDF report of audit entries for a window. - **Python** — `client.auditclient.export_pdf(case_id, template='default')` - **TypeScript** — `exportPdf(caseId, opts)` - **Go** — `ExportPDF(caseID, template)` **Parameters:** `case_id`, `template='default'` **Example** ```python AuditClient.from_client(relata).export_pdf(since="2025-01-01") ``` ### `entries(principal=None, purpose=None, decision=None, request_id=None, since_ns=None, until_ns=None, limit=100, cursor=None)` entries(principal=None, purpose=None, decision=None, request_id=None, since_ns=None, until_ns=None, limit=100, cursor=None)` - **Python** — `client.auditclient.entries(principal=None, purpose=None, decision=None, request_id=None, since_ns=None, until_ns=None, limit=100, cursor=None)` - **TypeScript** — `entries(opts)` - **Go** — `Entries()` **Parameters:** `principal=None`, `purpose=None`, `decision=None`, `request_id=None`, `since_ns=None`, `until_ns=None`, `limit=100`, `cursor=None` ### `find_by_request_id(request_id)` find_by_request_id(request_id)` - **Python** — `client.auditclient.find_by_request_id(request_id)` - **TypeScript** — `findByRequestId(requestId)` - **Go** — `FindByRequestID(requestID)` **Parameters:** `request_id` ### `sign_receipt(payload)` sign_receipt(payload)` - **Python** — `client.auditclient.sign_receipt(payload)` - **TypeScript** — `signReceipt(payload)` - **Go** — `SignReceipt(payload)` **Parameters:** `payload` ## `RelataClient` ### `health()` health()` - **Python** — `client.relataclient.health()` - **TypeScript** — `health()` - **Go** — `Health()` ### `ready()` ready()` - **Python** — `client.relataclient.ready()` - **TypeScript** — `ready()` - **Go** — `Ready()` ### `stats()` stats()` - **Python** — `client.relataclient.stats()` - **TypeScript** — `stats()` - **Go** — `Stats()` ### `status()` status()` - **Python** — `client.relataclient.status()` - **TypeScript** — `status()` - **Go** — `Status()` ### `version()` version()` - **Python** — `client.relataclient.version()` - **TypeScript** — `version()` - **Go** — `Version()` {/* END GENERATED */} ============================================================================== # Governance & Compliance URL: https://relatadb.dev/docs/sdks/methods/governance ============================================================================== # Governance & Compliance Governance is Cedar-inspired ABAC, deny-wins, with bitmap row filtering and cell masking. Break-glass is the audited emergency override; legal hold is row-scoped immutability; WORM is retention-floor storage; DSAR is the GDPR subject-access request. These are the product differentiators — policy is in the query path, not bolted on. {/* BEGIN GENERATED — overwritten by scripts/gen_sdk_methods_docs.py */} > **21 methods** across this domain. Signatures, parameters (incl. keyword-only tunable defaults), and the three SDK spellings are ported from the live SDK source; flagship methods carry a hand-written *When to use* + example. ## `GovernanceClient` ### `import_sigma(sigma_yaml, purpose=None)` **When to use.** Import a Sigma detection rule (YAML) into the governed rule engine. - **Python** — `client.governanceclient.import_sigma(sigma_yaml, purpose=None)` - **TypeScript** — `importSigma(sigmaYaml, opts)` - **Go** — `ImportSigma(sigmaYAML)` **Parameters:** `sigma_yaml`, `purpose=None` **Example** ```python GovernanceClient.from_client(relata).import_sigma(sigma_yaml=open("rule.yml").read()) ``` ### `place_legal_hold(case_id, object_type, field=None, value=None)` **When to use.** Freeze specific rows (row-scoped `field`/`value`) against modification/deletion. - **Python** — `client.governanceclient.place_legal_hold(case_id, object_type, field=None, value=None)` - **TypeScript** — `placeLegalHold(caseId, objectType, opts)` - **Go** — `PlaceLegalHold(caseID, objectType)` **Parameters:** `case_id`, `object_type`, `field=None`, `value=None` **Example** ```python GovernanceClient.from_client(relata).place_legal_hold("Person", field="_pk", value="p-42", reason="litigation-hold-7") ``` ### `request_breakglass(source_id, purpose=None, justification=None)` **When to use.** Emergency override of a deny — audited, time-boxed, requires approval (ADR-261). - **Python** — `client.governanceclient.request_breakglass(source_id, purpose=None, justification=None)` - **TypeScript** — `requestBreakglass(sourceId, opts)` - **Go** — `RequestBreakglass(sourceID)` **Parameters:** `source_id`, `purpose=None`, `justification=None` **Example** ```python bg = GovernanceClient.from_client(relata).request_breakglass( source_id="incident-42", purpose="forensics", justification="SEV1: locked-out owner") ``` ### `set_worm_policy(object_type, retention_secs)` **When to use.** Set a retention floor (WORM) on a type so rows cannot be deleted before `retention_secs`. - **Python** — `client.governanceclient.set_worm_policy(object_type, retention_secs)` - **TypeScript** — `setWormPolicy(objectType, retentionSecs)` - **Go** — `SetWormPolicy(objectType, retentionSecs)` **Parameters:** `object_type`, `retention_secs` **Example** ```python GovernanceClient.from_client(relata).set_worm_policy("AuditEvent", retention_secs=365*86400) ``` ### `submit_dsar(subject_identity, reason, scope=None)` **When to use.** File a GDPR data-subject-access request; returns a governed export job. - **Python** — `client.governanceclient.submit_dsar(subject_identity, reason, scope=None)` - **TypeScript** — `submitDsar(subjectIdentity, reason, opts)` - **Go** — `SubmitDSAR(subjectIdentity, reason)` **Parameters:** `subject_identity`, `reason`, `scope=None` **Example** ```python GovernanceClient.from_client(relata).submit_dsar(subject="alice@x.io", scope="all") ``` ### `add_rule_exception(rule_id, exception)` add_rule_exception(rule_id, exception)` - **Python** — `client.governanceclient.add_rule_exception(rule_id, exception)` - **TypeScript** — `addRuleException(ruleId, exception)` - **Go** — `AddRuleException(ruleID, exception)` **Parameters:** `rule_id`, `exception` ### `approve_breakglass(request_id, approver_note=None)` approve_breakglass(request_id, approver_note=None)` - **Python** — `client.governanceclient.approve_breakglass(request_id, approver_note=None)` - **TypeScript** — `approveBreakglass(requestId, opts)` - **Go** — `ApproveBreakglass(requestID)` **Parameters:** `request_id`, `approver_note=None` ### `breakglass_status(request_id)` breakglass_status(request_id)` - **Python** — `client.governanceclient.breakglass_status(request_id)` - **TypeScript** — `breakglassStatus(requestId)` - **Go** — `BreakglassStatus(requestID)` **Parameters:** `request_id` ### `create_rule(rule, purpose=None)` create_rule(rule, purpose=None)` - **Python** — `client.governanceclient.create_rule(rule, purpose=None)` - **TypeScript** — `createRule(rule, opts)` - **Go** — `CreateRule(rule)` **Parameters:** `rule`, `purpose=None` ### `disable_rule(rule_id)` disable_rule(rule_id)` - **Python** — `client.governanceclient.disable_rule(rule_id)` - **TypeScript** — `disableRule(ruleId)` - **Go** — `DisableRule(ruleID)` **Parameters:** `rule_id` ### `get_rule_tuning(rule_id)` get_rule_tuning(rule_id)` - **Python** — `client.governanceclient.get_rule_tuning(rule_id)` - **TypeScript** — `getRuleTuning(ruleId)` - **Go** — `GetRuleTuning(ruleID)` **Parameters:** `rule_id` ### `lift_legal_hold(case_id)` lift_legal_hold(case_id)` - **Python** — `client.governanceclient.lift_legal_hold(case_id)` - **TypeScript** — `liftLegalHold(caseId)` - **Go** — `LiftLegalHold(caseID)` **Parameters:** `case_id` ### `list_alerts(severity=None, since_ns=None, limit=100)` list_alerts(severity=None, since_ns=None, limit=100)` - **Python** — `client.governanceclient.list_alerts(severity=None, since_ns=None, limit=100)` - **TypeScript** — `listAlerts(opts)` - **Go** — `ListAlerts()` **Parameters:** `severity=None`, `since_ns=None`, `limit=100` ### `list_legal_holds()` list_legal_holds()` - **Python** — `client.governanceclient.list_legal_holds()` - **TypeScript** — `listLegalHolds()` - **Go** — `ListLegalHolds()` ### `list_retention_policies()` list_retention_policies()` - **Python** — `client.governanceclient.list_retention_policies()` - **TypeScript** — `listRetentionPolicies()` - **Go** — `ListRetentionPolicies()` ### `list_rules(object_type=None)` list_rules(object_type=None)` - **Python** — `client.governanceclient.list_rules(object_type=None)` - **TypeScript** — `listRules(opts)` - **Go** — `ListRules(objectType)` **Parameters:** `object_type=None` ### `list_worm_policies()` list_worm_policies()` - **Python** — `client.governanceclient.list_worm_policies()` - **TypeScript** — `listWormPolicies()` - **Go** — `ListWormPolicies()` ### `snooze_rule(rule_id, duration_secs)` snooze_rule(rule_id, duration_secs)` - **Python** — `client.governanceclient.snooze_rule(rule_id, duration_secs)` - **TypeScript** — `snoozeRule(ruleId, durationSecs)` - **Go** — `SnoozeRule(ruleID, durationSecs)` **Parameters:** `rule_id`, `duration_secs` ### `suppress_rule(rule_id, entity_id, condition=None)` suppress_rule(rule_id, entity_id, condition=None)` - **Python** — `client.governanceclient.suppress_rule(rule_id, entity_id, condition=None)` - **TypeScript** — `suppressRule(ruleId, entityId, opts)` - **Go** — `SuppressRule(ruleID, entityID)` **Parameters:** `rule_id`, `entity_id`, `condition=None` ### `update_alert(alert_id, status=None, assignee=None, note=None)` update_alert(alert_id, status=None, assignee=None, note=None)` - **Python** — `client.governanceclient.update_alert(alert_id, status=None, assignee=None, note=None)` - **TypeScript** — `updateAlert(alertId, opts)` - **Go** — `UpdateAlert(alertID)` **Parameters:** `alert_id`, `status=None`, `assignee=None`, `note=None` ### `fromClient(client)` fromClient(client)` - **TypeScript** — `fromClient(client)` **Parameters:** `client` {/* END GENERATED */} ============================================================================== # Graph Algorithms & Intelligence URL: https://relatadb.dev/docs/sdks/methods/graph ============================================================================== # Graph Algorithms & Intelligence Graph runs over a CSR adjacency with PLL distance (ADR graph). `graph_dijkstra` is shortest path; `graph_scc` finds fraud rings; `graph_pagerank` is centrality with tunable `damping`/`max_iter`. The intelligence operators (beneficial ownership, sanctions, convoy, burner, crypto trace, geofence…) compose the algorithms with domain logic. {/* BEGIN GENERATED — overwritten by scripts/gen_sdk_methods_docs.py */} > **23 methods** across this domain. Signatures, parameters (incl. keyword-only tunable defaults), and the three SDK spellings are ported from the live SDK source; flagship methods carry a hand-written *When to use* + example. ## `RelataClient` ### `beneficial_ownership_chain(party, max_depth=None, purpose=None)` **When to use.** Trace ultimate beneficial ownership through a corporate hierarchy (intel operator). - **Python** — `client.relataclient.beneficial_ownership_chain(party, max_depth=None, purpose=None)` - **TypeScript** — `beneficialOwnershipChain(party, opts)` - **Go** — `BeneficialOwnershipChain(purpose, party, maxDepth)` **Parameters:** `party`, `max_depth=None`, `purpose=None` **Example** ```python ubo = relata.beneficial_ownership_chain("ShellCo") ``` ### `graph_dijkstra(object_type, from_id, to_id, purpose=None)` **When to use.** Shortest path between two nodes, governed. - **Python** — `client.relataclient.graph_dijkstra(object_type, from_id, to_id, purpose=None)` - **TypeScript** — `graphDijkstra(objectType, fromId, toId, purpose)` - **Go** — `GraphDijkstra(purpose, objectType, from, to)` **Parameters:** `object_type`, `from_id`, `to_id`, `purpose=None` **Example** ```python path = relata.graph_dijkstra("Person", "p-1", "p-9") ``` ```go p, _ := relata.GraphDijkstra(ctx, "Person", "p-1", "p-9") ``` ### `graph_pagerank(object_type, damping=None, max_iter=None, purpose=None)` **When to use.** Centrality / influence ranking — tune `damping` (server default 0.85) and `max_iter`. - **Python** — `client.relataclient.graph_pagerank(object_type, damping=None, max_iter=None, purpose=None)` - **TypeScript** — `graphPageRank(objectType, opts)` - **Go** — `GraphPageRank(purpose, objectType)` **Parameters:** `object_type`, `damping=None`, `max_iter=None`, `purpose=None` **Example** ```python ranks = relata.graph_pagerank("Transaction", damping=0.85, max_iter=100) ``` ### `graph_scc(object_type, purpose=None)` **When to use.** Strongly-connected components — the fraud-ring primitive. - **Python** — `client.relataclient.graph_scc(object_type, purpose=None)` - **TypeScript** — `graphSCC(objectType, purpose)` - **Go** — `GraphSCC(purpose, objectType)` **Parameters:** `object_type`, `purpose=None` **Example** ```python rings = relata.graph_scc("Transaction") ``` ### `graph_traverse(from_id, direction=None, max_depth=None, limit=None)` **When to use.** BFS traversal from a node — tune `direction`/`max_depth`/`limit`. - **Python** — `client.relataclient.graph_traverse(from_id, direction=None, max_depth=None, limit=None)` - **TypeScript** — `graphTraverse(fromId, opts)` - **Go** — `GraphTraverse(from)` **Parameters:** `from_id`, `direction=None`, `max_depth=None`, `limit=None` **Example** ```python ring = relata.graph_traverse("p-1", direction="both", max_depth=4, limit=200) ``` ### `sanctions_screen(party, threshold=None, purpose=None)` **When to use.** Screen entities against sanctions lists via the governed intel operator. - **Python** — `client.relataclient.sanctions_screen(party, threshold=None, purpose=None)` - **TypeScript** — `sanctionsScreen(party, opts)` - **Go** — `SanctionsScreen(purpose, party)` **Parameters:** `party`, `threshold=None`, `purpose=None` **Example** ```python hits = relata.sanctions_screen("Person", watchlist="ofac") ``` ### `burner_detect(max_age_days=None, max_calls=None, purpose=None)` burner_detect(max_age_days=None, max_calls=None, purpose=None)` - **Python** — `client.relataclient.burner_detect(max_age_days=None, max_calls=None, purpose=None)` - **TypeScript** — `burnerDetect(opts)` - **Go** — `BurnerDetect(purpose)` **Parameters:** `max_age_days=None`, `max_calls=None`, `purpose=None` ### `convoy_detect(radius_m=None, time_tol_secs=None, min_points=None, purpose=None)` convoy_detect(radius_m=None, time_tol_secs=None, min_points=None, purpose=None)` - **Python** — `client.relataclient.convoy_detect(radius_m=None, time_tol_secs=None, min_points=None, purpose=None)` - **TypeScript** — `convoyDetect(opts)` - **Go** — `ConvoyDetect(purpose)` **Parameters:** `radius_m=None`, `time_tol_secs=None`, `min_points=None`, `purpose=None` ### `crime_pattern_cluster(area, purpose=None)` crime_pattern_cluster(area, purpose=None)` - **Python** — `client.relataclient.crime_pattern_cluster(area, purpose=None)` - **TypeScript** — `crimePatternCluster(area, purpose)` - **Go** — `CrimePatternCluster(purpose, area)` **Parameters:** `area`, `purpose=None` ### `crypto_trace(entity, purpose=None)` crypto_trace(entity, purpose=None)` - **Python** — `client.relataclient.crypto_trace(entity, purpose=None)` - **TypeScript** — `cryptoTrace(entity, purpose)` - **Go** — `CryptoTrace(purpose, entity)` **Parameters:** `entity`, `purpose=None` ### `dark_fleet_detect(max_gap_hours=None, purpose=None)` dark_fleet_detect(max_gap_hours=None, purpose=None)` - **Python** — `client.relataclient.dark_fleet_detect(max_gap_hours=None, purpose=None)` - **TypeScript** — `darkFleetDetect(opts)` - **Go** — `DarkFleetDetect(purpose, maxGapHours)` **Parameters:** `max_gap_hours=None`, `purpose=None` ### `dns_tunnel_detect(entity, purpose=None)` dns_tunnel_detect(entity, purpose=None)` - **Python** — `client.relataclient.dns_tunnel_detect(entity, purpose=None)` - **TypeScript** — `dnsTunnelDetect(entity, purpose)` - **Go** — `DnsTunnelDetect(purpose, entity)` **Parameters:** `entity`, `purpose=None` ### `geofence(fence, target_type=None, purpose=None)` geofence(fence, target_type=None, purpose=None)` - **Python** — `client.relataclient.geofence(fence, target_type=None, purpose=None)` - **TypeScript** — `geofence(fence, opts)` - **Go** — `Geofence(purpose, fence, targetType)` **Parameters:** `fence`, `target_type=None`, `purpose=None` ### `graph_community(object_type, purpose=None)` graph_community(object_type, purpose=None)` - **Python** — `client.relataclient.graph_community(object_type, purpose=None)` - **TypeScript** — `graphCommunity(objectType, purpose)` - **Go** — `GraphCommunity(purpose, objectType)` **Parameters:** `object_type`, `purpose=None` ### `graph_cycles(object_type, purpose=None)` graph_cycles(object_type, purpose=None)` - **Python** — `client.relataclient.graph_cycles(object_type, purpose=None)` - **TypeScript** — `graphCycles(objectType, purpose)` - **Go** — `GraphCycles(purpose, objectType)` **Parameters:** `object_type`, `purpose=None` ### `graph_link_predict(object_type, purpose=None)` graph_link_predict(object_type, purpose=None)` - **Python** — `client.relataclient.graph_link_predict(object_type, purpose=None)` - **TypeScript** — `graphLinkPredict(objectType, purpose)` - **Go** — `GraphLinkPredict(purpose, objectType)` **Parameters:** `object_type`, `purpose=None` ### `graph_node_similarity(object_type, node, purpose=None)` graph_node_similarity(object_type, node, purpose=None)` - **Python** — `client.relataclient.graph_node_similarity(object_type, node, purpose=None)` - **TypeScript** — `graphNodeSimilarity(objectType, node, purpose)` - **Go** — `GraphNodeSimilarity(purpose, objectType, node)` **Parameters:** `object_type`, `node`, `purpose=None` ### `graph_shortest_path(from_id, to_id, max_hops=None)` graph_shortest_path(from_id, to_id, max_hops=None)` - **Python** — `client.relataclient.graph_shortest_path(from_id, to_id, max_hops=None)` - **TypeScript** — `graphShortestPath(fromId, toId, opts)` - **Go** — `GraphShortestPath(from, to)` **Parameters:** `from_id`, `to_id`, `max_hops=None` ### `graph_triangle_count(object_type, purpose=None)` graph_triangle_count(object_type, purpose=None)` - **Python** — `client.relataclient.graph_triangle_count(object_type, purpose=None)` - **TypeScript** — `graphTriangleCount(objectType, purpose)` - **Go** — `GraphTriangleCount(purpose, objectType)` **Parameters:** `object_type`, `purpose=None` ### `hawala_trace(seed, max_hops=None, purpose=None)` hawala_trace(seed, max_hops=None, purpose=None)` - **Python** — `client.relataclient.hawala_trace(seed, max_hops=None, purpose=None)` - **TypeScript** — `hawalaTrace(seed, opts)` - **Go** — `HawalaTrace(purpose, seed)` **Parameters:** `seed`, `max_hops=None`, `purpose=None` ### `vessel_to_vessel_transfer(proximity_nm=None, time_window_minutes=None, purpose=None)` vessel_to_vessel_transfer(proximity_nm=None, time_window_minutes=None, purpose=None)` - **Python** — `client.relataclient.vessel_to_vessel_transfer(proximity_nm=None, time_window_minutes=None, purpose=None)` - **TypeScript** — `vesselToVesselTransfer(opts)` - **Go** — `VesselToVesselTransfer(purpose, proximityNm, timeWindowMinutes)` **Parameters:** `proximity_nm=None`, `time_window_minutes=None`, `purpose=None` ### `vessel_track(mmsi, window_secs=None, purpose=None)` vessel_track(mmsi, window_secs=None, purpose=None)` - **Python** — `client.relataclient.vessel_track(mmsi, window_secs=None, purpose=None)` - **TypeScript** — `vesselTrack(mmsi, opts)` - **Go** — `VesselTrack(purpose, mmsi, windowSecs)` **Parameters:** `mmsi`, `window_secs=None`, `purpose=None` ### `wire_reconstruction(account, tolerance_pct=None, purpose=None)` wire_reconstruction(account, tolerance_pct=None, purpose=None)` - **Python** — `client.relataclient.wire_reconstruction(account, tolerance_pct=None, purpose=None)` - **TypeScript** — `wireReconstruction(account, opts)` - **Go** — `WireReconstruction(purpose, account, tolerancePct)` **Parameters:** `account`, `tolerance_pct=None`, `purpose=None` {/* END GENERATED */} ============================================================================== # Identity & Entity Lifecycle URL: https://relatadb.dev/docs/sdks/methods/identity ============================================================================== # Identity & Entity Lifecycle Identity is a universal lookup MV keyed by `(CanonicalKind, payload_bytes)` (ADR-021). `resolve_identity` unifies a seed across all canonical validators; `fuse_identities` is the ontological merge; `erase_subject` is the governed right-to-erasure (cryptographic, ADR-151). {/* BEGIN GENERATED — overwritten by scripts/gen_sdk_methods_docs.py */} > **13 methods** across this domain. Signatures, parameters (incl. keyword-only tunable defaults), and the three SDK spellings are ported from the live SDK source; flagship methods carry a hand-written *When to use* + example. ## `IdentityClient` ### `erase_subject(subject_identity, reason, certify=True, purpose)` **When to use.** GDPR right-to-erasure — cryptographic, governed, audit-logged (ADR-151). - **Python** — `client.identityclient.erase_subject(subject_identity, reason, certify=True, purpose)` - **TypeScript** — `eraseSubject(subjectIdentity, reason, opts)` - **Go** — `EraseSubject(subjectIdentity, reason, purpose)` **Parameters:** `subject_identity`, `reason`, `certify=True`, `purpose` **Example** ```python IdentityClient.from_client(relata).erase_subject("alice@x.io", reason="gdpr-art17") ``` ### `invoke_lookup(name, key)` invoke_lookup(name, key)` - **Python** — `client.identityclient.invoke_lookup(name, key)` - **TypeScript** — `invokeLookup(name, key)` - **Go** — `InvokeLookup(name, key)` **Parameters:** `name`, `key` ### `label(identity, label, confidence=1.0, source_id=None)` label(identity, label, confidence=1.0, source_id=None)` - **Python** — `client.identityclient.label(identity, label, confidence=1.0, source_id=None)` - **TypeScript** — `label(identity, label, opts)` - **Go** — `Label(identity, label)` **Parameters:** `identity`, `label`, `confidence=1.0`, `source_id=None` ### `list_lookups()` list_lookups()` - **Python** — `client.identityclient.list_lookups()` - **TypeScript** — `listLookups()` - **Go** — `ListLookups()` ### `record_uncertainty(identity, reason, suggested_alternatives=None)` record_uncertainty(identity, reason, suggested_alternatives=None)` - **Python** — `client.identityclient.record_uncertainty(identity, reason, suggested_alternatives=None)` - **TypeScript** — `recordUncertainty(identity, reason, opts)` - **Go** — `RecordUncertainty(identity, reason)` **Parameters:** `identity`, `reason`, `suggested_alternatives=None` ### `register_lookup(name, csv_data, key_column, value_columns=None)` register_lookup(name, csv_data, key_column, value_columns=None)` - **Python** — `client.identityclient.register_lookup(name, csv_data, key_column, value_columns=None)` - **TypeScript** — `registerLookup(name, csvData, opts)` - **Go** — `RegisterLookup(name, csvData, keyColumn)` **Parameters:** `name`, `csv_data`, `key_column`, `value_columns=None` ## `RelataClient` ### `detect_identities(text, purpose=None)` **When to use.** Run SmartIngest's lazy identifier detection over a type's rows (ADR-051). - **Python** — `client.relataclient.detect_identities(text, purpose=None)` - **TypeScript** — `detectIdentities(text, purpose)` - **Go** — `DetectIdentities(purpose, text)` **Parameters:** `text`, `purpose=None` **Example** ```python relata.detect_identities("Person") ``` ### `fuse_identities(id_a, id_b, purpose=None)` **When to use.** Ontologically merge two entities into one governed identity (provenance-tracked). - **Python** — `client.relataclient.fuse_identities(id_a, id_b, purpose=None)` - **TypeScript** — `fuseIdentities(idA, idB, purpose)` - **Go** — `FuseIdentities(purpose, idA, idB)` **Parameters:** `id_a`, `id_b`, `purpose=None` **Example** ```python relata.fuse_identities(id_a, id_b, reason="sanctions-match") ``` ### `resolve_identity(value, mode=None, purpose=None)` **When to use.** Unify a seed identifier (email/phone/IBAN…) into one entity + aliases across all canonical validators. - **Python** — `client.relataclient.resolve_identity(value, mode=None, purpose=None)` - **TypeScript** — `resolveIdentity(value, purpose, mode)` - **Go** — `ResolveIdentity(purpose, value, mode)` **Parameters:** `value`, `mode=None`, `purpose=None` **Example** ```python entity = relata.resolve_identities("alice@x.io") print(entity.aliases) ``` ```go e, _ := relata.ResolveIdentity(ctx, "alice@x.io") ``` ### `erase_subject(subject, reason="gdpr-art17-request", purpose=None)` erase_subject(subject, reason="gdpr-art17-request", purpose=None)` - **Python** — `client.relataclient.erase_subject(subject, reason="gdpr-art17-request", purpose=None)` - **TypeScript** — `eraseSubject(subject, reason)` - **Go** — `EraseSubject(purpose, subject, reason)` **Parameters:** `subject`, `reason="gdpr-art17-request"`, `purpose=None` ### `identity_cluster(value, purpose=None)` identity_cluster(value, purpose=None)` - **Python** — `client.relataclient.identity_cluster(value, purpose=None)` - **TypeScript** — `identityCluster(value, purpose)` - **Go** — `IdentityCluster(purpose, value)` **Parameters:** `value`, `purpose=None` ### `same_identity(a, b, purpose=None)` same_identity(a, b, purpose=None)` - **Python** — `client.relataclient.same_identity(a, b, purpose=None)` - **TypeScript** — `sameIdentity(a, b, purpose)` - **Go** — `SameIdentity(purpose, idA, idB)` **Parameters:** `a`, `b`, `purpose=None` ### `split_identities(id_a, id_b, purpose=None)` split_identities(id_a, id_b, purpose=None)` - **Python** — `client.relataclient.split_identities(id_a, id_b, purpose=None)` - **TypeScript** — `splitIdentities(idA, idB, purpose)` - **Go** — `SplitIdentities(purpose, idA, idB)` **Parameters:** `id_a`, `id_b`, `purpose=None` {/* END GENERATED */} ============================================================================== # SDK Method Reference URL: https://relatadb.dev/docs/sdks/methods ============================================================================== # SDK Method Reference RelataDB is a **governed temporal knowledge database** — policy, provenance, and bi-temporal history live in the query path, not bolted on. The three consumer SDKs (Python, TypeScript, Go) expose one shape over that surface: every call declares a `purpose`, runs through ACL + cell-masking + audit, and writes bi-temporal rows with provenance. This reference indexes **every public method**, grouped by domain. Each entry shows the signature with keyword-only tunable defaults (e.g. `damping=None`, `top_k=5`), the three SDK spellings, and — for the flagship methods customers actually tune — a hand-written *When to use* note and a runnable example. Source of truth: the SDK source itself. This page is ported from RelataDB's generated `sdk-methods.md` and refreshed manually when the surface changes — re-run `scripts/gen_sdk_methods_docs.py` in this repo. ## Coverage by domain | Domain | Methods | Flagship (with example) | Clients | |---|---:|---:|---| | [Query & SQL](/docs/sdks/methods/query) | 15 | 4 | FlightClient, RelataClient | | [Search & Retrieval](/docs/sdks/methods/search) | 3 | 2 | RelataClient, SearchClient | | [Vectors & Embeddings](/docs/sdks/methods/vectors) | 12 | 2 | RelataClient, VectorClient | | [Ingest](/docs/sdks/methods/ingest) | 10 | 2 | IngestClient, RelataClient | | [Objects & CRUD](/docs/sdks/methods/objects) | 5 | 1 | ObjectClient | | [Types & Ontology](/docs/sdks/methods/ontology) | 14 | 2 | RelataClient | | [Identity & Entity Lifecycle](/docs/sdks/methods/identity) | 13 | 4 | IdentityClient, RelataClient | | [Graph Algorithms & Intelligence](/docs/sdks/methods/graph) | 23 | 6 | RelataClient | | [Governance & Compliance](/docs/sdks/methods/governance) | 21 | 5 | GovernanceClient | | [Agent Memory](/docs/sdks/methods/memory) | 16 | 3 | Memory, RelataClient | | [MCP & Agent-to-Agent](/docs/sdks/methods/agents) | 71 | 1 | McpClient | | [Streaming & Transparency Log](/docs/sdks/methods/streaming) | 8 | 1 | LogClient, StreamingClient | | [Audit & Reporting](/docs/sdks/methods/audit) | 10 | 2 | AuditClient, RelataClient | | [Admin, Ops & Multi-Tenancy](/docs/sdks/methods/admin) | 46 | 3 | BackupClient, RelataClient, SystemClient, TenantAdminClient, TokenClient | | **Total** | **267** | **42** | — | ## Cross-SDK spelling conventions | Concept | Python | TypeScript | Go | |---|---|---|---| | method naming | `snake_case` | `camelCase` | `PascalCase` | | async (Python only) | `a`-prefix (`aquery`) | `async` keyword | `ctx context.Context` first | | typed sub-client | `IngestClient.from_client(c)` | `c.ingest` | `c.Ingest` | See also: [SDK overview](/docs/sdks/overview) for install + the parity matrix, and the per-language quickstarts ([Python](/docs/sdks/python) · [TypeScript](/docs/sdks/typescript) · [Go](/docs/sdks/go)). ============================================================================== # Ingest URL: https://relatadb.dev/docs/sdks/methods/ingest ============================================================================== # Ingest Every ingest path declares `purpose` and respects tenant scope. `on_conflict` (`upsert`|`skip`|`error`) and per-call `detect_packs` (SmartIngest override) are uniform across bulk/CSV/iter. OTLP and document ingest land in the same bi-temporal store as everything else. {/* BEGIN GENERATED — overwritten by scripts/gen_sdk_methods_docs.py */} > **10 methods** across this domain. Signatures, parameters (incl. keyword-only tunable defaults), and the three SDK spellings are ported from the live SDK source; flagship methods carry a hand-written *When to use* + example. ## `IngestClient` ### `bulk(object_type, rows, purpose=None, detect_packs=None, on_conflict=None, tenant_id=None)` **When to use.** Insert/upsert many rows of one type with a conflict strategy and SmartIngest. - **Python** — `client.ingestclient.bulk(object_type, rows, purpose=None, detect_packs=None, on_conflict=None, tenant_id=None)` - **TypeScript** — `bulk(objectType, rows, opts)` - **Go** — `Bulk(objectType, rows)` **Parameters:** `object_type`, `rows`, `purpose=None`, `detect_packs=None`, `on_conflict=None`, `tenant_id=None` **Example** ```python IngestClient.from_client(relata).bulk( "Person", [{"_pk": "p1", "name": "Alice", "email": "a@x.io"}], on_conflict="upsert", detect_packs=["email", "phone"]) ``` ```typescript await relata.bulk("Person", [{_pk:'p1', name:'Alice'}], {onConflict:'upsert'}) ``` ```go _, _ = ingest.Bulk(ctx, "Person", []map[string]any{{"_pk":"p1","name":"Alice"}}) ``` ### `ingest_iter(object_type, rows, purpose=None, batch_size=500)` **When to use.** Stream a generator/iterator of rows in batches without loading all in memory. - **Python** — `client.ingestclient.ingest_iter(object_type, rows, purpose=None, batch_size=500)` - **Go** — `IngestIter(objectType, rows, purpose, batchSize)` **Parameters:** `object_type`, `rows`, `purpose=None`, `batch_size=500` **Example** ```python def gen(): for i in range(100_000): yield {"_pk": f"p{i}", "name": f"n{i}"} IngestClient.from_client(relata).ingest_iter("Person", gen(), batch_size=2000) ``` ### `bulk_csv(object_type, csv_text, purpose=None, detect_packs=None)` bulk_csv(object_type, csv_text, purpose=None, detect_packs=None)` - **Python** — `client.ingestclient.bulk_csv(object_type, csv_text, purpose=None, detect_packs=None)` - **TypeScript** — `bulkCsv(objectType, csvText, opts)` - **Go** — `BulkCSV(objectType, csvText)` **Parameters:** `object_type`, `csv_text`, `purpose=None`, `detect_packs=None` ### `ingest_cdr(rows, purpose=None)` ingest_cdr(rows, purpose=None)` - **Python** — `client.ingestclient.ingest_cdr(rows, purpose=None)` - **TypeScript** — `ingestCdr(rows, opts)` - **Go** — `IngestCDR(rows)` **Parameters:** `rows`, `purpose=None` ### `media_status(task_id)` media_status(task_id)` - **Python** — `client.ingestclient.media_status(task_id)` - **TypeScript** — `mediaStatus(taskId)` - **Go** — `MediaStatus(taskID)` **Parameters:** `task_id` ### `otlp_logs(payload, purpose=None)` otlp_logs(payload, purpose=None)` - **Python** — `client.ingestclient.otlp_logs(payload, purpose=None)` - **TypeScript** — `otlpLogs(payload, opts)` - **Go** — `OTLPLogs(payload)` **Parameters:** `payload`, `purpose=None` ### `otlp_metrics(payload, purpose=None)` otlp_metrics(payload, purpose=None)` - **Python** — `client.ingestclient.otlp_metrics(payload, purpose=None)` - **TypeScript** — `otlpMetrics(payload, opts)` - **Go** — `OTLPMetrics(payload)` **Parameters:** `payload`, `purpose=None` ### `otlp_traces(payload, purpose=None)` otlp_traces(payload, purpose=None)` - **Python** — `client.ingestclient.otlp_traces(payload, purpose=None)` - **TypeScript** — `otlpTraces(payload, opts)` - **Go** — `OTLPTraces(payload)` **Parameters:** `payload`, `purpose=None` ### `ingestAuto(objectType, rows, opts)` ingestAuto(objectType, rows, opts)` - **TypeScript** — `ingestAuto(objectType, rows, opts)` - **Go** — `IngestAuto(objectType, rows)` **Parameters:** `objectType`, `rows`, `opts` ## `RelataClient` ### `ingest_document(chunks_jsonl, manifest_json)` ingest_document(chunks_jsonl, manifest_json)` - **Python** — `client.relataclient.ingest_document(chunks_jsonl, manifest_json)` - **TypeScript** — `ingestDocument(chunksJsonl, manifestJson)` - **Go** — `IngestDocument(chunksJSONL, manifestJSON)` **Parameters:** `chunks_jsonl`, `manifest_json` {/* END GENERATED */} ============================================================================== # Agent Memory URL: https://relatadb.dev/docs/sdks/methods/memory ============================================================================== # Agent Memory `Memory` is governed agent memory: every verb runs through ACL + purpose + audit, and `forget` is a governed retract (not a hard delete). Recall blends hybrid search with recency and a confidence/stability model — the cognitive substrate for LangChain/LlamaIndex/CrewAI/AutoGen/LangGraph adapters. {/* BEGIN GENERATED — overwritten by scripts/gen_sdk_methods_docs.py */} > **16 methods** across this domain. Signatures, parameters (incl. keyword-only tunable defaults), and the three SDK spellings are ported from the live SDK source; flagship methods carry a hand-written *When to use* + example. ## `Memory` ### `add(content, confidence=1.0, memory_class='semantic', session_id=None)` **When to use.** Remember — store a memory with confidence/class; returns its id. - **Python** — `client.memory.add(content, confidence=1.0, memory_class='semantic', session_id=None)` - **TypeScript** — `add(content, opts)` - **Go** — `Add(content)` **Parameters:** `content`, `confidence=1.0`, `memory_class='semantic'`, `session_id=None` **Example** ```python from relata import Memory with Memory(url, purpose="agent-notes") as m: mid = m.add("Alice prefers dark mode", confidence=0.9) ``` ### `forget(memory_id)` **When to use.** Governed retract of a memory — audit-logged, not a hard delete. - **Python** — `client.memory.forget(memory_id)` - **TypeScript** — `forget(memoryId)` - **Go** — `Forget(memoryID)` **Parameters:** `memory_id` **Example** ```python m.forget(mid, reason="stale-preference") ``` ### `search(query, top_k=5, session_id=None, as_of=None, min_confidence=None, recency_half_life_secs=None, budget_tokens=None, stability_days=None, cancel_threshold=None)` **When to use.** Recall — hybrid + recency memory retrieval. - **Python** — `client.memory.search(query, top_k=5, session_id=None, as_of=None, min_confidence=None, recency_half_life_secs=None, budget_tokens=None, stability_days=None, cancel_threshold=None)` - **TypeScript** — `search(query, opts)` - **Go** — `Search(query)` **Parameters:** `query`, `top_k=5`, `session_id=None`, `as_of=None`, `min_confidence=None`, `recency_half_life_secs=None`, `budget_tokens=None`, `stability_days=None`, `cancel_threshold=None` **Example** ```python for hit in m.search("ui preferences", top_k=5): print(hit["content"], hit["score"]) ``` ### `add_batch(items, session_id=None)` add_batch(items, session_id=None)` - **Python** — `client.memory.add_batch(items, session_id=None)` - **TypeScript** — `addBatch(items, opts)` - **Go** — `AddBatch(items)` **Parameters:** `items`, `session_id=None` ### `associate(source_id, target_id, relation, confidence=1.0)` associate(source_id, target_id, relation, confidence=1.0)` - **Python** — `client.memory.associate(source_id, target_id, relation, confidence=1.0)` - **TypeScript** — `associate(sourceId, targetId, relation, opts)` - **Go** — `Associate(sourceID, targetID, relation)` **Parameters:** `source_id`, `target_id`, `relation`, `confidence=1.0` ### `batch_search(queries, top_k=5)` batch_search(queries, top_k=5)` - **Python** — `client.memory.batch_search(queries, top_k=5)` **Parameters:** `queries`, `top_k=5` ### `episodes(session_id=None, as_of=None)` episodes(session_id=None, as_of=None)` - **Python** — `client.memory.episodes(session_id=None, as_of=None)` - **TypeScript** — `episodes(opts)` - **Go** — `Episodes()` **Parameters:** `session_id=None`, `as_of=None` ### `get(memory_id)` get(memory_id)` - **Python** — `client.memory.get(memory_id)` - **TypeScript** — `get(memoryId)` - **Go** — `Get(memoryID)` **Parameters:** `memory_id` ### `justify(memory_id)` justify(memory_id)` - **Python** — `client.memory.justify(memory_id)` - **TypeScript** — `justify(memoryId)` - **Go** — `Justify(memoryID)` **Parameters:** `memory_id` ### `resolve(memory_id)` resolve(memory_id)` - **Python** — `client.memory.resolve(memory_id)` - **TypeScript** — `resolve(memoryId)` - **Go** — `Resolve(memoryID)` **Parameters:** `memory_id` ### `search_detailed(query, top_k=5, session_id=None, as_of=None, min_confidence=None, recency_half_life_secs=None, budget_tokens=None, stability_days=None, cancel_threshold=None)` search_detailed(query, top_k=5, session_id=None, as_of=None, min_confidence=None, recency_half_life_secs=None, budget_tokens=None, stability_days=None, cancel_threshold=None)` - **Python** — `client.memory.search_detailed(query, top_k=5, session_id=None, as_of=None, min_confidence=None, recency_half_life_secs=None, budget_tokens=None, stability_days=None, cancel_threshold=None)` - **TypeScript** — `searchDetailed(query, opts)` - **Go** — `SearchDetailed(query)` **Parameters:** `query`, `top_k=5`, `session_id=None`, `as_of=None`, `min_confidence=None`, `recency_half_life_secs=None`, `budget_tokens=None`, `stability_days=None`, `cancel_threshold=None` ### `summarise(source_ids, summary_content=None)` summarise(source_ids, summary_content=None)` - **Python** — `client.memory.summarise(source_ids, summary_content=None)` - **TypeScript** — `summarise(sourceIds, opts)` - **Go** — `Summarise(sourceIDs)` **Parameters:** `source_ids`, `summary_content=None` ### `update(memory_id, content)` update(memory_id, content)` - **Python** — `client.memory.update(memory_id, content)` - **TypeScript** — `update(memoryId, content)` - **Go** — `Update(memoryID, content)` **Parameters:** `memory_id`, `content` ## `RelataClient` ### `forget(id)` forget(id)` - **TypeScript** — `forget(id)` **Parameters:** `id` ### `recall(query, opts)` recall(query, opts)` - **TypeScript** — `recall(query, opts)` **Parameters:** `query`, `opts` ### `remember(content, opts)` remember(content, opts)` - **TypeScript** — `remember(content, opts)` **Parameters:** `content`, `opts` {/* END GENERATED */} ============================================================================== # Objects & CRUD URL: https://relatadb.dev/docs/sdks/methods/objects ============================================================================== # Objects & CRUD `ObjectClient` is the typed CRUD handle: `upsert` for one row, `batch_upsert` for many, `typed_upsert` for a Pydantic/dataclass-shaped object. All write bi-temporal rows with provenance. {/* BEGIN GENERATED — overwritten by scripts/gen_sdk_methods_docs.py */} > **5 methods** across this domain. Signatures, parameters (incl. keyword-only tunable defaults), and the three SDK spellings are ported from the live SDK source; flagship methods carry a hand-written *When to use* + example. ## `ObjectClient` ### `upsert(object_type, object_id, fields, purpose=None, source=None, on_conflict='upsert')` **When to use.** Write one typed object (dict) as a bi-temporal row. - **Python** — `client.objectclient.upsert(object_type, object_id, fields, purpose=None, source=None, on_conflict='upsert')` - **TypeScript** — `upsert(objectType, objectId, fields, opts)` - **Go** — `Upsert(objectType, objectID, fields)` **Parameters:** `object_type`, `object_id`, `fields`, `purpose=None`, `source=None`, `on_conflict='upsert'` **Example** ```python ObjectClient.from_client(relata).upsert("Person", {"_pk":"p1","name":"Alice"}) ``` ### `batch_upsert(object_type, rows, purpose=None, source=None, on_conflict='upsert')` batch_upsert(object_type, rows, purpose=None, source=None, on_conflict='upsert')` - **Python** — `client.objectclient.batch_upsert(object_type, rows, purpose=None, source=None, on_conflict='upsert')` - **TypeScript** — `batchUpsert(objectType, rows, opts)` - **Go** — `BatchUpsert(objectType, rows)` **Parameters:** `object_type`, `rows`, `purpose=None`, `source=None`, `on_conflict='upsert'` ### `delete(object_type, object_id, purpose=None)` delete(object_type, object_id, purpose=None)` - **Python** — `client.objectclient.delete(object_type, object_id, purpose=None)` **Parameters:** `object_type`, `object_id`, `purpose=None` ### `get(object_type, object_id, purpose=None)` get(object_type, object_id, purpose=None)` - **Python** — `client.objectclient.get(object_type, object_id, purpose=None)` - **TypeScript** — `get(objectType, objectId)` **Parameters:** `object_type`, `object_id`, `purpose=None` ### `typed_upsert(obj, purpose=None)` typed_upsert(obj, purpose=None)` - **Python** — `client.objectclient.typed_upsert(obj, purpose=None)` **Parameters:** `obj`, `purpose=None` {/* END GENERATED */} ============================================================================== # Types & Ontology URL: https://relatadb.dev/docs/sdks/methods/ontology ============================================================================== # Types & Ontology Types are git-branched, schema-as-code (ADR ontology). `schema_alter` evolves a type in place; `ontology_migrate` applies a SHACL constraint set; `create_link`/`register_edge_type` wire the graph. Modules group a type pack. {/* BEGIN GENERATED — overwritten by scripts/gen_sdk_methods_docs.py */} > **14 methods** across this domain. Signatures, parameters (incl. keyword-only tunable defaults), and the three SDK spellings are ported from the live SDK source; flagship methods carry a hand-written *When to use* + example. ## `RelataClient` ### `register_type(name, **kwargs: object)` **When to use.** Register a new governed type with its column schema and constraints. - **Python** — `client.relataclient.register_type(name, **kwargs: object)` - **TypeScript** — `registerType(name, spec)` - **Go** — `RegisterType(name, spec)` **Parameters:** `name`, `**kwargs: object` **Example** ```python relata.register_type("Company", {"name": "string", "vat": "string?"}, pk="name") ``` ### `schema_alter(type_name, action, column, new_column=None, col_type=None, optional=None)` **When to use.** Evolve a type in place (add/drop columns) without a full re-register (ADR-1307). - **Python** — `client.relataclient.schema_alter(type_name, action, column, new_column=None, col_type=None, optional=None)` - **TypeScript** — `schemaAlter(name, action, column, opts)` - **Go** — `SchemaAlter(name, action, column)` **Parameters:** `type_name`, `action`, `column`, `new_column=None`, `col_type=None`, `optional=None` **Example** ```python relata.schema_alter("Company", add={"lei": "string?"}) ``` ### `create_link(link_name, source_id, source_type, target_id, target_type)` create_link(link_name, source_id, source_type, target_id, target_type)` - **Python** — `client.relataclient.create_link(link_name, source_id, source_type, target_id, target_type)` - **TypeScript** — `createLink(params)` - **Go** — `CreateLink(params)` **Parameters:** `link_name`, `source_id`, `source_type`, `target_id`, `target_type` ### `delete_webhook(webhook_id)` delete_webhook(webhook_id)` - **Python** — `client.relataclient.delete_webhook(webhook_id)` - **TypeScript** — `deleteWebhook(id)` - **Go** — `DeleteWebhook(id)` **Parameters:** `webhook_id` ### `deregister_type(name)` deregister_type(name)` - **Python** — `client.relataclient.deregister_type(name)` - **TypeScript** — `deregisterType(name)` - **Go** — `DeregisterType(name)` **Parameters:** `name` ### `enrichment_rules(rules)` enrichment_rules(rules)` - **Python** — `client.relataclient.enrichment_rules(rules)` - **TypeScript** — `enrichmentRules(rules)` - **Go** — `EnrichmentRules(rules)` **Parameters:** `rules` ### `list_edge_types()` list_edge_types()` - **Python** — `client.relataclient.list_edge_types()` - **TypeScript** — `listEdgeTypes()` - **Go** — `ListEdgeTypes()` ### `list_modules()` list_modules()` - **Python** — `client.relataclient.list_modules()` - **TypeScript** — `listModules()` - **Go** — `ListModules()` ### `list_types()` list_types()` - **Python** — `client.relataclient.list_types()` - **TypeScript** — `listTypes()` - **Go** — `ListTypes()` ### `list_webhooks()` list_webhooks()` - **Python** — `client.relataclient.list_webhooks()` - **TypeScript** — `listWebhooks()` - **Go** — `ListWebhooks()` ### `ontology_migrate(schema)` ontology_migrate(schema)` - **Python** — `client.relataclient.ontology_migrate(schema)` - **TypeScript** — `ontologyMigrate(schema)` - **Go** — `OntologyMigrate(schema)` **Parameters:** `schema` ### `register_edge_type(from_type, to_type, label)` register_edge_type(from_type, to_type, label)` - **Python** — `client.relataclient.register_edge_type(from_type, to_type, label)` - **TypeScript** — `registerEdgeType(fromType, toType, label)` - **Go** — `RegisterEdgeType(fromType, toType, label)` **Parameters:** `from_type`, `to_type`, `label` ### `register_webhook(url, event_types=None)` register_webhook(url, event_types=None)` - **Python** — `client.relataclient.register_webhook(url, event_types=None)` - **TypeScript** — `registerWebhook(url, eventTypes)` - **Go** — `RegisterWebhook(url, eventTypes)` **Parameters:** `url`, `event_types=None` ### `type_detail(name)` type_detail(name)` - **Python** — `client.relataclient.type_detail(name)` - **TypeScript** — `typeDetail(name)` - **Go** — `TypeDetail(name)` **Parameters:** `name` {/* END GENERATED */} ============================================================================== # Query & SQL URL: https://relatadb.dev/docs/sdks/methods/query ============================================================================== # Query & SQL Every query runs through the governed path — ACL, cell masking, audit — and carries an optional `PURPOSE`. SQL is the primary door; GraphQL/SPARQL/Cypher are auto-routed. Bi-temporal `AS OF` time-travel and `WITH PROVENANCE` are first-class on every read. Async mirrors: every method has an `a`-prefixed async twin (`query` → `aquery`) with the identical signature and semantics on Python. {/* BEGIN GENERATED — overwritten by scripts/gen_sdk_methods_docs.py */} > **15 methods** across this domain. Signatures, parameters (incl. keyword-only tunable defaults), and the three SDK spellings are ported from the live SDK source; flagship methods carry a hand-written *When to use* + example. ## `FlightClient` ### `query_flight(sql, flight_endpoint=None, purpose=None, bearer_token=None)` query_flight(sql, flight_endpoint=None, purpose=None, bearer_token=None)` - **Python** — `client.flightclient.query_flight(sql, flight_endpoint=None, purpose=None, bearer_token=None)` **Parameters:** `sql`, `flight_endpoint=None`, `purpose=None`, `bearer_token=None` ## `RelataClient` ### `query(sql, purpose=None)` **When to use.** You have raw SQL or a Cypher/GraphQL/SPARQL string and want the governed result as iterable rows. - **Python** — `client.relataclient.query(sql, purpose=None)` - **Go** — `Query(sql)` **Parameters:** `sql`, `purpose=None` **Example** ```python for row in relata.query("SELECT * FROM Person WHERE nationality = 'IN' LIMIT 10", purpose="analytics"): print(row["name"]) ``` ```typescript const rows = await relata.query("SELECT * FROM Person LIMIT 10", { purpose: "analytics" }); ``` ```go res, _ := relata.Query(ctx, "SELECT * FROM Person LIMIT 10") for _, row := range res.Rows { fmt.Println(row["name"]) } ``` ### `query_arrow(sql, purpose=None)` **When to use.** You want a zero-copy `pyarrow.Table` for analytics / pandas / Flight interchange. - **Python** — `client.relataclient.query_arrow(sql, purpose=None)` **Parameters:** `sql`, `purpose=None` **Example** ```python import pyarrow as pa table: pa.Table = relata.query_arrow("SELECT * FROM Person") df = table.to_pandas() ``` ### `query_flight(sql, flight_endpoint=None, purpose=None, bearer_token=None)` **When to use.** Bulk-pull or stream large results over gRPC (Arrow Flight) — the highest-throughput door. - **Python** — `client.relataclient.query_flight(sql, flight_endpoint=None, purpose=None, bearer_token=None)` - **Go** — `QueryFlight(sql, flightEndpoint, purpose, bearer)` **Parameters:** `sql`, `flight_endpoint=None`, `purpose=None`, `bearer_token=None` **Example** ```python table = relata.query_flight("SELECT * FROM TradeHistory") ``` ### `query_params(sql, params, purpose=None)` **When to use.** You have untrusted values — bind them. `?` placeholders auto-rewrite to `$N` server-side. - **Python** — `client.relataclient.query_params(sql, params, purpose=None)` **Parameters:** `sql`, `params`, `purpose=None` **Example** ```python rows = relata.query_params("SELECT * FROM Person WHERE nationality = ? AND age > ?", ['IN', 25]) ``` ```go res, _ := relata.QueryWithParams(ctx, "SELECT * FROM Person WHERE nationality = $1 AND age > $2", []any{"IN", 25}) ``` ### `create_schema_branch(name, from_branch)` create_schema_branch(name, from_branch)` - **Python** — `client.relataclient.create_schema_branch(name, from_branch)` - **TypeScript** — `createSchemaBranch(name, fromBranch)` - **Go** — `CreateSchemaBranch(name, fromBranch)` **Parameters:** `name`, `from_branch` ### `delete_schema_branch(name)` delete_schema_branch(name)` - **Python** — `client.relataclient.delete_schema_branch(name)` - **TypeScript** — `deleteSchemaBranch(name)` - **Go** — `DeleteSchemaBranch(name)` **Parameters:** `name` ### `export_data(object_type, format="json")` export_data(object_type, format="json")` - **Python** — `client.relataclient.export_data(object_type, format="json")` - **TypeScript** — `exportData(objectType, format)` - **Go** — `ExportData(objectType, format)` **Parameters:** `object_type`, `format="json"` ### `graphql(query, variables=None, operation_name=None)` graphql(query, variables=None, operation_name=None)` - **Python** — `client.relataclient.graphql(query, variables=None, operation_name=None)` - **TypeScript** — `graphql(query, variables, operationName)` - **Go** — `GraphQL(gqlQuery, variables, operationName)` **Parameters:** `query`, `variables=None`, `operation_name=None` ### `namespace(name)` namespace(name)` - **Python** — `client.relataclient.namespace(name)` - **TypeScript** — `namespace(name)` - **Go** — `Namespace(name)` **Parameters:** `name` ### `select(*columns_or_table: str)` select(*columns_or_table: str)` - **Python** — `client.relataclient.select(*columns_or_table: str)` - **TypeScript** — `select(type)` **Parameters:** `*columns_or_table: str` ### `session_commit(session_id)` session_commit(session_id)` - **Python** — `client.relataclient.session_commit(session_id)` - **TypeScript** — `sessionCommit(sessionId)` - **Go** — `SessionCommit(sessionID)` **Parameters:** `session_id` ### `session_diff(session_id)` session_diff(session_id)` - **Python** — `client.relataclient.session_diff(session_id)` - **TypeScript** — `sessionDiff(sessionId)` - **Go** — `SessionDiff(sessionID)` **Parameters:** `session_id` ### `session_discard(session_id)` session_discard(session_id)` - **Python** — `client.relataclient.session_discard(session_id)` - **TypeScript** — `sessionDiscard(sessionId)` - **Go** — `SessionDiscard(sessionID)` **Parameters:** `session_id` ### `sparql(query)` sparql(query)` - **Python** — `client.relataclient.sparql(query)` - **TypeScript** — `sparql(query)` - **Go** — `Sparql(query)` **Parameters:** `query` {/* END GENERATED */} ============================================================================== # Search & Retrieval URL: https://relatadb.dev/docs/sdks/methods/search ============================================================================== # Search & Retrieval `search` is BM25-only by default; pass `metric` and/or `weights` to route through the server's hybrid fusion. The typed `SearchClient.query` door exposes rank-by, filters, facets, and consistency controls without raw SQL. {/* BEGIN GENERATED — overwritten by scripts/gen_sdk_methods_docs.py */} > **3 methods** across this domain. Signatures, parameters (incl. keyword-only tunable defaults), and the three SDK spellings are ported from the live SDK source; flagship methods carry a hand-written *When to use* + example. ## `RelataClient` ### `multi_search(queries)` **When to use.** Federate one query across multiple object types in a single round-trip. - **Python** — `client.relataclient.multi_search(queries)` - **TypeScript** — `multiSearch(queries)` - **Go** — `MultiSearch(queries)` **Parameters:** `queries` **Example** ```python res = relata.multi_search([{"type": "Person", "query": "ahmed"}, {"type": "Company", "query": "shell"}]) ``` ### `search(query, type, limit=None, facets=None, highlight=False, filters=None, matching_strategy=None, typo_tolerance=None, metric=None, weights=None)` **When to use.** BM25 full-text search; add `metric`/`weights` to fuse with graph + vector channels (hybrid). - **Python** — `client.relataclient.search(query, type, limit=None, facets=None, highlight=False, filters=None, matching_strategy=None, typo_tolerance=None, metric=None, weights=None)` - **TypeScript** — `search(params)` - **Go** — `Search(query, objectType)` **Parameters:** `query`, `type`, `limit=None`, `facets=None`, `highlight=False`, `filters=None`, `matching_strategy=None`, `typo_tolerance=None`, `metric=None`, `weights=None` **Example** ```python hits = relata.search("shell company", "IntelChunk", limit=10, highlight=True, metric="cosine", weights=[0.0, 0.5, 0.5]) # [graph, bm25, vector] ``` ## `SearchClient` ### `query(namespace, text=None, match_column='*', rank_by=None, filters=None, limit=20, include_attributes=None, consistency=None, compute_attributes=None, purpose=None)` query(namespace, text=None, match_column='*', rank_by=None, filters=None, limit=20, include_attributes=None, consistency=None, compute_attributes=None, purpose=None)` - **Python** — `client.searchclient.query(namespace, text=None, match_column='*', rank_by=None, filters=None, limit=20, include_attributes=None, consistency=None, compute_attributes=None, purpose=None)` - **Go** — `Query(req)` **Parameters:** `namespace`, `text=None`, `match_column='*'`, `rank_by=None`, `filters=None`, `limit=20`, `include_attributes=None`, `consistency=None`, `compute_attributes=None`, `purpose=None` {/* END GENERATED */} ============================================================================== # Streaming & Transparency Log URL: https://relatadb.dev/docs/sdks/methods/streaming ============================================================================== # Streaming & Transparency Log `StreamingClient.watch` is a live, ACL-filtered SSE change feed; `query_arrow_raw` streams Arrow IPC. `LogClient` is the hash-chained transparency log (append + leaf/head reads) — the tamper-evident audit spine. {/* BEGIN GENERATED — overwritten by scripts/gen_sdk_methods_docs.py */} > **8 methods** across this domain. Signatures, parameters (incl. keyword-only tunable defaults), and the three SDK spellings are ported from the live SDK source; flagship methods carry a hand-written *When to use* + example. ## `LogClient` ### `append(data)` append(data)` - **Python** — `client.logclient.append(data)` - **TypeScript** — `append(data)` - **Go** — `Append(data)` **Parameters:** `data` ### `head()` head()` - **Python** — `client.logclient.head()` - **TypeScript** — `head()` - **Go** — `Head()` ### `load_leaves(since=0, limit=1000)` load_leaves(since=0, limit=1000)` - **Python** — `client.logclient.load_leaves(since=0, limit=1000)` - **TypeScript** — `loadLeaves(opts)` - **Go** — `LoadLeaves()` **Parameters:** `since=0`, `limit=1000` ## `StreamingClient` ### `watch(sql, purpose, reconnect=True)` **When to use.** Live, ACL-filtered SSE change feed on a type — reactive pipelines. - **Python** — `client.streamingclient.watch(sql, purpose, reconnect=True)` - **TypeScript** — `watch(sql, purpose, opts)` - **Go** — `Watch(sql)` **Parameters:** `sql`, `purpose`, `reconnect=True` **Example** ```python for evt in StreamingClient.from_client(relata).watch("Person"): print(evt) ``` ```typescript for await (const evt of relata.streaming.watch("Person")) { console.log(evt); } ``` ### `alerts(reconnect=True)` alerts(reconnect=True)` - **Python** — `client.streamingclient.alerts(reconnect=True)` - **TypeScript** — `alerts(opts)` - **Go** — `Alerts()` **Parameters:** `reconnect=True` ### `query_arrow_raw(sql, purpose=None)` query_arrow_raw(sql, purpose=None)` - **Python** — `client.streamingclient.query_arrow_raw(sql, purpose=None)` - **TypeScript** — `queryArrowRaw(sql, opts)` - **Go** — `QueryArrowRaw(sql)` **Parameters:** `sql`, `purpose=None` ### `query_rows(sql, purpose=None)` query_rows(sql, purpose=None)` - **Python** — `client.streamingclient.query_rows(sql, purpose=None)` - **TypeScript** — `queryRows(sql, opts)` - **Go** — `QueryRows(sql)` **Parameters:** `sql`, `purpose=None` ### `watch_stream(type, since=None, reconnect=True)` watch_stream(type, since=None, reconnect=True)` - **Python** — `client.streamingclient.watch_stream(type, since=None, reconnect=True)` **Parameters:** `type`, `since=None`, `reconnect=True` {/* END GENERATED */} ============================================================================== # Vectors & Embeddings URL: https://relatadb.dev/docs/sdks/methods/vectors ============================================================================== # Vectors & Embeddings `hybrid_search` fuses graph + BM25 + vector channels in one governed call. `embed` produces vectors via the configured embedder; `embed_image`/`embed_face`/`embed_audio`/`embed_video` cover the media modalities (ADR-0276). A real media vector requires an embedding sidecar (`RELATA_ACCEL_ENDPOINT`). {/* BEGIN GENERATED — overwritten by scripts/gen_sdk_methods_docs.py */} > **12 methods** across this domain. Signatures, parameters (incl. keyword-only tunable defaults), and the three SDK spellings are ported from the live SDK source; flagship methods carry a hand-written *When to use* + example. ## `RelataClient` ### `face_search(gallery_id, embedding, k=10, threshold=0.7, purpose=None)` face_search(gallery_id, embedding, k=10, threshold=0.7, purpose=None)` - **Python** — `client.relataclient.face_search(gallery_id, embedding, k=10, threshold=0.7, purpose=None)` - **Go** — `FaceSearch(galleryID, embedding)` **Parameters:** `gallery_id`, `embedding`, `k=10`, `threshold=0.7`, `purpose=None` ### `match_pdq(corpus_id, query_hash, threshold=0.9, purpose=None)` match_pdq(corpus_id, query_hash, threshold=0.9, purpose=None)` - **Python** — `client.relataclient.match_pdq(corpus_id, query_hash, threshold=0.9, purpose=None)` - **Go** — `MatchPdq(corpusID, queryHash)` **Parameters:** `corpus_id`, `query_hash`, `threshold=0.9`, `purpose=None` ### `similar_image(media_ref, threshold=None, index=None, purpose=None)` similar_image(media_ref, threshold=None, index=None, purpose=None)` - **Python** — `client.relataclient.similar_image(media_ref, threshold=None, index=None, purpose=None)` - **Go** — `SimilarImage(mediaRef)` **Parameters:** `media_ref`, `threshold=None`, `index=None`, `purpose=None` ## `VectorClient` ### `embed(text, model=None)` **When to use.** Produce a vector for caller-side caching or custom ANN; uses the configured embedder. - **Python** — `client.vectorclient.embed(text, model=None)` - **TypeScript** — `embed(text, opts)` - **Go** — `Embed(text, model)` **Parameters:** `text`, `model=None` **Example** ```python vec = VectorClient.from_client(relata).embed("shell company filings") ``` ### `hybrid_search(object_type, query_text=None, k=10, purpose=None, rerank=False, metric=None, weights=None)` **When to use.** Explicit hybrid fusion over a type with a query text (server embeds the text). - **Python** — `client.vectorclient.hybrid_search(object_type, query_text=None, k=10, purpose=None, rerank=False, metric=None, weights=None)` - **TypeScript** — `hybridSearch(objectType, opts, number)` - **Go** — `HybridSearch(objectType, queryText)` **Parameters:** `object_type`, `query_text=None`, `k=10`, `purpose=None`, `rerank=False`, `metric=None`, `weights=None` **Example** ```python from relata import VectorClient vc = VectorClient.from_client(relata) hits = vc.hybrid_search("IntelChunk", "sanctions evasion", limit=20) ``` ### `embed_audio(bytes_b64, model=None)` embed_audio(bytes_b64, model=None)` - **Python** — `client.vectorclient.embed_audio(bytes_b64, model=None)` - **TypeScript** — `embedAudio(bytesB64, opts)` - **Go** — `EmbedAudio(bytesB64, model)` **Parameters:** `bytes_b64`, `model=None` ### `embed_batch(texts, model=None)` embed_batch(texts, model=None)` - **Python** — `client.vectorclient.embed_batch(texts, model=None)` - **TypeScript** — `embedBatch(texts, opts)` - **Go** — `EmbedBatch(texts, model)` **Parameters:** `texts`, `model=None` ### `embed_face(bytes_b64, model=None)` embed_face(bytes_b64, model=None)` - **Python** — `client.vectorclient.embed_face(bytes_b64, model=None)` - **TypeScript** — `embedFace(bytesB64, opts)` - **Go** — `EmbedFace(bytesB64, model)` **Parameters:** `bytes_b64`, `model=None` ### `embed_image(bytes_b64, model=None)` embed_image(bytes_b64, model=None)` - **Python** — `client.vectorclient.embed_image(bytes_b64, model=None)` - **TypeScript** — `embedImage(bytesB64, opts)` - **Go** — `EmbedImage(bytesB64, model)` **Parameters:** `bytes_b64`, `model=None` ### `embed_video(bytes_b64, model=None)` embed_video(bytes_b64, model=None)` - **Python** — `client.vectorclient.embed_video(bytes_b64, model=None)` - **TypeScript** — `embedVideo(bytesB64, opts)` - **Go** — `EmbedVideo(bytesB64, model)` **Parameters:** `bytes_b64`, `model=None` ### `knn_search(object_type, embedding_slot, query_embedding, k=10, ef_search=None, purpose=None)` knn_search(object_type, embedding_slot, query_embedding, k=10, ef_search=None, purpose=None)` - **Python** — `client.vectorclient.knn_search(object_type, embedding_slot, query_embedding, k=10, ef_search=None, purpose=None)` - **TypeScript** — `knnSearch(objectType, embeddingSlot, queryEmbedding, opts)` - **Go** — `KNNSearch(objectType, embeddingSlot, queryEmbedding, k)` **Parameters:** `object_type`, `embedding_slot`, `query_embedding`, `k=10`, `ef_search=None`, `purpose=None` ### `similar_to(object_type, reference_id, k=10, purpose=None)` similar_to(object_type, reference_id, k=10, purpose=None)` - **Python** — `client.vectorclient.similar_to(object_type, reference_id, k=10, purpose=None)` - **TypeScript** — `similarTo(objectType, referenceId, opts)` - **Go** — `SimilarTo(objectType, referenceID)` **Parameters:** `object_type`, `reference_id`, `k=10`, `purpose=None` {/* END GENERATED */} ============================================================================== # SDK guide URL: https://relatadb.dev/docs/sdks/overview ============================================================================== # SDK guide RelataDB ships **three published consumer SDKs** — **Python, TypeScript, and Go** — held in 3-way domain-module parity by `scripts/check_sdk_parity.py` and mirrored to `github.com/relatadb/sdk-{python,typescript,go}` on every develop push. This page routes you to the right one, shows what's covered, and tracks the roadmap. > **Rust?** There is an internal/reference Rust client (`crates/relata-sdk-rust`) used by the server binary, the tray app, the test harness, and `relata-bench`. It is first-party and load-bearing but **not published as a consumer SDK** — the three SDKs below are the supported app-facing surface. ## SDK locations | Language | Package | Status | |---|---|---| | **Python** | `relata-sdk` on PyPI | **Reference implementation** — fullest surface | | **TypeScript** | `@zysec-ai/relata-sdk` on npm | At parity with Python (typed clients) | | **Go** | `github.com/relatadb/sdk-go/v2` | At parity with Python (typed clients) | Quickstart pages: [Python](/docs/sdks/python) · [TypeScript](/docs/sdks/typescript) · [Go](/docs/sdks/go). ## Quick examples — what "covered" looks like Python is the reference implementation; **TypeScript and Go mirror the same verbs** (see the parity matrix below for exact identifiers, and your SDK's quickstart for the local spelling). Every call declares a `purpose` and runs through the governed path — ACL, cell masking, audit. ### Connect & query ```python from relata import RelataClient with RelataClient("http://localhost:9090", purpose="analytics") as relata: # Raw SQL — QueryResult is iterable for row in relata.query("SELECT * FROM Person WHERE name LIKE 'Ahmed%' LIMIT 10"): print(row["name"]) # Fluent builder + bi-temporal time-travel + provenance res = (relata.select("Person") .where("nationality = 'IN'") .as_of("2025-01-01T00:00:00Z") .with_provenance() .limit(5) .execute()) ``` ### Hybrid search ```python hits = relata.search("shell company", "IntelChunk", limit=10, highlight=True) for h in hits.hits: print(h.score, h.highlights) ``` ### Ingest (bulk / CSV / streaming / OTLP / document) ```python from relata import IngestClient ing = IngestClient.from_client(relata) ing.bulk("Person", [{"name": "Alice", "email": "a@x.io"}], on_conflict="upsert") # upsert | skip | error ing.ingest_csv("people.csv", "Person") # typed CSV loader ing.otlp_traces(payload) # OTLP traces / logs / metrics ing.ingest_document(source="report.pdf", content=blob, auto_chunk=True) ``` ### Identity resolution & entity lifecycle ```python from relata import IdentityClient idc = IdentityClient.from_client(relata) cluster = relata.resolve_identities("alice@x.io") # → unified entity + aliases relata.fuse_identities(id_a, id_b) # ontological merge idc.erase_subject("alice@x.io", reason="gdpr-art17") # governed right-to-erasure ``` ### Agent memory — 10 cognitive verbs ```python from relata import Memory with Memory("http://localhost:9090", purpose="agent-notes") as m: mid = m.add("Alice prefers dark mode") # remember for hit in m.search("ui preferences", top_k=5): # recall (hybrid + recency) print(hit["content"]) m.forget(mid) # governed retention retract, not a hard delete ``` ### Graph + intelligence operators ```python path = relata.graph_dijkstra("Person", "p-1", "p-9") # shortest path ring = relata.graph_scc("Transaction") # fraud-ring detection ubo = relata.beneficial_ownership_chain("ShellCo") # intel operator ``` ### Governance, audit & A2A (typed clients) ```python from relata.audit import AuditClient from relata.a2a import A2AClient audit = AuditClient.from_client(relata) print(audit.count()) # chain_valid + entry count a2a = A2AClient.from_client(relata) print(a2a.agent_card()) # discover the agent task = a2a.submit_task({"name": "enrich", "input": {...}}) # agent-to-agent task ``` ### Streaming (governed SSE change feed) ```python from relata import StreamingClient sc = StreamingClient.from_client(relata) for evt in sc.watch("Person"): # live, ACL-filtered change feed print(evt) ``` ### Point an agent at it (MCP + framework adapters) ```python from relata import McpClient mcp = McpClient.from_client(relata) # 68 typed tool wrappers; also call_tool(name, args) # Or drop Relata in as governed memory for an existing framework: # from relata_adapters.langchain import RelataMemory # LangChain / LlamaIndex / CrewAI / # from relata_adapters.crewai import RelataStorage # AutoGen / AG2 / Pydantic-AI / # from relata_langgraph import RelataCheckpointer # smolagents / LangGraph ``` See the [Python](/docs/sdks/python), [TypeScript](/docs/sdks/typescript), and [Go](/docs/sdks/go) quickstarts for install + run instructions, and the [AI in RelataDB](/docs/concepts/ai-in-relatadb) page for the full agent/RAG loop. ## Feature parity matrix (verified against source) | Feature | Python | TypeScript | Go | Notes | |---|:---:|:---:|:---:|---| | Core client (`query`/`health`/`status`) | ✅ | ✅ | ✅ | HTTP | | **Parameterized queries** (`$N` server-side binding) | ✅ `query_params` + `aquery_params` | ✅ `queryWithParams` | ✅ `QueryWithParams` | `?` placeholders auto-rewritten in Python | | **Text embedding** (`/embed`, `/embed/batch`) | ✅ `VectorClient.embed` + `embed_batch` | ✅ `VectorClient.embed` + `embedBatch` | ✅ `VectorClient.Embed` + `EmbedBatch` | Server endpoint always available (CPU fallback); GPU sidecar via `RELATA_ACCEL_ENDPOINT` | | **Media embedding** (`/embed/{image,face,audio,video}`) | ✅ `embed_image/face/audio/video` | ✅ `embedImage/Face/Audio/Video` | ✅ `EmbedImage/Face/Audio/Video` | CLIP / ArcFace / CLAP; 503 when active embedder doesn't support media | | Fluent `QueryBuilder` | ✅ | ✅ | ✅ | | | `SearchBuilder` (`/search`) | ✅ | ✅ | ✅ | Facets, highlight, filter, fuzzy preset | | `Memory` cognitive verbs | ✅ **10 + `add_batch`** | ✅ **10 + `add_batch`** | ✅ **10 + `add_batch`** | 10 cognitive verbs + `add_batch` (batch convenience wrapper). See the [verb matrix](#memory-cognitive-verb-matrix) below. | | Typed v1.1 clients (`fromClient`) | ✅ **16** | ✅ **16** | ✅ **16** | | | Typed response models | ✅ Pydantic | ✅ interfaces | ✅ structs | | | RFC 7807 `ProblemDetails` errors | ✅ | ✅ **13 classes** | ✅ | | | `X-Request-ID` per attempt | ✅ | ✅ | ✅ | UUIDv7 | | Retry on 502/503/504 | ✅ configurable | ✅ configurable | ✅ configurable + `Retry-After` | | | Multi-tenant (`X-Organization-Id`) | ✅ | ✅ | ✅ | | | Delegation (`X-Acting-As` / `X-Delegated-By`) | ✅ | ✅ | ✅ | | | Sync + async mirrors | ✅ both | async-native | ctx-based | | | Streaming (SSE watch/alerts) | ✅ `StreamingClient` | ✅ `StreamingClient` | ✅ `StreamingClient` | | | Arrow `RecordBatch` / `Table` | ✅ `query_arrow` + `query_flight` (pyarrow) | ✅ `ArrowFlightTransport` (apache-arrow optional peer) | ✅ `QueryFlight` (arrow/go) | Arrow IPC + Flight `DoGet` in all three | | Agent-framework adapters | ✅ **7** | ✅ **3** | — | Python: LangChain/LlamaIndex/CrewAI/AutoGen(AG2)/Pydantic-AI/smolagents/LangGraph. TS: LangChain/LlamaIndex/LangGraph. Go/Rust: legitimately `—` (no idiomatic ecosystem to adapt). | | Typed MCP tool wrappers | ✅ **68** | ✅ **68** | ✅ **68** | Each SDK ports the full union of MCP tools; Rust (internal) ships 58. See [MCP tools reference](/docs/reference/mcp-tools). | | **Bi-temporal travel helper** (`as_of` + `with_provenance`) | ✅ | ✅ | ✅ | | | **Graph traversal DSL** | ✅ `paths_between` + `graph_*` | ✅ `graph()` + `graph_*` | ✅ `PathsBetween` + `Graph*` | All 3 ship 10+ graph operators; TS adds a fluent `graph()` DSL helper. See [Graph analytics](/docs/reference/graph-analytics) | | **Bulk ingest streaming** (`ingest_iter`) | ✅ | ✅ | ✅ | | ## Typed v1.1 client inventory Each typed client wraps a server-side domain surface. Construct with `.from_client(client)` (Python/Go) or `new (client)` (TS). Every client inherits auth, tenant, purpose, and retry config. | Client | Surface | Python | TS | Go | |---|---|:---:|:---:|:---:| | `GovernanceClient` | Rules, retention (holds + WORM), breakglass, alerts, DSAR | ✅ | ✅ | ✅ | | `McpClient` | 68 typed MCP tool wrappers + generic `call_tool` | ✅ | ✅ | ✅ | | `A2AClient` | A2A tasks + LangGraph checkpoints + agent card | ✅ | ✅ | ✅ | | `AuditClient` | Audit entries (filtered/paginated) + signed receipts + PDF export | ✅ | ✅ | ✅ | | `IdentityClient` | Identity label/uncertainty + lookup tables + ERASE SUBJECT | ✅ | ✅ | ✅ | | `ObjectClient` | Typed upsert + batch via `/ingest?object_type=` | ✅ | ✅ | ✅ | | `IngestClient` | Bulk NDJSON + CSV + media status | ✅ | ✅ | ✅ | | `VectorClient` | KNN + hybrid search + similar-to (SQL-backed) | ✅ | ✅ | ✅ | | `S3Client` | S3 protocol door wrapper | ✅ | ✅ | ✅ | | `SystemClient` | LLM config + test + jobs status | ✅ | ✅ | ✅ | | `StreamingClient` | NDJSON row streams + SSE consumers (watch/alerts) + Arrow IPC | ✅ | ✅ | ✅ | | `TenantAdminClient` | Tenant CRUD + quota + sharing agreements | ✅ | ✅ | ✅ | | `BackupClient` | Backup create / list / restore | — | ✅ | ✅ | | `TokenClient` | Token create / check / revoke / stats | — | ✅ | ✅ | | `LogClient` | Structured log query / tail | — | ✅ | ✅ | | `RulesClient` | Detection-rule CRUD + Sigma import | ✅ (on Gov) | ✅ (on Gov) | ✅ (on Gov) | ## Memory cognitive-verb matrix | Verb | HTTP | Python | TS | Go | |---|---|:---:|:---:|:---:| | `add` | `POST /memory/remember` | ✅ | ✅ | ✅ | | `add_batch` | `POST /memory/remember/batch` | ✅ | ✅ | ✅ | | `search` (recall) | `GET /memory/recall` | ✅ | ✅ | ✅ | | `get` (recognize) | `GET /memory/recognize/:id` | ✅ | ✅ | ✅ | | `update` (consolidate) | `POST /memory/consolidate` | ✅ | ✅ | ✅ | | `forget` | `DELETE /memory/forget/:id` | ✅ | ✅ | ✅ | | `associate` | `POST /memory/associate` | ✅ | ✅ | ✅ | | `episodes` | `GET /memory/episodes` | ✅ | ✅ | ✅ | | `justify` | `GET /memory/justify/:id` | ✅ | ✅ | ✅ | | `resolve` | `POST /memory/resolve/:id` | ✅ | ✅ | ✅ | | `summarise` | `POST /memory/summarise` | ✅ | ✅ | ✅ | ## Platform capability coverage Coverage is computed from the capability matrix in `sdks/COVERAGE.md` (the CI-gated canonical tracker in the source repo, verified by `scripts/check_sdk_parity.py`). 1 partial = ½. | SDK | Coverage | Strengths | |---|---|---| | **Python** | **99.6%** | Reference implementation; governance, identity, 76 canonical types, detection rules (all 8), ontology, streaming, all typed clients, 7 framework adapters, SPARQL, cluster ops, sessions, OTLP ingest | | **TypeScript** | **99.6%** | Types, rules (all 8), ontology, links, identity helpers, 16 typed clients, 13 error classes, 3 framework adapters, SPARQL, cluster ops, sessions, OTLP ingest | | **Go** | **99.6%** | Types, rules (all 8), ontology, links, identity helpers, SSE streaming, SPARQL, cluster ops, sessions, OTLP ingest | ### The one shared gap A single capability is `⚠️ partial` across all three published SDKs (and Rust): - **KNN by caller-supplied embedding** — `knn_search`/`knnSearch`/`KNNSearch` emits `ORDER BY <=> '[…]'`, a pgvector-ism the server parser rejects (`ORDER BY` only takes a bare column). Hybrid search (`HYBRID_SEARCH`) and reference-row similarity (`SIMILAR TO`) are unaffected and fully ✅. Tracked pending a server-side vector-literal grammar. The remaining differentials are minor: `SIMILAR_IMAGE` shipped to Python/TypeScript/Go but not yet to the internal Rust client; the `ClientPool` connection-pool helper is TypeScript+Rust only. Every other capability in the 22-section matrix is green across all three published SDKs. ### Server-only surfaces (raw HTTP, no SDK wrapper) These endpoints are reachable via raw HTTP but deliberately not wrapped by the typed SDKs: - **Admin**: `reindex`, `rotate-dek`, `dashboard`, `system`, `logs` (operator surfaces, run via `relata` CLI or the admin dashboard) - **Config**: `GET /config` (operator introspection) - **Attestation**: `GET /attestation` (supply-chain verification, run via `cosign verify-blob`) ## Runnable example inventory Each SDK ships a parallel set of self-contained examples. | Capability | Python | TypeScript | Go | |---|:---:|:---:|:---:| | Basic query / quickstart | ✅ `basic_query.py` | ✅ `basic-query.ts` | ✅ `basic/` | | Ingest | ✅ `ingest.py` | ✅ `ingest.ts` | ✅ `ingest/` | | Advanced query / Arrow | ✅ `advanced_query.py` | ✅ `advanced-query.ts` | ✅ `advanced_query/` | | Governance | ✅ `governance.py` | ✅ `governance.ts` | ✅ `governance/` | | Memory (cognitive verbs) | ✅ `memory_quickstart.py` | ✅ `memory-quickstart.ts` | ✅ `memory_quickstart/` | | Multi-tenant | ✅ `multi_tenant.py` | ✅ `multi-tenant.ts` | ✅ `multi_tenant/` | | Ephemeral server | ✅ `ephemeral_server.py` | ✅ `ephemeral-server.ts` | ✅ `ephemeral_server/` | | GraphQL | ✅ `graphql.py` | ✅ `graphql.ts` | ✅ `graphql/` | | Graph algorithms | ✅ `graph_traversal.py` | ✅ `graph-algorithms.ts` | ✅ `graph_algorithms/` | | Intelligence operators | ✅ `intelligence.py` | ✅ `intelligence.ts` | ✅ `intelligence/` | | Multi-search | ✅ `multi_search.py` | ✅ `multi-search.ts` | ✅ `multi_search/` | | Parameterized queries | ✅ `parameterized.py` | ✅ `parameterized.ts` | ✅ `parameterized/` | | Lookup tables | ✅ `lookups.py` | ✅ `lookups.ts` | ✅ `lookups/` | | Streaming (SSE watch + log) | ✅ `streaming.py` | ✅ `streaming.ts` | ✅ `streaming/` | | A2A (tasks + checkpoints) | ✅ `a2a.py` | ✅ `a2a.ts` | ✅ `a2a/` | | Dedup tokens (replay defence) | ✅ `tokens.py` | ✅ `tokens.ts` | ✅ `tokens/` | | Tenant admin (lifecycle) | ✅ `tenant_admin.py` | ✅ `tenant-admin.ts` | ✅ `tenant_admin/` | | Bi-temporal (`AS OF` + `WITH PROVENANCE`) | ✅ `bitemporal.py` | ✅ `bitemporal.ts` | ✅ `bitemporal/` | | Audit | ✅ `audit.py` | ✅ `audit.ts` | ✅ `audit/` | | Analytics (SQL exploration) | ✅ `analytics.py` | ✅ `analytics.ts` | ✅ `analytics/` | | Jobs & workflows | ✅ `jobs_workflows.py` | ✅ `jobs-workflows.ts` | ✅ `jobs_workflows/` | | Face search (multimodal) | ✅ `face_search.py` | ✅ `face-search.ts` | ✅ `face_search/` | | Investigation (paths_between) | ✅ `investigation.py` | ✅ `investigation.ts` | ✅ `investigation/` | Each example file is self-contained — connect, run, print, exit. Run from the language's `sdks//` directory: ```bash # Python RELATA_TOKEN=secret python -m examples.graphql # TypeScript (Node 23+ / Deno / Bun) node --experimental-strip-types examples/graphql.ts # Go go run ./examples/graphql -url http://localhost:9090 -token $RELATA_TOKEN ``` ## Roadmap | Work item | Deliverable | Status | |---|---|---| | OpenAPI contract + drift gate | Server `ROUTES` table drives `docs/.../api-reference.md` via `gen_api_reference.py`; `check_docs.sh` fails on drift | ✅ Done | | SDK contract-test suite | Shared `fixtures.yaml` wire contract consumed by `sdks/contract-tests/{python,typescript,go}/` (hermetic, no live server) + `run_sdk_contract_tests.py` (live-server, all 4 SDKs) | ✅ Done | | 3-way domain-module parity | `check_sdk_parity.py` holds Python/TypeScript/Go to the same domain modules on every PR | ✅ Done | | 68 typed MCP wrappers per SDK | Each of Python/TS/Go ports the full union of MCP tools (22 original + 46 from Rust's 58-tool set) | ✅ Done | | Sessions / OTLP / cluster ops / SPARQL | All four capability families wrapped across all three published SDKs | ✅ Done | | KNN caller-supplied-embedding | Awaits a server-side vector-literal grammar; hybrid search + `SIMILAR TO` cover the common case today | Open | | Offline query plan cache | SDK-side cache of `sha256(sql)` → plan verdict | P2 | | OpenTelemetry auto-instrumentation | Every SDK call auto-emits an OTel span | P3 | | Java/Kotlin SDK | JVM SDK for enterprise/Spring Boot integrations (pgwire/JDBC wire-driver guides exist today) | P3 | | C# / .NET SDK | .NET SDK for Microsoft-ecosystem customers | P3 | ## Minimum public SDK example shape Relata is the governed memory layer for AI agents, so every SDK shows the **agent-memory loop**, not just a SQL call: connect → `remember` (with purpose + provenance) → `recall` (hybrid retrieval, optionally `AS OF`) → `justify` (provenance/audit chain) → handle errors → close. The verbs are reached the same way in every language: `POST /memory/{remember|recall|recognize|justify|consolidate|forget}` (or the matching MCP tool). See the [query cookbook](/docs/reference/query-cookbook) → Part 2 for canonical request bodies. ## See also - [Python](/docs/sdks/python) · [TypeScript](/docs/sdks/typescript) · [Go](/docs/sdks/go) quickstarts - [Agent Memory](/docs/concepts/agent-memory) — the 10 cognitive verbs - [AI in RelataDB](/docs/concepts/ai-in-relatadb) — the full agent/RAG loop - [Error reference](/errors) — deep-linkable RFC 7807 error codes ============================================================================== # Python SDK URL: https://relatadb.dev/docs/sdks/python ============================================================================== # Python SDK `pip install relata-sdk` — Python 3.11+. Hard deps: `httpx`, `pydantic`. Async extras: `pyarrow`, `pandas`, `langgraph`, `boto3`/`aiobotocore` (all optional — install only what you use). The SDK ships **sync + async mirrors of every client** (42 classes total) so the same code shape works in scripts and in `asyncio` servers. > See the [SDK overview](/docs/sdks/overview) for the cross-language parity matrix. This page is the Python capability catalog. ## Quickstart ```bash pip install relata-sdk ``` ```python from relata import RelataClient with RelataClient("http://localhost:9090", purpose="analytics") as client: client.query("INSERT INTO Person (_pk, name, email) VALUES ('p1', 'Alice', 'a@x.com')") for row in client.query("SELECT * FROM Person LIMIT 5"): print(row["name"], row["email"]) ``` Async is the same surface with an `a`-prefix: `await client.aquery(...)`, `await client.asearch(...)`, etc. Use `async with RelataClient(...)` for scoped lifecycles. ## The query surface | Path | Method | Returns | |---|---|---| | SQL | `query(sql, purpose=None, dialect=None)` / `aquery` | `QueryResult` (iterable) | | Parameterized | `query_params(sql, params, purpose=None)` / `aquery_params` | `QueryResult` — `?` auto-rewrites to `$N` | | Arrow IPC | `query_arrow(sql, purpose=None)` | `pyarrow.Table` — zero-copy | | Arrow Flight (gRPC) | `query_flight(sql, flight_endpoint=None, ...)` / `aquery_flight` | `pyarrow.Table` | | GraphQL | `graphql(query, variables=None, operation_name=None)` / `agraphql` | `dict` — `variables` bound server-side (#3260) | | SPARQL | `sparql(query)` | `dict` | | Cypher | any `MATCH`-prefixed string via `query()` | auto-routed, governed | | GQL (ISO 39075) | `query(stmt, dialect="gql")` | header-selected, governed (#3265) | | Fluent builder | `client.select(*cols).where(...).limit(10).execute()` / `.aexecute()` | `QueryResult` | ```python from relata import select result = (select("*").from_("Person") .where("age > $1").where_param("age > $1", 25) .order_by("name").limit(10) .purpose("analytics").execute()) ``` ## Typed domain clients Every domain client has sync + async mirrors and a `.from_client(client)` factory that inherits auth/tenant/purpose/timeout: ```python from relata import ( GovernanceClient, IdentityClient, ObjectClient, IngestClient, VectorClient, SearchClient, StreamingClient, AuditClient, TenantAdminClient, BackupClient, TokenClient, LogClient, SystemClient, A2AClient, McpClient, Namespace, ) ``` | Client | Key methods | Cross-ref | |---|---|---| | `GovernanceClient` | rules CRUD, Sigma import, retention/WORM/legal-holds, breakglass, alerts, DSAR | [Detection Rules](/docs/guides/detection-rules) | | `IdentityClient` | `label`, `record_uncertainty`, `register_lookup`/`list_lookups`/`invoke_lookup`, `erase_subject` | [Identity](/docs/concepts/identity) (active learning) | | `ObjectClient` | `upsert`, `typed_upsert`, `batch_upsert`, `get`, `delete` | — | | `IngestClient` | `bulk`, `bulk_csv`, `ingest_auto`, `ingest_cdr`, `otlp_traces/logs/metrics`, `ingest_iter` | [Ingestion](/docs/guides/ingestion) | | `VectorClient` | `knn_search`, `hybrid_search`, `similar_to`, `embed`/`embed_batch` + `embed_image/face/audio/video` | [Hybrid Search](/docs/concepts/hybrid-search) | | `SearchClient` | typed `/search` JSON door: `query(namespace, text=..., rank_by=..., filters=..., limit=...)` | [Search reference](/docs/reference/search) | | `StreamingClient` | `query_rows` (NDJSON), `query_arrow_raw`, `watch`/`watch_stream` (SSE), `alerts` (SSE) | — | | `AuditClient` | `count`, `entries(...)`, `find_by_request_id`, `sign_receipt`, `export_pdf` → bytes | — | | `TenantAdminClient` | tenant CRUD, quota, sharing, platform usage/license | [Multi-Tenancy](/docs/guides/multi-tenancy) | | `BackupClient` | `create`, `list`, `restore`, `restore_status`, `compact`, `wait_for_restore` | [Backup & Restore](/docs/guides/backup-restore) | | `TokenClient` | `remember`, `check`, `revoke`, `stats` (dedup tokens) | — | | `LogClient` | `append`, `head`, `load_leaves` (integrity log) | — | | `SystemClient` | LLM config/test, jobs/workflows, feeds, notifications, pipelines | — | | `A2AClient` | `submit_task`, `get_task`, checkpoints, `agent_card` | — | | `McpClient` | `initialize`, `list_tools`, `call_tool` + 68 typed tool wrappers | [MCP Tools](/docs/reference/mcp-tools) | | `Namespace` | `client.namespace("Document")` → `query/write/get/delete_all/branch_from` | [Search reference](/docs/reference/search) | ## Vectors & embeddings ```python vc = client.vector_client # or: VectorClient.from_client(client) # Pure KNN over a named slot vc.knn_search("Document", "embedding", [0.1, ...], k=10, ef_search=200) # Hybrid: BM25 + vector + graph, RRF-fused vc.hybrid_search("Document", query_text="graph retrieval", k=10, rerank=True, weights=[0.2, 0.5, 0.3]) # Embedding (6 modalities) — uses server's CPU lexical default or GPU sidecar emb = vc.embed("Alice Smith") # → {embedding, model, dim} vc.embed_image(base64_bytes) # CLIP vc.embed_face(base64_bytes) # ArcFace vc.embed_audio(base64_bytes) # CLAP vc.embed_video(base64_bytes) # CLIP keyframe ``` ## Graph & intelligence operators All on `RelataClient` directly — 10 graph algorithms + 10 AML/financial + 3 maritime: ```python client.graph_pagerank("Person", damping=0.85, max_iter=20) client.graph_shortest_path("alice-id", "bob-id", max_hops=5) client.graph_community("Person") # Financial intelligence client.sanctions_screen("Acme Holdings", threshold=0.85) client.beneficial_ownership_chain("Acme Holdings", max_depth=6) client.crypto_trace("0xabc...", purpose="compliance") # Maritime client.vessel_track(mmsi=123456789, window_secs=86400) client.dark_fleet_detect(max_gap_hours=48) ``` See [Graph Analytics](/docs/reference/graph-analytics) for the algorithm matrix and the SQL TVF / `gds.*` / `traverse.*` surfaces. ## Agent memory — 10 cognitive verbs + recall-quality knobs ```python from relata import Memory mem = Memory("http://localhost:9090", bearer_token="", purpose="agent") mid = mem.add("Alice prefers dark mode", confidence=0.9, memory_class="semantic") # retrieval-quality operators — tune what comes back results = mem.search( "ui preferences", top_k=10, min_confidence=0.6, # CONFIDENCE floor recency_half_life_secs=259200, # 3-day decay (RECENCY) budget_tokens=1500, # hard prompt budget (BUDGET) cancel_threshold=0.92, # stop on a great match (CANCEL_WHEN) ) detail = mem.search_detailed(...) # exposes recall_cost_tokens + cancelled ``` The full verb set: `add`, `add_batch`, `search`, `search_detailed`, `get`, `update`, `forget`, `associate`, `episodes`, `justify`, `resolve`, `summarise`. See [Agent memory reference](/docs/reference/agent-memory) (recall knobs + the 5 operators). ## Ecosystem (Python-only) | Extra | Install | Surface | |---|---|---| | **7 framework adapters** | `relata_adapters` (ships with the package) | `RelataMemory` for LangChain / LlamaIndex / CrewAI / AutoGen(+AG2) / Pydantic-AI / smolagents — duck-typed, install only your framework | | **LangGraph checkpointer** (the 7th adapter) | `pip install relata-sdk[langgraph]` | `RelataCheckpointer` + `AsyncRelataCheckpointer` (real `BaseCheckpointSaver` subclasses; persist via the governed A2A door) | | **IPython / Jupyter magic** | `pip install relata-sdk[ipython]` | `%%relata --purpose analytics` cell magic → results render as a pandas DataFrame | | **S3 door helper** | `pip install relata-sdk[s3]` | `S3Client.boto3()` / `AsyncS3Client.aio()` / `S3Client.httpx()` — returns a configured boto3/aiobotocore/httpx client pointed at Relata's S3 door | ```python # Auto-detect which framework is installed and return the right adapter from relata_adapters.registry import get_memory_adapter Adapter = get_memory_adapter() # LangChain/LlamaIndex/CrewAI/... or None mem = Adapter(relata_memory_backend) if Adapter else None ``` ## Authentication & multi-tenant ```python client = RelataClient( "http://localhost:9090", bearer_token="", purpose="analytics", tenant="org-acme", # X-Relata-Tenant-Id on every request acting_as="user-42", # X-Acting-As (delegation) delegated_by="admin-1", # X-Delegated-By timeout=30.0, max_retries=3, admin_base_url="http://admin.internal:9090", # /admin/* + /platform/* zero-trust split ) ``` ## Examples The SDK ships ~25 runnable examples in `sdks/python/examples/`. Run any with `python -m examples.`: ```bash RELATA_TOKEN=secret python -m examples.basic_query # minimal connect + SELECT RELATA_TOKEN=secret python -m examples.ingest # bulk + CSV ingest RELATA_TOKEN=secret python -m examples.advanced_query # filter + aggregate + Arrow RELATA_TOKEN=secret python -m examples.governance # PURPOSE + audit + types RELATA_TOKEN=secret python -m examples.memory_quickstart # add / search / forget RELATA_TOKEN=secret python -m examples.multi_tenant # org isolation RELATA_TOKEN=secret python -m examples.intelligence # sanctions / UBO / crypto / convoy / DNS RELATA_TOKEN=secret python -m examples.face_search # FACE_SEARCH operator RELATA_TOKEN=secret python -m examples.streaming # SSE watch + transparency log RELATA_TOKEN=secret python -m examples.a2a # agent-to-agent + checkpoints RELATA_TOKEN=secret python -m examples.bitemporal # AS OF + WITH PROVENANCE ``` Full set: [`sdks/python/examples/`](https://github.com/relatadb/tree/main/sdks/python/examples). ## Next steps - [Search and retrieval](/docs/reference/search) — typed `/search`, multi-query batch + RRF - [Agent memory reference](/docs/reference/agent-memory) — 10 verbs + recall-quality knobs - [Graph analytics](/docs/reference/graph-analytics) — 10+ algorithms, `gds.*` portability - [Query cookbook](/docs/reference/query-cookbook) - [Full Python SDK source](https://github.com/relatadb/tree/main/sdks/python) ============================================================================== # TypeScript SDK URL: https://relatadb.dev/docs/sdks/typescript ============================================================================== # TypeScript SDK `npm install @zysec-ai/relata-sdk` — **zero runtime dependencies**. Uses native `fetch`, so it runs unchanged in Node.js 18+, Deno, Bun, modern browsers, and edge runtimes (Cloudflare Workers, Vercel Edge). Optional peer deps: `apache-arrow` + `@grpc/grpc-js` for Arrow Flight, `@langchain/*` for the adapters. > See the [SDK overview](/docs/sdks/overview) for the cross-language parity matrix. This page is the TypeScript capability catalog. ## Quickstart ```bash npm install @zysec-ai/relata-sdk ``` ```typescript import { createClient } from "@zysec-ai/relata-sdk"; const relata = createClient("http://localhost:9090", { defaultPurpose: "analytics", bearerToken: process.env.RELATA_TOKEN, }); await relata.query( "INSERT INTO Person (_pk, name, email) VALUES ('p1', 'Alice', 'a@x.com')", ); const result = await relata.query("SELECT * FROM Person LIMIT 5"); for (const row of result.rows) console.log(row.name, row.email); ``` The SDK is async-native — every method returns a `Promise`. Native `fetch` means no connection pool to close. ## The query surface | Path | Method | Returns | |---|---|---| | SQL | `query(sql, opts?)` | `QueryResult` — `opts.dialect`: `"sql" \| "cypher" \| "gql"` | | Parameterized | `queryWithParams(sql, params, opts?)` | `QueryResult` | | Arrow Flight (gRPC) | `queryFlight(sql, opts?)` | apache-arrow `Table` (optional peer) | | GraphQL | `graphql(query, variables?, operationName?)` | `unknown` — `variables` bound server-side (#3260) | | SPARQL | `sparql(query)` | `Record` | | Cypher | any `MATCH`-prefixed string via `query()` | auto-routed, governed | | GQL (ISO 39075) | `query(stmt, { dialect: "gql" })` | header-selected, governed (#3265) | | Fluent builder | `relata.select("Person").where(...).limit(10).execute()` | `QueryResult` | | Paths builder | `relata.select("Person").pathsBetween("a","b",{maxHops:5}).execute()` | `QueryResult` | ```typescript import { createClient } from "@zysec-ai/relata-sdk"; const relata = createClient("http://localhost:9090", { defaultPurpose: "investigation" }); const paths = await relata.select("Person") .pathsBetween("alice-id", "bob-id", { maxHops: 5 }) .withProvenance() .execute(); ``` ## Typed domain clients Each typed client has a `static fromClient(client)` factory inheriting auth/tenant/purpose/timeout: ```typescript import { GovernanceClient, IdentityClient, ObjectClient, IngestClient, VectorClient, SearchClient, StreamingClient, AuditClient, TenantAdminClient, BackupClient, TokenClient, LogClient, SystemClient, A2AClient, McpClient, S3Client, } from "@zysec-ai/relata-sdk"; ``` | Client | Key methods | Cross-ref | |---|---|---| | `GovernanceClient` | rules CRUD, Sigma import, retention/WORM/legal-holds, breakglass, alerts, DSAR | [Detection Rules](/docs/guides/detection-rules) | | `IdentityClient` | `label`, `recordUncertainty`, `registerLookup`/`listLookups`/`invokeLookup`, `eraseSubject` | [Identity](/docs/concepts/identity) | | `ObjectClient` | `upsert`, `typedUpsert`, `batchUpsert`, `get`, `delete`, **`list`** (TS-only) | — | | `IngestClient` | `bulk`, `bulkCsv`, `ingestAuto`, `ingestCdr`, `otlpTraces/Logs/Metrics`, `ingestIter` (AsyncIterable) | [Ingestion](/docs/guides/ingestion) | | `VectorClient` | `knnSearch`, `hybridSearch`, `similarTo`, `embed`/`embedBatch` + `embedImage/Face/Audio/Video` | [Hybrid Search](/docs/concepts/hybrid-search) | | `SearchClient` | typed `/search` JSON door: `query({from, text, rankBy, filters, limit, ...})` | [Search reference](/docs/reference/search) | | `StreamingClient` | `queryRows` (NDJSON), `queryArrowRaw`, `watch` (SSE), `alerts` (SSE) — all async generators | — | | `AuditClient` | `count`, `entries`, `findByRequestId`, `signReceipt`, `exportPdf` → `Uint8Array` | — | | `TenantAdminClient` | tenant CRUD, quota, sharing, platform usage/license | [Multi-Tenancy](/docs/guides/multi-tenancy) | | `BackupClient` | `create`, `list`, `restore`, `restoreStatus`, `compact`, `waitForRestore` | [Backup & Restore](/docs/guides/backup-restore) | | `TokenClient` | `remember`, `check`, `revoke`, `stats` | — | | `LogClient` | `append`, `head`, `loadLeaves` | — | | `SystemClient` | LLM config/test, jobs/workflows, feeds, notifications, pipelines | — | | `A2AClient` | `submitTask`, `getTask`, checkpoints, `agentCard` | — | | `McpClient` | `initialize`, `listTools`, `callTool` + **68 typed wrappers** | [MCP Tools](/docs/reference/mcp-tools) | | `S3Client` | `http`, `listBuckets`, `createBucket`, `putObject`, `getObject`, `deleteObject` | [S3 door](/docs/guides/s3-door) | ## Vectors & embeddings ```typescript const vc = new VectorClient(relata); // Pure KNN over a named slot const knn = await vc.knnSearch("Document", "embedding", [0.1,...], { k: 10, efSearch: 200 }); // Hybrid: BM25 + vector + graph, RRF-fused const hybrid = await vc.hybridSearch("Document", { queryText: "graph retrieval", k: 10, rerank: true, weights: [0.2, 0.5, 0.3] }); // Embedding (6 modalities) const e = await vc.embed("Alice Smith"); // → {embedding, model, dim} await vc.embedImage(b64); await vc.embedFace(b64); await vc.embedAudio(b64); await vc.embedVideo(b64); ``` ## Graph & intelligence operators All on `RelataClient` — 10 graph algorithms + 10 AML/financial + 3 maritime: ```typescript await relata.graphPageRank("Person", { damping: 0.85, maxIter: 20 }); await relata.graphShortestPath("alice-id", "bob-id", { maxHops: 5 }); await relata.graphCommunity("Person"); await relata.sanctionsScreen("Acme Holdings", { threshold: 0.85 }); await relata.beneficialOwnershipChain("Acme Holdings", { maxDepth: 6 }); await relata.cryptoTrace("0xabc...", "compliance"); await relata.vesselTrack(123456789, { windowSecs: 86400 }); await relata.darkFleetDetect({ maxGapHours: 48 }); ``` See [Graph Analytics](/docs/reference/graph-analytics). ## Agent memory — 10 cognitive verbs + recall-quality knobs ```typescript import { Memory } from "@zysec-ai/relata-sdk"; const mem = new Memory("http://localhost:9090", { purpose: "agent", bearerToken: "" }); const id = await mem.add("Alice prefers dark mode", { confidence: 0.9 }); // retrieval-quality operators const results = await mem.search("ui preferences", { topK: 10, minConfidence: 0.6, recencyHalfLifeSecs: 259200, budgetTokens: 1500, cancelThreshold: 0.92, }); const detail = await mem.searchDetailed("ui preferences", { /* same opts */ }); // detail.recall_cost_tokens + detail.cancelled — observe the knobs' effect ``` Full verb set: `add`, `addBatch`, `search`, `searchDetailed`, `get`, `update`, `forget`, `associate`, `episodes`, `justify`, `resolve`, `summarise`. See [Agent memory reference](/docs/reference/agent-memory). ## Ecosystem (TS extras) | Extra | Surface | |---|---| | **LangChain adapter** | `RelataMemory` — duck-typed `BaseMemory` (`loadMemoryVariables`, `saveContext`, `clear`) | | **LlamaIndex adapter** | `RelataMemory` — `put`/`get`/`getAll`/`reset` | | **LangGraph checkpointer** | `RelataCheckpointer` — real `BaseCheckpointSaver` subclass (`getTuple`/`list`/`put`/`putWrites`; `deleteThread` throws — server has no DELETE route) | | **Arrow Flight transport** | `ArrowFlightTransport` + `createArrowFlightTransport()` — hand-rolled protobuf codec, no `proto-loader` | | **CLI binary** | `npx @zysec-ai/relata-sdk` — `health`, `status`, `audit` (exit 2 on broken chain), `nodes`, `query `; reads `RELATA_TOKEN`/`RELATA_PURPOSE` | Install the optional peers (`@langchain/langgraph-checkpoint`, `@langchain/core`, `apache-arrow`, `@grpc/grpc-js`) to activate the matching surface. ## Authentication & multi-tenant ```typescript const relata = createClient("http://localhost:9090", { bearerToken: process.env.RELATA_TOKEN, defaultPurpose: "analytics", tenant: "org-acme", // X-Relata-Tenant-Id on every request actingAs: "user-42", // X-Acting-As (delegation) delegatedBy: "admin-1", // X-Delegated-By timeoutMs: 15_000, maxRetries: 3, retryBackoffMs: 500, adminBaseUrl: "http://admin.internal:9090", // /admin/* + /platform/* split }); ``` ## Runtime compatibility Native `fetch`, zero deps — runs in Node.js 18+, Deno, Bun, modern browsers, Cloudflare Workers, Vercel Edge Functions. ## Examples ~25 runnable examples in `sdks/typescript/examples/`. Run with Node 23+ (`--experimental-strip-types`), Deno, or Bun: ```bash RELATA_TOKEN=secret node --experimental-strip-types examples/basic-query.ts RELATA_TOKEN=secret node --experimental-strip-types examples/memory-quickstart.ts RELATA_TOKEN=secret node --experimental-strip-types examples/intelligence.ts RELATA_TOKEN=secret node --experimental-strip-types examples/graph-algorithms.ts RELATA_TOKEN=secret node --experimental-strip-types examples/streaming.ts RELATA_TOKEN=secret node --experimental-strip-types examples/face-search.ts ``` Full set: [`sdks/typescript/examples/`](https://github.com/relatadb/tree/main/sdks/typescript/examples). ## Next steps - [Search and retrieval](/docs/reference/search) — typed `/search`, multi-query batch + RRF - [Agent memory reference](/docs/reference/agent-memory) — 10 verbs + recall-quality knobs - [Graph analytics](/docs/reference/graph-analytics) — 10+ algorithms, `gds.*` portability - [Query cookbook](/docs/reference/query-cookbook) - [Full TypeScript SDK source](https://github.com/relatadb/tree/main/sdks/typescript) ============================================================================== # AML: Sanctions screening with an audit trail regulators can trust URL: https://relatadb.dev/docs/use-cases/aml-sanctions-screening ============================================================================== # AML: Sanctions screening with an audit trail regulators can trust ## The problem A correspondent-banking compliance analyst gets a wire transfer flagged by monitoring rules. Before it clears, they need three answers: is the counterparty (or anyone who beneficially owns it) on a sanctions list, who else in the ownership chain might be hiding exposure, and — if a regulator asks six months from now — can the bank prove exactly what was checked, when, by whom, and under what legal authority? Today that's a spreadsheet export from a sanctions-screening vendor, a manual pull from a corporate registry, and a hope that someone remembers to log the decision. The screening tool and the audit trail live in different systems, so "prove it" means reconstructing a timeline from logs that were never designed to be evidence. ## The scenario A $2.3M wire is flagged. The receiving entity is a Cyprus-registered holding company the bank has never screened directly — but its ultimate beneficial owner might already be on the OFAC SDN list under a different legal entity. The analyst needs a single, provable action: screen the entity, trace ownership up to the natural person, and get a receipt that survives a regulator's audit request. ## Screen and trace in one governed pass Relata's `SANCTIONS_SCREEN` and `BENEFICIAL_OWNERSHIP_CHAIN` are both SQL-reachable operators — no separate screening product, no CSV round-trip. Every call carries a `PURPOSE` that is written to the tamper-evident audit hash chain, and `WITH PROVENANCE` attaches the source and confidence of every row returned. ```sql PURPOSE 'aml_investigation:WIRE-2026-04412' -- 1. Screen the receiving entity directly SELECT * FROM SANCTIONS_SCREEN('Meridian Holdings Ltd') WITH PROVENANCE; -- 2. Trace beneficial ownership up to natural persons, then -- screen every name the chain surfaces SELECT chain.depth, chain.owner_name, chain.ownership_pct, SANCTIONS_SCREEN(chain.owner_name) AS screen_result FROM BENEFICIAL_OWNERSHIP_CHAIN('Meridian Holdings Ltd', 6) chain WITH PROVENANCE; ``` ## Typed SDK snippet ```python from relata import RelataClient, AuditClient with RelataClient( "http://localhost:9090", bearer_token="relata-dev", purpose="aml_investigation:WIRE-2026-04412", ) as client: # Direct screen hits = client.query("SELECT * FROM SANCTIONS_SCREEN('Meridian Holdings Ltd')") for row in hits: print(row["list_id"], row["designation_date"], row["match_confidence"]) # Walk the ownership chain and screen every name it surfaces chain = client.query( "SELECT * FROM BENEFICIAL_OWNERSHIP_CHAIN('Meridian Holdings Ltd', 6)" ) for link in chain: result = client.query( f"SELECT * FROM SANCTIONS_SCREEN('{link['owner_name']}')" ) if result: print(f"Hit at depth {link['depth']}: {link['owner_name']}") # Pull the signed, court-defensible receipt for this exact investigation with RelataClient("http://localhost:9090", bearer_token="relata-dev", purpose="compliance_review") as client: audit = AuditClient.from_client(client) receipt = audit.signed_receipt(exhibit_id="WIRE-2026-04412") ``` ## Why this is defensible, not just fast - **Every screen is audited, not logged.** `PURPOSE 'aml_investigation:WIRE-2026-04412'` is recorded against the principal, timestamp, and every row touched in the same hash-chained audit log that governs writes — see [Governance](/docs/concepts/governance). - **The receipt is signed, not exported.** `AuditClient.signed_receipt()` returns a verifiable record of exactly what was screened and when — the same primitive used for regulatory exam responses, not a bolt-on report generator. - **Ownership tracing and sanctions screening are the same engine.** There's no join between a KYC vendor's API and a separate case-management tool — `BENEFICIAL_OWNERSHIP_CHAIN` and `SANCTIONS_SCREEN` run against the same governed store, so the two results are provably about the same investigation. - **Bi-temporal by default.** A sanctions designation added last week doesn't rewrite history — `AS OF` queries can show what was knowable at the time the wire actually cleared, which is exactly what a regulator asks about. ## See also - [Governance](/docs/concepts/governance) — `PURPOSE`, the audit hash chain, and signed receipts - [Identity Resolution](/docs/concepts/identity) — how `IdentityIndex` links a legal-entity name to every canonical identifier it appears under - [Query Cookbook](/docs/reference/query-cookbook) — the full operator TVF reference, including `SANCTIONS_SCREEN` and `BENEFICIAL_OWNERSHIP_CHAIN` - [LEA: court-admissible investigation graph](/docs/use-cases/lea-investigation-graph) — the same audit primitives applied to a criminal case file ============================================================================== # App dev: RAG over governed knowledge URL: https://relatadb.dev/docs/use-cases/appdev-governed-rag ============================================================================== # App dev: RAG over governed knowledge ## The problem Most RAG stacks are a vector database bolted next to the system of record: embeddings live in one place, access control lives somewhere else (if it exists at all), and by the time a document is chunked and embedded, its provenance and permissions are usually lost. That's fine for a demo. It's not fine for an agent answering questions over data that has row-level access rules, or for a product that needs to say *why* it gave the answer it gave. ## The scenario A developer is building an internal assistant that answers questions over a governed knowledge base — policy documents, case notes, whatever the tenant has ingested — and needs every answer to (a) respect the same ACL that protects the underlying rows, (b) come back with a source citation, and (c) persist what the agent learns across sessions, not just within one conversation. ## Retrieve with governance built in, not bolted on `HYBRID_SEARCH` (aliased as `RAG_RETRIEVE` for a RAG-shaped surface) fuses BM25 and vector similarity with reciprocal-rank fusion, runs through the same ACL and cell-masking path as every other query, and returns results a caller can cite. ```sql PURPOSE 'investigation' RAG_RETRIEVE FROM Document QUERY 'what were the Q3 findings on vendor risk' LIMIT 10 ``` ## Typed SDK snippet ```python from relata import RelataClient, SearchBuilder, Memory with RelataClient( "http://localhost:9090", bearer_token="relata-dev", purpose="product_research", ) as client: # Governed retrieval: BM25 + vector, ACL-filtered before results are returned results = client.search( SearchBuilder("Q3 findings on vendor risk") .types(["Document"]) .limit(10) .highlight(True) ) context = "\n\n".join( f"[{hit.id}] {hit.highlight or ''}" for hit in results.hits ) # ... pass `context` to your LLM call; each chunk carries its own # source id, so the model's answer can cite exactly what it used ... # Give the agent memory that persists across sessions with Memory("http://localhost:9090", purpose="agent-notes") as mem: mid = mem.add("User's team owns vendor-risk review for Q3") recalled = mem.search("who owns vendor risk", top_k=3) # recalled entries are re-ranked by confidence x recency, not just similarity ``` ## Why this is more than "add a vector column" - **Retrieval and access control are the same query.** `HYBRID_SEARCH` / `RAG_RETRIEVE` runs through the same ACL-aware path as any other read — a principal never gets a chunk back that a direct `SELECT` on the same row would have denied. See [Hybrid Search](/docs/concepts/hybrid-search) for the pre-filter vs. post-filter strategy that keeps this fast even for narrow principals. - **Every result is citable.** Hits carry `id`, `score`, and `highlight` — an agent's answer can point back to the exact source row, not a black-box embedding. - **Three signals, not one.** BM25 catches exact jargon and account numbers an embedding blurs past; vector similarity catches paraphrase; identity matching catches the same entity referenced under a different surface form. Fused with RRF, not manually weighted. - **Memory is rows in the same governed store, not a separate cache.** The `Memory` client's `add` / `search` / `forget` verbs are governed writes and reads — bi-temporal, purpose-scoped, and erasable — not an ungoverned side-channel next to your real database. See [Agent Memory](/docs/concepts/agent-memory). ## See also - [Hybrid Search](/docs/concepts/hybrid-search) — `HYBRID_SEARCH`, `RAG_RETRIEVE`, `RERANK`, and the BM25 + HNSW/DiskANN internals - [Agent Memory](/docs/concepts/agent-memory) — the `remember / recall / forget / justify` verb surface - [Governance](/docs/concepts/governance) — ACL-aware pre-filtering and why retrieval can't bypass it - [Python SDK quickstart](/docs/sdks/python) — full `RelataClient`, `SearchBuilder`, and `Memory` setup ============================================================================== # Who Relata is for URL: https://relatadb.dev/docs/use-cases/by-role ============================================================================== # Who Relata is for RelataDB is one engine, but it earns its keep differently for each team. This page is the **map** — pick your role, see what Relata does for you that your current stack doesn't, and jump to the deep page. Every capability listed is shipped today (verified against source), with a concrete starting point. > Don't see your team? The common thread is: **messy, sensitive, connected data that must be provable.** If that's you, Relata fits. [Relata vs others](/docs/concepts/relata-vs-others) has the honest "when NOT to use it" too. ## At a glance | Team | What you replace / augment | The Relata wedge | Start here | |---|---|---|---| | **Security / SecOps / Threat Intel** | SIEM + sidecar + graph DB | Detection rules that fire at commit time, Sigma import, MITRE-tagged alerts, investigation graph, tamper-evident audit | [For Security Teams](/docs/use-cases/for-security-teams) | | **Finance / AML / KYC** | Sanctions screening + transaction monitoring + data warehouse | Deterministic identity fusion across phone/email/IBAN/crypto, UBO + wire/crypto/hawala tracing, regulator-ready audit | [For Finance & AML](/docs/use-cases/for-finance-teams) | | **Legal / Compliance / Audit** | e-discovery + records management + audit log | Court-grade query replay (`EXPLAIN_REPLAY`), PROV-O provenance per fact, legal holds, WORM, GDPR DSAR, signed PDF reports | [For Legal & Compliance](/docs/use-cases/for-legal-compliance) | | **AI Agent builders** | Vector DB + memory tool + graph + governance glue | 10 cognitive memory verbs, MCP tools, governed RAG, framework adapters, A2A, multimodal embedding | [For AI Agent Builders](/docs/use-cases/for-ai-agents) | | **Data / Analytics / Big Data** | Polyglot stack (Postgres + Neo4j + Elastic + Qdrant + Iceberg) | One query plane: relational + graph + vector + FTS + temporal, open Parquet/Arrow interchange, `gds.*` portability | [Hybrid Search](/docs/concepts/hybrid-search) · [Graph Analytics](/docs/reference/graph-analytics) | | **Investigators / Analysts** | Link analysis tool + spreadsheet + email | Self-forming identity graph, `PATHS_BETWEEN`, bi-temporal "what did we know when", governed sharing | [LEA Investigation Graph](/docs/use-cases/lea-investigation-graph) | ## What every team gets (the common core) Regardless of role, four things are built in — not bolted on: 1. **Identity resolution** — phone / email / IBAN / IMEI / MMSI / VIN / crypto address… standardized deterministically (76 canonical types), and any two records sharing one auto-link into the graph. 2. **Bi-temporal history** — every row carries `valid_from/to` (when it was true) + `system_from/to` (when Relata learned it). "What did we know on Tuesday?" is one `AS OF` query. 3. **Provenance on every fact** — PROV-O lineage + a tamper-evident audit hash chain. Every belief is traceable to its source. 4. **Cell-level governance** — Cedar-inspired ABAC, PURPOSE tracking, org isolation, GDPR Art. 17 erasure. Plus your **existing tools work unchanged**: 13 wire protocols (MongoDB / Postgres+pgvector / Redis / Neo4j / ClickHouse / S3 / Bolt / Flight / HTTP / gRPC / MCP / SPARQL) from one binary. See [Compatibility & Doors](/docs/compatibility). ## Pick a starting point - **"I want to try it in 60 seconds with my existing client."** → [Compatibility & Doors](/docs/compatibility) - **"Show me a worked example in my industry."** → [Use cases](/docs/use-cases/aml-sanctions-screening) (AML, LEA, telecom, maritime, cyber, OSINT, RAG) - **"I build AI agents."** → [For AI Agent Builders](/docs/use-cases/for-ai-agents) · [AI in RelataDB](/docs/concepts/ai-in-relatadb) - **"I run production."** → [Deployment](/docs/deployment) · [Deploying Protocol Doors](/docs/deployment/protocol-doors) ## See also - [Relata vs others](/docs/concepts/relata-vs-others) — honest comparison, including when NOT to use Relata - [How it works](/docs/architecture/how-it-works) — the conceptual mental model - [Limits & Caveats](/docs/reference/limits) — what honestly ships vs. roadmap ============================================================================== # From 4 databases to 1 — consolidating the polyglot stack URL: https://relatadb.dev/docs/use-cases/consolidate-polyglot-stack ============================================================================== # From 4 databases to 1 — consolidating the polyglot stack Most data-driven teams end up running **four** databases without ever deciding to. Postgres for the relational system-of-record. MongoDB for the flexible document store. Redis for cache and sessions. Neo4j for the relationship graph. Each was the right tool for one job — and each added a full operational tax: replication, backups, security review, schema migration, monitoring, on-call rotation. They share data by ETL that loses identity, history, and provenance at every hop, and *governance* becomes a fifth product bolted on top. RelataDB collapses all four into **one binary** that speaks every one of those wire protocols. Your Postgres app keeps using `psql`/`psycopg2`. Your Mongo app keeps using the official driver. Your Redis calls keep working. Your graph queries still run over Bolt. But they all read and write **one governed store** — with identity fusion, bi-temporal history, PROV-O provenance, and cell-level ACL built in. > This is the story of how that consolidation actually happens — the before, the after, and the phased path between them. Every capability is shipped today; the maturity table at the end is honest about which migration connectors are live vs. stubbed. ## The "before" — a typical 4-database stack Meet a fictional but realistic team: **Sentinel Watch**, a mid-market fraud-and-comms analytics SaaS. Their platform ingests call records, transactions, sanctions feeds, and customer profiles, and serves investigators a linked view of "who knows whom, who paid whom." Their stack grew organically: ``` ┌───────────────┐ ETL ┌───────────────┐ app ─► │ Postgres │ ──────► │ Neo4j │ ◄─ graph queries │ customers, │ │ relationships │ │ transactions │ │ rebuilt nightly│ └───────────────┘ └───────────────┘ ┌───────────────┐ sync ┌───────────────┐ app ─► │ MongoDB │ ◄─────► │ Redis │ ◄─ cache + sessions │ event logs, │ │ │ │ flexible docs │ └───────────────┘ └───────────────┘ │ ▼ nightly identity-resolution batch ┌───────────────┐ │ identity tool │ (lossy, LLM-guessed, expensive) └───────────────┘ ``` **The pain they live with:** - **Same entity, four copies.** Alice is `customer_id=42` in Postgres, `{_id: "a91f…"}` in Mongo, `user:alice` in Redis, and a disconnected node in Neo4j. Reconciling them is a nightly batch job that's lossy and never quite current. - **History is gone.** Postgres keeps "latest value wins." Mongo overwrites. When an auditor asks "what did we know on March 15?", the answer is "restore a backup and guess." - **No provenance.** Who put a fact there? When? From which feed? Nobody knows — `created_at` columns are half-populated. - **Governance is a separate product.** Cell-level access (redact SSN for team X, mask PII for country Y) is enforced in application code, inconsistently, across four read paths. - **Operational tax ×4.** Four replication setups, four backup schedules, four security reviews, four schema-migration pipelines, four on-call pages. ## The "after" — one binary, four protocols, one governed store ``` ┌── psql / psycopg2 / pgvector (pgwire :5433) ├── Mongo driver (Mongo wire :27017) ┌────────────────────────────────────┐ ├── redis-cli / resp (Redis :6379) app ─► │ RelataDB (1 binary) │ ◄─ all four├── Neo4j driver / Bolt (Bolt :7687) │ │ protocols└── HTTP / gRPC / Flight / MCP / SPARQL │ one bi-temporal governed store │ │ + identity fusion (deterministic) │ │ + provenance on every row │ │ + Cedar cell-level ACL │ └────────────────────────────────────┘ ``` Same app. Same drivers. Same query shapes. Four databases become **one fronted by thirteen wire doors** — and behind those doors is one store where identity is auto-merged, history is bi-temporal, every fact is provable, and access is enforced in the query path. **What Sentinel Watch stops doing on day one:** - Stop running the nightly identity-resolution batch — SmartIngest canonicalizes 76 identifier kinds on write and auto-links as records land. - Stop rebuilding the Neo4j graph nightly — edges derive from rows (and `GraphTrigger` makes the graph self-building from typed columns). - Stop hand-cleaning `+44 7700…` vs `07700…` — deterministic phone/IBAN/email/crypto canonicalization, byte-identical every run. - Stop bolting governance on each read path — Cedar ABAC fires once, in the planner, on every door. - Stop operating four databases — one binary, one backup, one audit chain, one on-call. ## What you keep unchanged - **Drivers & ORMs** — `psycopg2`, the official Mongo driver, `redis-py`, the Neo4j driver, `boto3`, `psql`, `mongosh`, `redis-cli`, `cypher-shell`, DBeaver, TablePlus, LangChain PGVector… all work unchanged. Repoint host/port and use the bearer token as the password. - **Query shape** — your SQL keeps being SQL. Your Cypher keeps being Cypher (`MATCH` auto-routes). Your Mongo queries keep working (within the supported subset). - **App code** — zero rewrite. The doors are wire-compatible servers. ## A worked migration — Sentinel Watch, phased ### Phase 1 — stand up Relata in parallel (week 1) Run Relata alongside the existing four DBs. No app changes. Enable the doors you need: ```bash RELATA_BEARER_TOKEN= \ RELATA_PG_ENABLE=true RELATA_PG_BIND=0.0.0.0 \ RELATA_MONGO_ENABLE=true RELATA_MONGO_BIND=0.0.0.0 \ RELATA_REDIS_ENABLE=true RELATA_REDIS_BIND=0.0.0.0 \ RELATA_BOLT_ENABLE=true RELATA_BOLT_BIND=0.0.0.0 \ RELATA_PROFILE=server relata serve ``` Point **one read-only copy** of each app at Relata and confirm queries return correct shape. No data has moved yet — this is a smoke test that your clients are wire-compatible. See [Compatibility & Doors](/docs/compatibility) for the per-protocol quickstarts. ### Phase 2 — migrate the historical data (weeks 2–3) Migrate data out of the old DBs into Relata's governed store. Honest maturity: | Source | Migration path | Status | |---|---|---| | **Postgres** | `relata import --from postgres --dsn "postgresql://..." --table --type ` — server-side cursor, streaming pages, type-faithful | ✅ **Live** | | **CSV / NDJSON** | `relata import --from csv --file ` (or `--from ndjson`) | ✅ **Live** | | **MongoDB** | Export to NDJSON (`mongoexport`) → `relata import --from ndjson`; the live door handles ongoing traffic meanwhile | 🟡 Import stub documented; use the export workaround | | **Neo4j** | Export nodes/edges to CSV/NDJSON → `relata import`; the live Bolt door handles ongoing traffic | 🟡 Import stub documented; use the export workaround | | **Redis** | Re-hydrate from your snapshot by writing through the Redis door (keys land as governed `KvEntry` rows) | ✅ Door path | > **Important nuance:** the **doors** (Mongo/Bolt/ClickHouse) are production-grade — your app's *ongoing* reads and writes through them work today. The one-time `relata import --from ` connector for Mongo/Neo4j/ClickHouse is the part that's still an honest stub; until it lands, export-then-import gets the historical rows in. See [Connectors & Extensions](/docs/guides/connectors). Every migrated row lands through the governed write path (`governed_upsert_many`) — so SmartIngest identity detection, ACL, tenant-ownership, and audit logging all apply automatically. ### Phase 3 — cut the app over, door by door (weeks 3–4) One door at a time, repoint the app's connection string from the old DB to Relata's door. Roll back is trivial — point back at the old DB. Each cutover is independent: - **Postgres app** → `psql -h relata-vip -p 5433 -U relata` (password = token) - **Mongo app** → `mongodb://relata-vip:27017` (auth `relata` / token) - **Redis app** → `redis-cli -h relata-vip -p 6379 -a ` - **Neo4j app** → `bolt://relata-vip:7687` (auth `neo4j` / token) Writes now land in the governed store. Reads over **any** door see them — the Mongo app's write is visible to the Postgres app's `SELECT`, the Neo4j app's graph traversal, the S3 door's objects, and the SQL search. ### Phase 4 — decommission the old DBs (week 5+) Once traffic is stable on Relata, retire the old databases one at a time. The operational tax collapses from ×4 to ×1: one backup (object-store-native), one audit chain, one schema model, one on-call rotation. ## The payoff — what becomes possible that wasn't before Consolidation alone is a win. But because all four data shapes now live in **one governed store**, capabilities that were impossible in the polyglot stack become a query: - **Cross-source identity fusion** — the Mongo event log's `user_id` resolves to the Postgres customer's `email` resolves to the Neo4j node. `LOOKUP_IDENTITY` / `RESOLVE_IDENTITY` / `PATHS_BETWEEN`. - **Bi-temporal "what did we know when"** — `AS OF ''` across types that came from different original DBs. - **Cross-shape joins** — `SELECT … FROM MongoDocument JOIN Person ON …` — impossible when they lived in separate engines. - **One audit chain** — a forensic query spans what used to be four separate logs. - **Governance once** — Cedar ABAC enforces cell-level access uniformly across every protocol a request came in on. ## Honest maturity table | Capability | Status | |---|---| | Postgres / pgvector door | ✅ Production | | MongoDB wire door | ✅ Production (subset: SCRAM-SHA-256, no transactions/change-streams/`$push`/`$pull`/`$unset`) | | Redis RESP door | ✅ Production (no `MULTI/EXEC`, scripting, cluster commands) | | Neo4j HTTP Cypher + Bolt doors | ✅ Production (Cypher subset; read + governed write) | | ClickHouse HTTP / native doors | ✅ Production (read-only) | | S3-compatible door | ✅ Production (with bi-temporal `?versions`; cluster fan-out pending) | | `relata import --from postgres` | ✅ Live | | `relata import --from {csv,ndjson}` | ✅ Live | | `relata import --from {mongo,neo4j,clickhouse}` | 🟡 Honest stubs — use export-then-import today | See [Limits & Caveats](/docs/reference/limits) for the full per-protocol status. ## When NOT to do this - Your data is clean, public, and low-stakes → a single Postgres is simpler than consolidation. - You only need one database shape (pure document, pure graph, pure vector) → a specialty DB is lighter. - Your polyglot stack is already paid-down and stable with no governance pain → don't migrate for migration's sake. Relata earns its keep when the four-DB tax is real **and** the data is messy, sensitive, connected, and needs proving. ## How to start 1. **Try one door** against a running Relata in 60 seconds — [Compatibility & Doors](/docs/compatibility). 2. **Migrate one Postgres table** — `relata import --from postgres --dry-run` to preview the mapping. 3. **Cut over one app** as a pilot, measure, expand. ## See also - [Compatibility & Doors](/docs/compatibility) — the wire-protocol story - [Deploying Protocol Doors](/docs/deployment/protocol-doors) — production door wiring - [Connectors & Extensions](/docs/guides/connectors) — `relata import` + the ETL framework - [Relata vs others](/docs/concepts/relata-vs-others) — when to pick Relata - [Who Relata is for](/docs/use-cases/by-role) — by-team map ============================================================================== # Cyber: Sigma detection over governed telemetry URL: https://relatadb.dev/docs/use-cases/cyber-sigma-detection ============================================================================== # Cyber: Sigma detection over governed telemetry ## The problem Sigma is the industry-standard way to write portable detection logic, but running a Sigma rule usually means shipping it to a SIEM that's a separate system from the one holding your audit trail, your identity graph, and your access controls. When a rule fires, the analyst pivots to a different tool to trace the alert back to a host, a user, and — eventually — a provenance chain a compliance reviewer can trust. Two systems, two access models, two places for the trail to break. ## The scenario A SOC analyst wants to catch a specific lateral-movement pattern — a process running as `SYSTEM` immediately after a user-level logon from an unfamiliar host — across ingested EDR and SIEM telemetry, then confirm the same account touched other hosts in the same window, all without exporting anything to a separate detection engine. ## Import the rule, query the telemetry it protects Sigma rules import directly into Relata's governance layer — detection logic and the governed store it runs against are the same system. ```python from relata import RelataClient, GovernanceClient with RelataClient( "http://localhost:9090", bearer_token="relata-dev", purpose="threat_hunting", ) as client: gov = GovernanceClient.from_client(client) # Import a Sigma rule — same governed store, no SIEM round-trip gov.import_sigma(open("sigma/lateral-movement-system-escalation.yml").read()) ``` ```sql PURPOSE 'threat_hunting' -- Full-text hunt across ingested telemetry for the pattern the Sigma rule targets SELECT hostname, process_name, command_line, event_time FROM AttackEvent WHERE MATCH(command_line, 'runas', PHRASE) ORDER BY event_time DESC LIMIT 50 WITH PROVENANCE; ``` ## Typed SDK snippet ```python from relata import RelataClient, GovernanceClient with RelataClient( "http://localhost:9090", bearer_token="relata-dev", purpose="threat_hunting", ) as client: gov = GovernanceClient.from_client(client) gov.import_sigma(open("sigma/lateral-movement-system-escalation.yml").read()) # Hunt for the pattern across governed telemetry, provenance attached hits = client.query( "SELECT hostname, process_name, command_line, event_time " "FROM AttackEvent WHERE MATCH(command_line, 'runas', PHRASE) " "ORDER BY event_time DESC LIMIT 50 WITH PROVENANCE" ) for row in hits: print(row["hostname"], row["process_name"], row["event_time"]) # Reconstruct the account's footprint across other hosts in the window account_hosts = client.query( "SELECT * FROM RESOLVE_IDENTITY('svc_account', MODE => 'cluster')" ) ``` ## Why the trail doesn't break - **Detection logic and the data plane are one system.** `GovernanceClient.import_sigma()` loads a standard Sigma rule directly into the same store the query runs against — no export to a SIEM and back. See [Governance](/docs/concepts/governance). - **Full-text search is native, not bolted on.** `MATCH(...)` with `PHRASE`, `FUZZY`, or `STEMMED` modes runs against a custom BM25 index over the same governed rows — see [Hybrid Search](/docs/concepts/hybrid-search) for the full retrieval-signal breakdown. - **Every hit carries provenance.** `WITH PROVENANCE` means an incident-response write-up can cite exactly which ingested record, from which sensor, produced each match. - **Identity resolution reaches across hosts and accounts the same way it reaches across a financial or OSINT investigation.** `RESOLVE_IDENTITY` is the same operator used everywhere else in the platform — a SOC analyst is one query away from the identity graph, not a separate UEBA product. ## See also - [Governance](/docs/concepts/governance) — `import_sigma`, `PURPOSE`, and the audit hash chain - [Hybrid Search](/docs/concepts/hybrid-search) — `MATCH` modes and the BM25 engine internals - [OSINT: cross-platform identity fusion](/docs/use-cases/osint-identity-fusion) — the same `RESOLVE_IDENTITY` operator used for account/actor correlation - [Query Cookbook](/docs/reference/query-cookbook) — the full graph-operator surface for lateral-movement and pivot queries ============================================================================== # For AI agent builders URL: https://relatadb.dev/docs/use-cases/for-ai-agents ============================================================================== # For AI agent builders Most agent stacks are glued together from five pieces: a vector DB for retrieval, a memory tool (Mem0/Zep) for recall, a graph DB for relationships, a governance layer you build yourself, and an audit log you hope is defensible. Each piece has its own consistency model, its own identity, and its own idea of "current truth." RelataDB replaces all five with **one governed store** — and your agent gets memory that's bi-temporal, recall that's tunable, and every belief traceable to the tool call that produced it. This is the agent-memory layer for **regulated, audited, defensible** AI — where a hallucinated or unexplainable memory is a liability, not a quirk. ## What you replace | Today (polyglot) | With Relata | |---|---| | Vector DB (Pinecone / Qdrant / pgvector) | Hybrid search (BM25 + HNSW + identity fusion, RRF) in one store | | Memory tool (Mem0 / Zep / Cognee) | 10 governed cognitive verbs with recall-quality knobs | | Graph DB for relationships | The graph forms itself from standardized identities | | Framework-specific memory glue | 7 Python adapters + 3 TS adapters + a LangGraph checkpointer | | Hallucinated entity extraction | Deterministic canonical-type detection (76 kinds, zero hallucination) | | Bolt-on audit/provenance | PROV-O per memory + tamper-evident hash chain + `justify` | ## The agent stack ``` ┌─────────────── your agent (LangGraph / CrewAI / custom) ──────────────┐ │ │ │ MemoryClient ◄──► 10 cognitive verbs (remember … summarise) │ │ │ + recall-quality knobs │ │ │ │ │ McpClient ◄──► 40+ tools: investigate, find_threats, │ │ rag_store_answer, nl_query, hybrid_search … │ │ │ │ VectorClient ◄──► embed (text + image/face/audio/video) + KNN/hybrid│ │ │ │ RelataClient ◄──► governed SQL / Cypher / GraphQL over the same store│ └───────────────────────────────┬───────────────────────────────────────┘ │ one governed bi-temporal store ▼ identity fusion + provenance + cell-level ACL + audit chain ``` ## The 10 cognitive verbs + recall-quality knobs Every memory is a governed, bi-temporal `MemoryItem` (content + confidence + memory_class + valid_from/to + system_from/to) linked back to the `ToolCall` and `AgentSession` that produced it. | Verb | What it does | |---|---| | `remember` / `add` | Store a memory (episodic / semantic / procedural). | | `recall` / `search` | Hybrid BM25 + vector retrieval, re-scored by confidence × recency × forgetting curve. | | `recognize` / `get` | Fetch one memory with full provenance attached. | | `justify` | The PROV-O chain (ToolCall → MemoryItem → DecisionRecord) — *why* the agent believes this. | | `consolidate` / `update` | Supersede a memory; old retained in history, new gets higher confidence. | | `forget` | Governed retention-policy retract (NOT hard delete). | | `associate` | Typed, provenance-stamped link between two memories/entities. | | `episodes` | List `Episode` records for a session, ordered by valid_from. | | `resolve` | Follow the supersession chain to the canonical live memory. | | `summarise` | Produce a governed, provenance-stamped summary belief from source memories. | **The recall-quality knobs** — tune *what* comes back, not just how many: ```python mem.search("how do we reset the IR sensor?", top_k=10, min_confidence=0.6, # CONFIDENCE floor recency_half_life_secs=259200, # 3-day decay (RECENCY) budget_tokens=1500, # hard prompt budget — can't overflow (BUDGET) stability_days=30, # Ebbinghaus forgetting (FORGETTING_CURVE) cancel_threshold=0.92, # stop early on a great match (CANCEL_WHEN) ) ``` `search_detailed` exposes `recall_cost_tokens` + `cancelled` so you can **observe** the knobs' effect. See [Agent memory reference](/docs/reference/agent-memory) (recall knobs + the 5 operators). ## RAG — governed, with provenance per answer A RAG answer in most stacks is a black-box string. In Relata it's a governed `RagAnswer` row linked to its `RagSource` rows, with PROV-O and the `ToolCall` that produced it — defensible and replayable. ```python # MCP path — the simplest surface mcp.call_tool("rag_store_answer", { "question": "What's our refund policy for EU customers?", "answer": "14-day no-questions refund under EU consumer law…", "confidence": 0.92, "sources": ["doc-refund-policy", "regulation-eu-2011-83"], "purpose": "support", }) # Retrieve for the next turn — governed hybrid search over your knowledge corpus mcp.call_tool("search_knowledge", { "query": "EU refund window", "min_confidence": 0.5, "purpose": "support", }) ``` The RAG ingest path is `POST /ingest/document` (NDJSON chunks + a manifest) or the MCP `ingest_document` tool — async, returns a `task_id` you poll. See [Ingestion](/docs/guides/ingestion). ## MCP tools — natural-language investigation + retrieval 40+ MCP tools, callable from any MCP-compatible agent runtime (Claude Desktop, Cursor, your own): ```python mcp.call_tool("nl_query", {"query": "show me high-risk customers added this week", "interpret": True, "purpose": "analytics"}) # → rows + generated_sql + model_id + llm_used (audit fields) mcp.call_tool("investigate_entity", {"entity_type": "Person", "entity_id": "alice-001", "purpose": "security_incident"}) mcp.call_tool("find_threats", {"entity_type": "Alert", "purpose": "security_incident"}) mcp.call_tool("hybrid_search", {"entity_type": "Document", "query": "refund policy EU", "top_k": 10, "purpose": "support"}) ``` The `nl_query` response carries `generated_sql` + `model_id` + `llm_used` so you know whether the SQL came from the **deterministic local translator** or an LLM — essential for audit. Deterministic local translator runs when `RELATA_LLM_URL` is unset (air-gap friendly); the LLM is used when set. See [MCP Tools](/docs/reference/mcp-tools). ## Framework adapters — drop Relata in as governed memory **Python — 7 adapters** (`relata_adapters`, ships with the package; install only your framework): ```python # LangChain from relata_adapters.langchain import RelataMemory memory = RelataMemory(base_url="http://localhost:9090", purpose="agent", bearer_token="") # CrewAI / AutoGen / AG2 / Pydantic-AI / smolagents / LlamaIndex — same shape from relata_adapters.crewai import RelataStorage # Auto-detect which is installed and return the right class from relata_adapters.registry import get_memory_adapter Adapter = get_memory_adapter() ``` **LangGraph checkpointer** (`pip install relata-sdk[langgraph]`): ```python from relata_langgraph import RelataCheckpointer, AsyncRelataCheckpointer checkpointer = RelataCheckpointer(endpoint="http://localhost:9090", token="") graph = builder.compile(checkpointer=checkpointer) ``` A real `BaseCheckpointSaver` subclass — persists graph state via the governed A2A checkpoint door, so an agent's full trajectory is as auditable as its memories. **TypeScript — 3 adapters** (LangChain / LlamaIndex / LangGraph) + a CLI binary (`npx @zysec-ai/relata-sdk health`). ## A2A — agent-to-agent tasks Agents delegate and share state through governed `A2ATask` rows + checkpoints: ```python a2a = A2AClient.from_client(client) task_id = a2a.submit_task({"kind": "summarize_case", "case_id": "case-7", "purpose": "investigation"}) a2a.save_checkpoint("thread-1", "step-3", {"draft": "..."}) ``` ## 6 embedding modalities Embed through `VectorClient` or `/embed` — text, image (CLIP), face crop (ArcFace), audio (CLAP), video keyframe (CLIP), plus batch text: ```python vc.embed("Alice Smith") # text (CPU lexical default; GPU sidecar via RELATA_ACCEL_ENDPOINT) vc.embed_image(b64) # CLIP — multimodal RAG vc.embed_face(b64) # ArcFace — see [Multimedia search](/docs/guides/multimedia-search) vc.embed_audio(b64) # CLAP vc.embed_video(b64) # CLIP keyframe ``` ## Worked design — a governed support agent ```python from relata import RelataClient, Memory from relata_adapters.langchain import RelataMemory client = RelataClient("http://localhost:9090", bearer_token="", purpose="support") # 1. RAG: ingest your policy docs once client.ingest_document(chunks_jsonl=open("policies.jsonl").read(), manifest_json=open("manifest.json").read()) # 2. Per-conversation governed memory mem = Memory("http://localhost:9090", bearer_token="", purpose="support") mem.add("Customer AC-042 prefers email replies; past refund disputes.", memory_class="semantic", confidence=0.9) # 3. Answer with retrieval + memory, store the answer with provenance answer = mcp.call_tool("hybrid_search", {"entity_type": "Document", "query": "", "top_k": 5}) mcp.call_tool("rag_store_answer", { "question": "", "answer": "", "sources": [s["id"] for s in answer], "confidence": 0.88, "purpose": "support", }) # 4. Every belief is justifiable later mem.justify(answer_id) # → the ToolCall + sources that produced it ``` Every step is governed, bi-temporal, and audit-logged — a compliance review can reconstruct exactly what the agent believed, retrieved, and answered at any past moment. ## Tips & takeaways - **Start with `budget_tokens`.** It's the single biggest agent-loop win — the prompt literally cannot overflow the model's context window. - **Use deterministic extraction where you can.** SmartIngest's canonical-type detectors are zero-hallucination; reserve the LLM for genuine NL tasks, not for "is this the same person." - **`justify` is your compliance superpower.** It turns "why did the agent say that?" from a forensic nightmare into a one-call answer. - **Pair `RagAnswer` rows with `PURPOSE`.** A RAG answer that drives an automated action should carry a purpose so the action is as auditable as the retrieval. - **Don't replicate your warehouse into memory.** `recall` is selective (ranked, bounded, early-cancelable) — let the agent query a billion-row memory at the same prompt cost as a megabyte one. - **LangGraph state belongs in the checkpointer**, not in ad-hoc JSON — that's what makes an agent's trajectory replayable and audit-ready. ## See also - [Agent memory reference](/docs/reference/agent-memory) — 10 verbs + the recall-quality knobs in full - [Concepts: Agent Memory](/docs/concepts/agent-memory) — why governed memory - [AI in RelataDB](/docs/concepts/ai-in-relatadb) — the agent/RAG loop - [MCP Tools](/docs/reference/mcp-tools) — the 40+ tool surface - [Hybrid Search](/docs/concepts/hybrid-search) — BM25 + vector + identity RRF - [Multimedia search](/docs/guides/multimedia-search) — image/face/audio/video embedding - [App Dev — Governed RAG](/docs/use-cases/appdev-governed-rag) — worked use case ============================================================================== # For finance & AML teams URL: https://relatadb.dev/docs/use-cases/for-finance-teams ============================================================================== # For finance & AML teams Financial-crime tooling today is a patchwork: a sanctions screener, a transaction-monitoring engine, a data warehouse, a case-management tool, and an entity-resolution vendor — each holding a different copy of "who is who." RelataDB collapses it into one governed fabric where identity is **deterministic** (not LLM-guessed), every money trail is queryable as a graph, and every fact you act on is provable to a regulator. ## What you replace | Today (polyglot) | With Relata | |---|---| | Sanctions screening vendor | Native `sanctions_screen` + `relata pull-ioc` (OFAC/UN/EU/OFSI/MEA/RBI/OpenSanctions) | | Transaction monitoring + alerting | Detection rules over `Transaction` types, firing at commit time | | Entity-resolution vendor (lossy, LLM) | Deterministic canonicalization (76 types: IBAN, phone, crypto wallet…) — zero hallucination | | Graph DB for ring/UBO detection | `beneficial_ownership_chain`, `crypto_trace`, `wire_reconstruction`, `hawala_trace` as governed SQL operators | | Case management + audit warehouse | Bi-temporal case rows + PROV-O provenance + tamper-evident audit chain | ## Identity fusion — deterministic, not LLM-guessed A person shows up as `+44 7700 900123` in a wire, `07700 900123` in a CRM, an email at onboarding, and a wallet `0xabc…` on-chain. Relata canonicalizes each (E.164 uint64 for phone, checksum-verified IBAN, checksummed address) and **auto-merges them into one entity** — the same identity across every source. No model in the loop, no hallucination, byte-identical every run. ```sql PURPOSE 'compliance' -- Resolve everything known about this identity across all sources SELECT * FROM RESOLVE_IDENTITY('+44 7700 900123', MODE => 'cluster'); -- Is this the same person as the wallet owner? SELECT SAME_IDENTITY('Person:alice', 'Wallet:0xabc...') AS same; ``` SmartIngest runs the 76 canonical-type detectors on ingest — you don't build the matching pipeline, you configure it. See [Identity](/docs/concepts/identity). ## Financial-intelligence operators Ten governed AML/intel operators, all in SQL or via the SDK/MCP: ```sql PURPOSE 'compliance' -- Trace a crypto wallet through hops SELECT * FROM CRYPTO_TRACE('0xabc...', MAX_HOPS => 5, MIN_AMOUNT => 1000); -- Reconstruct a wire chain with tolerance for timing/amount gaps SELECT * FROM WIRE_RECONSTRUCTION('ACC-12345', TOLERANCE_PCT => 5); -- Beneficial ownership up to N levels deep SELECT * FROM BENEFICIAL_OWNERSHIP('Acme Holdings', MAX_DEPTH => 6); -- Hawala/informal-value-transfer pairing SELECT * FROM HAWALA_TRACE('seed-lead', MAX_HOPS => 5); ``` ```python # Python SDK — same operators client.crypto_trace("0xabc...", purpose="compliance") client.wire_reconstruction("ACC-12345", tolerance_pct=5.0) client.beneficial_ownership_chain("Acme Holdings", max_depth=6) client.hawala_trace("seed-lead", max_hops=5) ``` Plus `sanctions_screen`, `convoy_detect`, `burner_detect`, `dns_tunnel_detect`, `crime_pattern_cluster`, `geofence`. Typed decoders in `relata.aml` (`decode_sanctions_hits`, `decode_beneficial_owners`, `decode_crypto_trace`, `decode_wire_hops`, `decode_hawala_pairs`) give you typed result objects. See [AML — Sanctions Screening](/docs/use-cases/aml-sanctions-screening) for the end-to-end worked example. ## Sanctions ingest, native ```bash relata pull-ioc # OFAC / UN / EU / OFSI / MEA / RBI / OpenSanctions ``` Sanctions lists land as governed rows; `sanctions_screen` runs at query time against the live list. STIX/MISP/TAXII feeds land via `POST /import?format=stix`, `relata misp-pull`, `relata taxii-poll`. ## Bi-temporal — answer the regulator's real question The regulator doesn't ask "what do you know now?" — they ask "what did you know on the date you approved this transaction?" `AS OF` reconstructs the exact state of the world at any past moment, including the sanctions list version and the identity graph as it was then. ```sql PURPOSE 'regulator-response' -- What did we know about this entity when we filed the SAR? SELECT * FROM Person AS OF '2026-03-15T00:00:00' WHERE _pk = 'alice-001'; -- Was this wallet on any sanctions list at the time of the transfer? SELECT * FROM SanctionsHit AS OF '2026-03-15T09:30:00' WHERE entity_id = '0xabc...'; ``` Every decision is reconstructable — court-grade replay via `EXPLAIN_REPLAY` if you need byte-identical exhibit reconstruction. See [For Legal & Compliance](/docs/use-cases/for-legal-compliance). ## Provenance — every fact is defensible Every row carries PROV-O provenance (source connector, batch, offset, observed_at, recorded_at) and is stamped into the tamper-evident audit hash chain. When a regulator asks "where did this fact come from?", `justify` returns the full chain: ```bash curl 'http://127.0.0.1:9090/memory/justify/' \ -H 'Authorization: Bearer ' ``` ```bash curl 'http://127.0.0.1:9090/audit/proof' \ -H 'Authorization: Bearer ' # hash-chain validity proof ``` Sign a case report as a governed, timestamped PDF: `POST /report/pdf`. ## How to start 1. **Pull sanctions lists** — `relata pull-ioc`. 2. **Ingest customers + transactions** via `POST /ingest/bulk` or the Mongo/pg doors (your existing app keeps writing). 3. **Configure detector packs** — `RELATA_DETECT_PACKS=network,contact,crypto,financial,payment`. 4. **Write AML detection rules** as SQL `WHERE` over `Transaction` / `Wire` / `CryptoTransfer`. 5. **Investigate** with the AML operators + `PATHS_BETWEEN` graph traversals. 6. **Respond to regulators** with `AS OF` reconstruction + signed PDF reports. ## See also - [AML — Sanctions Screening](/docs/use-cases/aml-sanctions-screening) — worked example - [Identity](/docs/concepts/identity) — deterministic fusion + active learning - [Detection Rules](/docs/guides/detection-rules) — monitoring rules - [For Legal & Compliance](/docs/use-cases/for-legal-compliance) — regulator-grade audit - [Graph Analytics](/docs/reference/graph-analytics) — ring/UBO detection ============================================================================== # For legal & compliance teams URL: https://relatadb.dev/docs/use-cases/for-legal-compliance ============================================================================== # For legal & compliance teams In most systems, "audit" means a `created_at` column you hope people fill in, and "e-discovery" means restoring a backup and guessing. RelataDB treats provenance, history, and tamper-evidence as **first-class query-time properties of the data** — every fact is traceable, every past state is reconstructable, and every deletion is governed. This is the platform for regulated, audited, defensible workloads. ## What you replace | Today | With Relata | |---|---| | Audit log (append-only, unverifiable) | Tamper-evident hash chain (`/audit/proof`) on every fact | | Records management + legal-hold tooling | Native legal holds + WORM retention policies per type | | e-discovery + snapshot restoration | Bi-temporal `AS OF` — any past state is a live query | | "Chain of custody" spreadsheets | `VERIFY_CUSTODY` + `EXPLAIN_REPLAY` — byte-identical exhibit replay | | DLP / cell-level access tooling | Cedar ABAC in the query path — cell masking, by purpose / team / country | | GDPR/DSAR request tooling | Native DSAR + Art. 17 erasure with retention windows | ## Court-grade query replay — `EXPLAIN_REPLAY` Every executed query persists an immutable replay record (plan SHA, params, principal, ACL bitmap digest, branch HEAD, materialized-view-set digest, detector versions, snapshot pointer). `EXPLAIN_REPLAY` re-executes that plan against the recorded snapshot and asserts **byte-identical Arrow IPC output**. This is *the* "court-grade" answer to the defence's question: *"what exactly did the system tell the analyst on the date of the decision?"* It's stronger than `AS OF` — it captures plan + ACL state + MV freshness at the decision moment, which row-level time-travel alone doesn't. ```sql PURPOSE 'legal' EXPLAIN_REPLAY('exhibit-7', SEQ => 5); -- → reconstructed exhibit seal + chain_valid: true confirmation VERIFY_CUSTODY('exhibit-001'); -- chain-of-custody assertion ``` Signed, timestamped PDF case reports: `POST /report/pdf`. Per-row receipts via `AuditClient.sign_receipt(exhibit_id=...)` in every SDK. ## Bi-temporal history — every past state is a query Every row carries `valid_from/to` (when the fact was true) **and** `system_from/to` (when Relata learned it). No restore, no snapshot digging — the state of the world at any past moment is one query: ```sql PURPOSE 'regulator-response' -- What was the sanctions status of this entity on the filing date? SELECT * FROM SanctionsHit AS OF '2026-03-15T00:00:00' WHERE entity_id = '0xabc...'; -- Reconstruct the entire case file as the analyst saw it SELECT * FROM Person AS OF '2026-03-15T09:30:00' WHERE case_id = 'case-7'; ``` See [Bi-temporal queries](/docs/reference/bitemporal). ## Provenance on every fact (PROV-O) Every row carries a provenance chain: source connector, batch id, record offset, observed_at, recorded_at — stamped into the tamper-evident audit hash chain. Any mutation leaves a forensic trail; any deletion is detectable. ```bash # Is the audit chain intact? curl 'http://127.0.0.1:9090/audit/proof' -H 'Authorization: Bearer ' # Where did this fact come from? curl 'http://127.0.0.1:9090/memory/justify/' \ -H 'Authorization: Bearer ' ``` ```sql -- Every row that touched a case, with provenance SELECT _pk, _provenance_source, _provenance_observed_at, _audit_seq FROM Person WHERE case_id = 'case-7' ORDER BY _audit_seq; ``` ## Legal holds, WORM, retention Prevent destruction of records under legal hold or WORM (write-once-read-many) policy — enforced in the write path, not by application discipline: ```bash # Place a legal hold on a case (across all object types) curl -X POST http://127.0.0.1:9090/retention/holds \ -H 'Authorization: Bearer ' -H 'Content-Type: application/json' \ -d '{"case_id":"case-7","object_type":"Person"}' # Set WORM retention on a records type curl -X POST 'http://127.0.0.1:9090/retention/worm/TradeRecord' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{"retention_secs":2592000}' # 30 days immutable ``` ```python client.governance_client.place_legal_hold("case-7", "Person") client.governance_client.set_worm_policy("TradeRecord", retention_secs=2_592_000) ``` ## Cell-level governance + PURPOSE Cedar-inspired ABAC fires **in the query path**, not as a view layer. Who-can-see-what differs by purpose, team, country, and clearance — enforced on every read, on every door. `PURPOSE` is recorded in the audit log on every request. ```sql -- A query with a declared purpose — recorded for audit PURPOSE 'cross-border-finance' SELECT name, country FROM Person WHERE risk_score > 0.7; -- Cell-level mask: redact SSN unless the caller's purpose is 'compliance' -- (configured via Cedar policy / forbid clause — see Per-Door ACL) ``` ```cedar // Compliance team may read ssn; nobody else — deny-wins permit( principal in Role::"compliance", action == Action::"read", resource == Resource::"Person.ssn" ); ``` See [Governance](/docs/concepts/governance) and [Per-Door ACL](/docs/guides/per-door-acl). ## GDPR / DSAR / Art. 17 erasure Subject-access requests and right-to-erasure are native operations, not bespoke scripts: ```bash # Generate a DSAR export for a subject curl -X POST http://127.0.0.1:9090/gdpr/dsar \ -H 'Authorization: Bearer ' \ -d '{"subject_identity":"Person:alice-001"}' # Erase a subject (governed — audit-logged, retention-aware) curl -X POST http://127.0.0.1:9090/gdpr/erase \ -H 'Authorization: Bearer ' \ -d '{"entity_id":"Person:alice-001","purpose":"gdpr-art17"}' ``` ```python client.governance_client.submit_dsar("Person:alice-001", reason="DSAR-2026-042", scope="all") client.erase_subject("Person:alice-001", purpose="gdpr-art17-request") ``` `forget` (on memories) is retention-policy governed, not a hard delete — the item stays queryable for the window, then retracts. See [Privacy & GDPR](/docs/guides/privacy). ## How to start 1. **Ingest records** through any door (pg/Mongo/S3/HTTP) — provenance attaches automatically. 2. **Set WORM / retention policies** on regulated types. 3. **Write Cedar policies** for cell-level access (or use `RELATA_ACL_GRANT` shorthand). 4. **Place legal holds** when a matter opens. 5. **Respond to requests** with `AS OF` + `EXPLAIN_REPLAY` + signed PDF reports. ## See also - [Provenance](/docs/concepts/provenance) — PROV-O + the audit hash chain - [Bi-temporal queries](/docs/reference/bitemporal) — `AS OF` / `WITH PROVENANCE` - [Governance](/docs/concepts/governance) — Cedar ABAC, PURPOSE, cell masking - [Privacy & GDPR](/docs/guides/privacy) — DSAR + Art. 17 - [Per-Door ACL](/docs/guides/per-door-acl) — least privilege per integration - [For Security Teams](/docs/use-cases/for-security-teams) — detection + audit ============================================================================== # For security teams URL: https://relatadb.dev/docs/use-cases/for-security-teams ============================================================================== # For security teams Security tooling today is a triangle: a SIEM for detection, a graph or link-analysis tool for investigation, and a data warehouse for the cold store — glued together by ETL that loses context, identity, and provenance at every hop. RelataDB collapses the triangle into one engine. **Ingest → detect → investigate → prove** is one closed loop over one governed store, with your existing tools (Sigma, MITRE ATT&CK, psql, your SOAR webhook) working unchanged. ## What you replace | Today (polyglot) | With Relata | |---|---| | SIEM (Splunk/Elastic SIEM) + detection sidecar | Detection rules fire **at commit time** in-process — sub-request latency, no poller, no missed events | | Sigma rule ecosystem | Native `relata import-sigma` — 10 000+ community rules port as-is | | Graph DB (Neo4j) for investigation | The graph forms itself from identities; 10+ algorithms + Cypher in the same query | | Audit log + chain-of-custody tooling | Tamper-evident hash chain on every fact + `EXPLAIN_REPLAY` for court-grade replay | | Per-integration credential sprawl | Per-door Cedar principals (`s3-client`, `pgwire-client`, …) — least privilege per integration | ## Detection rules — commit-driven, Sigma-native Write a rule as a SQL `WHERE` against any governed type. It fires **within the request cycle on commit** (the `GraphChangeEvent` commit bus), not on a 30-second poll. A bitmap-indexed candidate filter keeps the per-write check cheap. ```bash curl -X POST http://127.0.0.1:9090/rules \ -H 'Authorization: Bearer ' -H 'Content-Type: application/json' \ -d '{ "name": "suspicious-dns-exfil", "target_type": "DnsEvent", "condition_sql": "query_length > 100 AND rdata_type = '\''TXT'\'' AND query LIKE '\''%.xyz'\''", "severity": "high", "mitre_technique": "T1048.002", "purpose": "security" }' ``` Import Sigma rules directly: ```bash relata import-sigma rules/azure-ad-anomalous-signin.yaml ``` Alerts are **bi-temporal** `Alert` rows with full PROV-O — so "would this rule have fired last Tuesday?" is one `AS OF` query (impossible in a SIEM that overwrites alerts). Push to your SOAR over per-tenant webhooks with **queryable delivery status**: ```bash # Find alerts whose webhook delivery failed curl 'http://127.0.0.1:9090/alerts/list?delivery_status=failed' \ -H 'Authorization: Bearer ' ``` Stream alerts live over SSE for a dashboard: `GET /alerts/stream`. Full deep-dive: [Detection Rules](/docs/guides/detection-rules). ## The investigation graph — it forms itself Every phone / email / IP / username that lands through any door (HTTP, Mongo, S3, OTLP, Kafka) is canonicalized and auto-linked. You query a graph that built itself — no separate edge-loading pipeline. ```sql PURPOSE 'investigation' -- How are these two entities connected, in ≤5 hops? SELECT * FROM PATHS_BETWEEN('Person:alice', 'Person:suspect', 5); -- Who are the influencers in the observed comms graph? SELECT id, pagerank FROM GRAPH_PAGERANK('Person', 'CONTACTED') ORDER BY pagerank DESC LIMIT 10; -- Match Cypher over Bolt with the Neo4j driver — same graph -- CALL gds.pageRank.stream('Person', {maxIterations: 20}) ``` 10+ algorithms (PageRank, SCC, community detection, link prediction, triangle count, SSSP/PLL…) callable as SQL TVFs, `CALL traverse.*`, or `CALL gds.*` (Neo4j-GDS portability). See [Graph Analytics](/docs/reference/graph-analytics). ## OTLP-native — spans link into the identity graph Native OTLP JSON receivers (`/ingest/{traces,logs,metrics}` and `/v1/{traces,logs,metrics}`). Spans map to bi-temporal `TraceSpan` rows, and identities in span attributes flow into the `IdentityIndex` — so a `trace_id` links to `Person`/`NetworkFlow`/`Transaction` through the shared identity graph. This is the cross-dataset correlation SIEM tools lack. ## Court-grade replay — defensible findings Every executed query persists an immutable replay record (plan SHA, params, ACL bitmap digest, branch HEAD, MV-set digest, snapshot pointer). `EXPLAIN_REPLAY` re-executes that plan against the recorded snapshot and asserts byte-identical output — the "court-grade" answer to *"what did the system tell the analyst on date X?"* ```sql PURPOSE 'legal' EXPLAIN_REPLAY('exhibit-7', SEQ => 5); -- → reconstructed exhibit seal + chain_valid: true VERIFY_CUSTODY('exhibit-001'); -- chain-of-custody assertion ``` Plus tamper-evident audit (`GET /audit/proof`), signed PDF reports (`POST /report/pdf`), and `AuditClient.sign_receipt(...)` in every SDK. See [For Legal & Compliance](/docs/use-cases/for-legal-compliance). ## Per-door least privilege Every one of the 13 wire doors presents a distinct Cedar principal. The S3 ingest scraper gets `read`-only; the BI tool's psql gets read-write on specific types; the compromised scraper cannot write even if its credentials leak. Every audit row is forensically attributable to the protocol that wrote it. ```bash # Audit every row the S3 door touched curl 'http://127.0.0.1:9090/audit/entries?purpose=s3-client&limit=50' \ -H 'Authorization: Bearer ' ``` See [Per-Door ACL](/docs/guides/per-door-acl). ## How to start 1. **Ingest your logs** via OTLP (`/ingest/traces`) or Kafka/CDR doors — see [Ingestion](/docs/guides/ingestion). 2. **Import your Sigma rules** — `relata import-sigma

`. 3. **Wire your SOAR** webhook: `register_webhook(url, event_types=["alert.high"])`. 4. **Investigate** with `PATHS_BETWEEN`, Cypher over Bolt, or the MCP `investigate_entity` / `find_threats` tools. 5. **Prove** with `EXPLAIN_REPLAY` + signed PDF reports for the case file. ## See also - [Detection Rules](/docs/guides/detection-rules) — the full rule engine - [Cyber — Sigma detection](/docs/use-cases/cyber-sigma-detection) — worked use case - [Graph Analytics](/docs/reference/graph-analytics) - [Per-Door ACL](/docs/guides/per-door-acl) - [For Legal & Compliance](/docs/use-cases/for-legal-compliance) — court-grade replay ============================================================================== # Use cases URL: https://relatadb.dev/docs/use-cases ============================================================================== # Use cases Problem-first walkthroughs with real SQL and SDK code — one per GA vertical. Each page states the problem, the RelataDB shape that solves it, and an honest account of what works today. ## Who it's for - [Who Relata is for](/docs/use-cases/by-role) — by role: platform engineers, security/finance/legal teams, and AI-agent builders - [From 4 databases to 1](/docs/use-cases/consolidate-polyglot-stack) — collapsing a polyglot stack into one governed engine - [For Security Teams](/docs/use-cases/for-security-teams) — SOC co-pilot with real memory and audit - [For Finance & AML](/docs/use-cases/for-finance-teams) — funds-flow, KYC, and fraud graphs - [For Legal & Compliance](/docs/use-cases/for-legal-compliance) — court-admissible, replayable records - [For AI Agent Builders](/docs/use-cases/for-ai-agents) — persistent, governed agent memory ## Vertical playbooks - [AML — Sanctions Screening](/docs/use-cases/aml-sanctions-screening) — screen a counterparty and trace beneficial ownership in one governed query - [LEA — Investigation Graph](/docs/use-cases/lea-investigation-graph) — link-analysis case file with provenance on every edge - [Telecom — CDR Co-location](/docs/use-cases/telecom-colocation-network) — turn raw call-detail records into a queryable subscriber graph - [Maritime — Dark-Fleet Detection](/docs/use-cases/maritime-dark-fleet) — AIS gaps, transhipment, and ownership fusion - [Cyber — Sigma Detection](/docs/use-cases/cyber-sigma-detection) — hunt with Sigma rules directly over governed telemetry - [OSINT — Identity Fusion](/docs/use-cases/osint-identity-fusion) — resolve a subject's full footprint from one identifier - [App Dev — Governed RAG](/docs/use-cases/appdev-governed-rag) — retrieval that respects the same ACL as a direct query See also: [Concepts](/docs/concepts) for the model behind each workload and [Comparisons](/docs/compare) for how RelataDB stacks up against alternatives. ============================================================================== # LEA: A court-admissible investigation graph URL: https://relatadb.dev/docs/use-cases/lea-investigation-graph ============================================================================== # LEA: A court-admissible investigation graph ## The problem Link analysis is the core tool of every serious investigation: which people, vehicles, and locations connect to a suspect, and how. But most link-analysis tools produce a picture, not evidence. When the defence asks "how was this connection established, and can we verify it independently?", a screenshot of a graph doesn't answer that. Every edge needs a source, every hop needs a confidence score, and the whole export needs a signature that survives cross-examination. ## The scenario A vehicle theft ring case: investigators have a suspect's license plate from an ANPR hit near the scene, and need to build out the full network — associates, other vehicles, and communication patterns — before requesting a disclosure bundle the defence counsel can independently verify. ## Build the graph, then export it as evidence `ANPR_TRACE`, `PATHS_BETWEEN`, and `DISPATCH_PRIORITY` are SQL-reachable operators, not a separate case-management product bolted on top. `WITH PROVENANCE` attaches the source, method, and confidence of every edge the query returns. ```sql PURPOSE 'investigation_specific:C-2026-04412' -- 1. ANPR timeline for the plate that put the suspect at the scene -- (WINDOW is a lookback in nanoseconds — 1209600000000000 = 14 days) SELECT * FROM ANPR_TRACE('WB02-XX-1234', WINDOW => 1209600000000000, REGION => 'WB') WITH PROVENANCE; -- 2. Expand outward: every path from the suspect to a known associate, -- up to 4 hops, with the confidence and source of each connecting edge SELECT * FROM PATHS_BETWEEN('person-suspect-441', 'person-associate-118', 4) WITH PROVENANCE; ``` ## Typed SDK snippet ```python from relata import RelataClient, AuditClient with RelataClient( "http://localhost:9090", bearer_token="relata-dev", purpose="investigation_specific:C-2026-04412", ) as client: # Vehicle timeline from the ANPR hit — WINDOW is a lookback in ns sightings = client.query( "SELECT * FROM ANPR_TRACE('WB02-XX-1234', " "WINDOW => 1209600000000000, REGION => 'WB')" ) for hit in sightings: print(hit["cell_id"], hit["timestamp"], hit["confidence"]) # Every path connecting the suspect to a known associate paths = client.query( "SELECT * FROM PATHS_BETWEEN('person-suspect-441', 'person-associate-118', 4)" ) for path in paths: print(path["hops"], path["score"], path["provenance"]) # Export a signed, defence-ready evidence package with RelataClient("http://localhost:9090", bearer_token="relata-dev", purpose="legal_disclosure") as client: audit = AuditClient.from_client(client) pdf = audit.export_pdf(filter={"case_id": "C-2026-04412"}) with open("case-04412-disclosure.pdf", "wb") as f: f.write(pdf) ``` ## Why the defence can verify this independently - **Every edge has a source.** `WITH PROVENANCE` attaches the originating record, the resolution method, and a confidence score to each connection — not just the connection itself. See [Provenance](/docs/concepts/provenance). - **The export is signed, not printed.** `AuditClient.export_pdf()` produces a hash-chained, HSM-signable bundle, not a screenshot — the same primitive that stands behind [Governance](/docs/concepts/governance)'s audit trail. - **History doesn't silently change.** The bi-temporal model means a later correction to a record doesn't rewrite what the investigator saw on the day they built the case — `AS OF` reproduces exactly that state. See [Bi-Temporal Model](/docs/concepts/bitemporal). - **Purpose-bound access, logged.** `PURPOSE 'investigation_specific:C-2026-04412'` scopes every read to this case and is recorded against the querying principal — the same access-control model that governs every protocol door, not a case-tool-specific bypass. ## See also - [Governance](/docs/concepts/governance) — audit hash chain, `PURPOSE`, and signed exports - [Identity Resolution](/docs/concepts/identity) — `PATHS_BETWEEN` and the identity graph it walks - [Query Cookbook](/docs/reference/query-cookbook) — `ANPR_TRACE`, `CRIME_PATTERN_CLUSTER`, `DISPATCH_PRIORITY`, and the full graph-operator surface - [Telecom: CDR to co-location network](/docs/use-cases/telecom-colocation-network) — the same graph primitives applied to call-detail records ============================================================================== # Maritime: dark-fleet detection from AIS URL: https://relatadb.dev/docs/use-cases/maritime-dark-fleet ============================================================================== # Maritime: dark-fleet detection from AIS ## The problem Vessels evading sanctions have a well-known playbook: turn off the AIS transponder in open water, transfer cargo ship-to-ship, re-flag, rename, and re-appear somewhere else. Catching this today means stitching together an AIS aggregator subscription, a vessel registry, a sanctions screening tool, and a GIS package — four vendors, four exports, and no single place where "this vessel went dark near a sanctioned port, and its beneficial owner is designated" is one traceable answer. ## The scenario An analyst is reviewing a tanker with a history of AIS gaps in the Gulf of Oman. Before escalating to enforcement, they need three things joined together: confirmation the vessel (or a vessel in the same footprint) was actually in the area during the gap, a sanctions screen on the vessel and its registered owner, and the ownership chain traced up to whoever ultimately controls it — because the vessel itself is rarely the sanctioned party. ## Geofence, screen, and trace ownership — one engine `GEOFENCE`, `SANCTIONS_SCREEN`, and `BENEFICIAL_OWNERSHIP_CHAIN` are all SQL-reachable operators against the same governed store — there's no export between the AIS platform and the sanctions tool. ```sql PURPOSE 'maritime:sanctions-investigation' -- 1. What else was moving through the same waters during the AIS gap window? -- (fence is a pre-registered geofence name; from_ts/to_ts are epoch-ns) SELECT * FROM GEOFENCE( 'gulf-of-oman-50km', target_type => 'VesselPositionReport', from_ts => 1773327600000000000, to_ts => 1773597600000000000 ); -- 2. Screen the vessel's registered owner against sanctions lists SELECT * FROM SANCTIONS_SCREEN('Meridian Shipping SA') WITH PROVENANCE; -- 3. Trace beneficial ownership to the natural person or ultimate parent SELECT * FROM BENEFICIAL_OWNERSHIP_CHAIN('Meridian Shipping SA', 7) WITH PROVENANCE; ``` ## Typed SDK snippet ```python from relata import RelataClient with RelataClient( "http://localhost:9090", bearer_token="relata-dev", purpose="maritime:sanctions-investigation", ) as client: # Vessels sharing the geofence during the AIS gap window # (from_ts / to_ts are epoch-ns; fence is a pre-registered geofence name) nearby = client.query( "SELECT * FROM GEOFENCE('gulf-of-oman-50km', " "target_type => 'VesselPositionReport', " "from_ts => 1773327600000000000, to_ts => 1773597600000000000)" ) for vessel in nearby: print(vessel["mmsi"], vessel["last_position"]) # Screen the registered owner hits = client.query("SELECT * FROM SANCTIONS_SCREEN('Meridian Shipping SA')") for hit in hits: print(f"Sanctions hit: {hit['list_id']} ({hit['designation_date']})") # Walk the ownership chain looking for the actual designated party chain = client.query( "SELECT * FROM BENEFICIAL_OWNERSHIP_CHAIN('Meridian Shipping SA', 7)" ) for link in chain: print(f"depth {link['depth']}: {link['owner_name']} ({link['ownership_pct']}%)") ``` ## Why the same engine wins on this workload - **Geospatial, graph, and identity queries are one plan, not three tools.** `GEOFENCE` (S2-cell indexed), the ownership graph, and sanctions screening all run against the same governed store — no CSV round-trip between the AIS aggregator and the sanctions vendor. See the [Query Cookbook](/docs/reference/query-cookbook) for the full operator surface. - **The vessel isn't the sanctioned party — the owner usually is.** `BENEFICIAL_OWNERSHIP_CHAIN` traces through shell-company layers the same way it would for a wire transfer, because it's the same operator, not a maritime-specific reimplementation. - **Every hit carries provenance.** `WITH PROVENANCE` on the sanctions screen and ownership trace means the eventual enforcement package cites exactly which list, which designation date, and which ownership record produced each finding. See [Governance](/docs/concepts/governance). - **Bi-temporal by default.** Flag changes, renames, and ownership transfers are tracked over time, not overwritten — `AS OF` reconstructs the ownership structure as it stood on the date of the incident, not as it stands today. ## See also - [AML: sanctions screening with an audit trail](/docs/use-cases/aml-sanctions-screening) — the same `SANCTIONS_SCREEN` and `BENEFICIAL_OWNERSHIP_CHAIN` operators applied to correspondent banking - [Governance](/docs/concepts/governance) — `PURPOSE`, provenance, and the audit hash chain behind every finding - [Query Cookbook](/docs/reference/query-cookbook) — `GEOFENCE` and the full graph/geospatial operator reference - [Identity Resolution](/docs/concepts/identity) — how `Mmsi` and `ImoNumber` resolve to the same vessel across a flag change ============================================================================== # OSINT: cross-platform identity fusion URL: https://relatadb.dev/docs/use-cases/osint-identity-fusion ============================================================================== # OSINT: cross-platform identity fusion ## The problem A subject of interest doesn't confine themselves to one platform. The same person shows up as a phone number in a leaked database, a handle on one social network, a different handle on another, and possibly a face in a CCTV frame — and open-source tooling today treats each of those as a separate lookup in a separate tool, with the analyst doing the correlation by hand and by memory. That doesn't scale past a handful of subjects, and it leaves no defensible trail for how the correlation was made. ## The scenario An analyst has a single verified phone number for a subject of interest. They need every platform that number — or an identifier chained from it — resolves to, ranked and with a confidence score, and they need to know if it's the *same* underlying identity linking two specific profiles the team has already flagged independently. ## Resolve, cluster, and verify — from one identifier `RESOLVE_IDENTITY`, `IDENTITY_CLUSTER`, and `SAME_IDENTITY` all run against the same `IdentityIndex` that SmartIngest builds automatically at ingest time — there's no separate cross-platform correlation product to license and feed. ```sql PURPOSE 'osint_investigation:CT-2026-0441' -- Every identity value linked to this phone number — handles, emails, wallets SELECT * FROM RESOLVE_IDENTITY('+966501234567', MODE => 'cluster'); -- Does this specific pair of flagged profiles resolve to the same person? SELECT * FROM SAME_IDENTITY('profile-twitter-suspectA', 'profile-telegram-suspectB'); -- Full cluster for a confirmed entity, once the analyst confirms the match SELECT * FROM IDENTITY_CLUSTER('person-441'); ``` ## Typed SDK snippet ```python from relata import RelataClient, IdentityClient with RelataClient( "http://localhost:9090", bearer_token="relata-dev", purpose="osint_investigation:CT-2026-0441", ) as client: id_client = IdentityClient.from_client(client) # Every identity value this phone number resolves to, across platforms cluster = id_client.cluster("+966501234567") for identity in cluster: print(identity) # Verdict: do these two independently-flagged profiles belong # to the same underlying entity? verdict = client.query( "SELECT * FROM SAME_IDENTITY(" "'profile-twitter-suspectA', 'profile-telegram-suspectB')" ) # Expand outward from the confirmed subject to known associates network = client.query( "SELECT * FROM PATHS_BETWEEN('person-441', 'person-associate-118', max_hops => 3)" ) ``` ## Why this replaces five platform-specific lookups - **One identity graph, not five platform APIs.** `IdentityIndex` links every canonical identifier — phone, email, handle, wallet — to the same underlying entity at write time via SmartIngest, so `RESOLVE_IDENTITY` is a single query, not a fan-out across tools. See [Identity Resolution](/docs/concepts/identity). - **The correlation is a verdict, not a guess.** `SAME_IDENTITY` returns a confidence-scored decision on two specific profiles, so an analyst's cross-platform attribution is a reproducible query result, not a judgment call buried in a report. - **The pivot from identity to network is the same engine.** `PATHS_BETWEEN` walks the same graph `RESOLVE_IDENTITY` just populated — there's no export into a separate link-analysis tool once identity resolution is done. - **Every collection act is purpose-scoped and audited.** `PURPOSE 'osint_investigation:CT-2026-0441'` ties every lookup to the case it supports, in the same audit trail that governs every other protected read. See [Governance](/docs/concepts/governance). ## See also - [Identity Resolution](/docs/concepts/identity) — `RESOLVE_IDENTITY`, `IDENTITY_CLUSTER`, `SAME_IDENTITY`, and the 76 canonical identifier types - [Governance](/docs/concepts/governance) — `PURPOSE` scoping and the audit hash chain - [LEA: court-admissible investigation graph](/docs/use-cases/lea-investigation-graph) — the same `PATHS_BETWEEN` expansion applied to a case file - [Cyber: Sigma over governed telemetry](/docs/use-cases/cyber-sigma-detection) — `RESOLVE_IDENTITY` used for account/actor correlation in an incident ============================================================================== # Telecom: from raw CDRs to a co-location network URL: https://relatadb.dev/docs/use-cases/telecom-colocation-network ============================================================================== # Telecom: from raw CDRs to a co-location network ## The problem Call detail records are the highest-volume, lowest-tooled workload in telecom fraud and law-enforcement investigation. A single case can mean tens of millions of rows, and the actual analytical question — "who has this number been in contact with, and who do those contacts also talk to?" — usually ends up answered in a spreadsheet or a $100k/seat proprietary tool, because turning flat CDR rows into a navigable graph is genuinely hard: it needs time-series scan, graph traversal, and identity resolution (a number can be reassigned, a SIM can be swapped) all in the same query. ## The scenario A fraud team has a suspect MSISDN linked to a SIM-swap fraud ring. They need the immediate contact network, ranked by call volume, and they need to resolve any contact number to a known identity — without exporting CSVs between three tools. ## Ingest, then query the graph directly CDR ingest and analysis are native — `CdrRecord` is a first-class governed type, not a staging table you build yourself. ```bash # Ingest a CSV export (columns: caller, callee, duration_secs, timestamp_utc) relata cdr ingest calls.csv --purpose law_enforcement ``` ```sql PURPOSE 'law_enforcement' -- Common-contact analysis: who has this number called or been called by, -- ranked by call volume SELECT callee, COUNT(*) AS call_count, SUM(duration_secs) AS total_secs FROM CdrRecord WHERE caller = '+919876543210' OR callee = '+919876543210' GROUP BY callee ORDER BY call_count DESC LIMIT 20; -- Resolve a contact number to a known identity, if one exists SELECT * FROM RESOLVE_IDENTITY('+447700900123'); ``` ## Typed SDK snippet ```python from relata import RelataClient with RelataClient( "http://localhost:9090", bearer_token="relata-dev", purpose="law_enforcement", ) as client: # Common-contact / hand-off analysis for the suspect number contacts = client.query( "SELECT callee, COUNT(*) AS call_count, SUM(duration_secs) AS total_secs " "FROM CdrRecord " "WHERE caller = '+919876543210' OR callee = '+919876543210' " "GROUP BY callee ORDER BY call_count DESC LIMIT 20" ) # Build the co-location network: for each frequent contact, find # their own frequent contacts, and rank the resulting network by degree network = {} for row in contacts: second_hop = client.query( f"SELECT callee, COUNT(*) AS call_count FROM CdrRecord " f"WHERE caller = '{row['callee']}' " f"GROUP BY callee ORDER BY call_count DESC LIMIT 10" ) network[row["callee"]] = [r["callee"] for r in second_hop] # Resolve any number in the network to a known identity for number in network: identity = client.query(f"SELECT * FROM RESOLVE_IDENTITY('{number}')") if identity: print(f"{number} -> {identity[0]['linked_entity_ids']}") ``` CLI shorthand for the same two-step flow, if you're working interactively: ```bash relata cdr analyze +919876543210 relata cdr timeline +919876543210 ``` ## Why this replaces the bespoke stack - **`CdrRecord` is bi-temporal, not a flat import.** Number portability and SIM reassignment don't corrupt history — a query against last month's data resolves identity as it stood then, via `AS OF`. See [Bi-Temporal Model](/docs/concepts/bitemporal). - **Identity resolution is a query, not a separate lookup service.** `RESOLVE_IDENTITY` runs against the same `IdentityIndex` that SmartIngest built at ingest time — no batch reconciliation job between the CDR table and a subscriber master. See [Identity Resolution](/docs/concepts/identity). - **The graph traversal and the row scan are the same engine.** `DEGREE()` and `PATHS_BETWEEN` operate on the CSR graph layer derived from `CdrRecord` at ingest — building a co-location network is a query, not an ETL job into a separate graph database. - **Governed by default.** `PURPOSE 'law_enforcement'` scopes and audits the query the same way it would for any other protected dataset — there's no separate ACL model for CDR data. ## See also - [Query Cookbook — CDR analysis](/docs/reference/query-cookbook) — the full `CdrRecord` schema, CLI shorthand, and verified query patterns - [Identity Resolution](/docs/concepts/identity) — `RESOLVE_IDENTITY` and the canonical `Msisdn`/`Imei` types - [LEA: court-admissible investigation graph](/docs/use-cases/lea-investigation-graph) — the same `PATHS_BETWEEN` traversal applied to a full case file - [Ingestion & SmartIngest](/docs/guides/ingestion) — how identifiers are auto-detected on write