SmartIngest#

SmartIngest is RelataDB's identity-resolution pipeline. It runs on the lazy side of ingest (see Data Flow) and is what turns a pile of records from different sources into one connected identity graph — without you writing matching logic.

This page explains its role, how it works, and — just as importantly — what it deliberately does not do.

The role: "who is this, really?"#

The same entity shows up across your sources wearing different disguises: an email at signup, a phone number in a call log, an IBAN in a transfer, an account ID in billing. Without identity resolution, these are unrelated strings in unrelated tables.

SmartIngest recognizes each identifier, validates it, and fuses records that share a validated identifier into one cluster — even when they arrived through different doors. The graph that powers PATHS_BETWEEN, PageRank, and community detection forms itself out of these fused identities.

How it works: deterministic and checksum-gated#

SmartIngest is not a probabilistic matcher. It is a strict-first, checksum-gated detection pipeline.

raw text / cell value
   │
   ▼
tokenize  ──►  per-token shape gate (regex)
                   │ pass │ fail ──► skip
                   ▼
              format + checksum validate
                   │ pass │ fail ──► skip
                   ▼
              DetectionHit (CanonicalKind + Identity)
                   │
                   ▼
              IdentityIndex (bloom-pruned lookup)
                   │
                   ▼
              identity cluster (fuse same entity)

Each canonical type has its own gate plus a validator. The pipeline tries types in order of certainty:

1. Checksum-gated types — Aadhaar (Verhoeff), PAN (Luhn), IBAN (mod-97), IMEI, VIN. A value that fails its checksum is skipped. This keeps false positives near zero on identifiers that carry integrity digits.
2. Regex-only types — emails, E.164 phone numbers, MAC addresses, URLs, domains.
3. Heuristics — last, with a string fallback.

Because the inputs are checksum-validated, the fusion is high-precision: you trust the merge because you trust each link.

What gets detected (and what does not)#

The honest boundary: RelataDB does identifier extraction, not general named-entity recognition (NER). There is no in-tree spaCy/BERT/transformer. This is a deliberate design choice — identifiers are verifiable, names in prose are not.

When you need general NER#

For names-in-prose, intent, or sentiment, you register an external scorer (configured via an accel endpoint). RelataDB takes that scorer's output and writes it back as governed, provenance-stamped typed assertions, then fuses it into the identity graph. RelataDB hosts the governed graph and identity layer; you bring the model for the fuzzy parts.

From detection to a resolvable identity#

Source A                    Source B
├─ alice@x.com              ├─ alice@x.com
└─ +14155550111             └─ device D7
       │                          │
       └────────┬─────────────────┘
                ▼
        IdentityIndex fuse
                │
                ▼
   one identity cluster: Alice = {email, phone, device}

Three SQL operators consume the result:

• RESOLVE_IDENTITY(value) — returns the full cluster for an identifier.
• IDENTITY_CLUSTER(value) — expands a value into its cluster.
• SAME_IDENTITY(a, b) — a real-time gate: are these two values the same entity?

Where declared columns fit in#

SmartIngest also handles the eager path for declared columns. When you declare a property as Identity on your ontology, that column is validated and canonicalized on the write hot path (synchronously), and an entry is written into the universal IdentityIndex:

IdentityIndex: (CanonicalKind, bytes) → (object_id, source_table, source_column, observed_at)

One index, every observation, every source. Free-text fields are mined lazily; declared identity fields are typed eagerly. Either way they end up in the same resolvable graph.

Why this design pays off#

• Cross-source fusion is cheap. The same phone number written by three feeds has one byte representation, so joins need no LOWER()/TRIM()/normalization in the predicate.
• You trust the merge. Checksum validation makes each link high-precision; the cluster inherits that trust.
• Upgrades are safe. Because detection runs lazily as materialized views, improving a detector triggers an MV refresh — not a source backfill.

See also#

• Data Flow — where SmartIngest sits in the eager/lazy split.
• Identity — the umbrella type and resolution operators in depth.
• Ingestion & SmartIngest — the operational guide (CLI, HTTP, endpoints).
• How It Works · Data Model