Connectors & Extensions#

Relata does not fetch from external sources inside the database process. Per the ETL Boundary policy, this repository ships canonical type contracts + ingest trait impls + schema validators only — zero network dependencies in the signed binary. The actual fetching (polling, OAuth, vendor SDKs, webhooks) lives in datagrep, the external ETL tool that consumes these contracts, fetches from each source, and pushes governed rows into Relata. This keeps the binary SLSA-clean and credentials out of the DB process.

Relata's role at the boundary: map incoming data onto the identity model and build the governed graph — canonicalize → detect identities (SmartIngest) → build the graph → attach provenance (PROV-O) on every row.

The extension framework#

Six extension kinds share one framework — one Manifest, one signing chain, one principal + ACL + audit contract:

Three deployment modes:

The Connector trait#

A typed Connector trait in relata-core::connectors enforces the bi-temporal + provenance contract at the connector boundary, preventing untagged rows:

pub trait Connector {
    fn name(&self) -> &'static str;
    fn schema(&self) -> HashMap<String, String>;          // field → canonical-type label
    fn ingest(&self, batch: ConnectorBatch) -> Result<Vec<String>, ConnectorError>;
}

ConnectorBatch carries object_type, a BiTemporalRange, a ProvenanceRef, and Vec<ConnectorRow> — so connectors never need to know the object-store API. ConnectorError has three variants: InvalidSchema, IngestFailed, Io. The reference impls live in relata-connector-stub (NoopConnector for tests, DbtConnector which parses dbt NDJSON model export).

dbt adapter#

The dbt path parses a pre-exported dbt NDJSON model and maps JSON values to CanonicalValue (DbtConnector::parse_ndjson), making it testable without a running dbt instance. Full dbt run integration ships in the Python sdks/python/dbt_relata/ package.

Migration connectors: relata import --from#

Migrate an existing database straight into the governed identity fabric — no CSV export step. Every row lands through POST /ingest/bulk → governed_upsert_many, the same write path every protocol door uses, so SmartIngest identity detection, registered ingest pipelines, ACL, tenant-ownership, and audit logging all apply.

# Postgres is a real, wired connector
relata import --from postgres \
  --dsn "postgresql://user:pass@localhost:5432/appdb" \
  --table users --type Person \
  --dry-run                 # prints ≤5 mapped rows, opens no write transaction

Key v1 limitations: --type must name an already-registered governed type (no DDL); TLS is not wired (NoTls); quoted/mixed-case identifiers rejected; no relationship/FK import yet (one table → one governed type per invocation).

Connector catalog#

~120 catalogued type contracts across telco, social, news, email, documents, threat-intel, cyber telemetry, cloud audit, financial/AML, blockchain, identity/KYC, geospatial, and streams. Three-tier status per entry:

• ✅ live — registered + fetches (noop, dbt; MISP/TAXII/STIX bundle import via POST /import?format=stix)
• 🟡 registered stub — ?connector=<name> resolves; returns a clear not yet implemented (never a silent no-op); fetch lives in datagrep
• 🔲 catalog spec — canonical type contract documented for datagrep; not yet registered

Highlights that are live in this repo (not through datagrep):

The full catalog (telco IPDR/tower-dump/SMS/EDR, OSINT 13, FHIR R4, AIS stream, FININT 40, cloud audit, cyber telemetry, etc.) is documented as canonical type contracts in the source repo's connector catalog. For sources not in the catalog, author a custom connector as a relata-connector-<name> crate implementing Connector.

Ontology packs (domain packs)#

Packs bundle a domain's ontology types, detectors, jobs, reports, and detection rules into a signed, versioned unit. The repo ships ~20 domain packs + ~25 jurisdiction packs; the portal's use-cases map to these:

relata-pack-stub demonstrates the pack layout for authoring new packs.

Field-mapping convention (all connectors)#

Every connector follows the same canonicalization recipe:

1. Parse the source field to its native type.
2. Validate against the canonical type's checksum/format/registry (76 canonical types — email, IBAN, MMSI, VIN, IMEI…).
3. Canonicalize to the binary representation (uint32 for IPv4, E.164 uint64 for phone, S2 cell for geo…).
4. Attach provenance — (source_connector_id, file_or_batch_id, record_offset, observed_at, recorded_at).
5. Index in IdentityIndex with observed_in = (object_type, object_id, property_path).
6. Quarantine on validation failure (strict reject / permissive auto-create / auto-map suggest).

See also#

• Ingestion & SmartIngest — Kafka, CDR, ingest pipelines, relata import
• Identity — canonical types and entity resolution
• Jobs, Workflows & Detection — the Job/Report extension kinds
• SQL Reference — DETECT_IDENTITIES, RESOLVE_IDENTITY, domain TVFs