SIEM/SOC migration and cutover

Relata DB 3.0
This page describes Relata DB 3.0, the current release. Paths use the new /v1 prefix; see the route restructure.

This is the procedure for moving event, telemetry, threat-intelligence and detection-rule data into Relata DB 3.0 (one node, one tenant, local storage).

Rehearsed, not qualified
One synthetic dry run and one cutover rehearsal exist. No customer or SIEM-product export has been migrated, so no source path here is claimed as qualified for production. Rehearse with your own data before you cut over. An in-place 2.7.1 to 3.0 data-directory upgrade is not tested and is not what this procedure covers.

Unsupported: read before you start

Not supportedWhat to do instead
Native SIEM formats (CEF, LEEF, syslog, Windows EVTX/XML, Splunk/Elastic/Sentinel index or archive formats, pcap)Relata ships no parser. Convert to JSON, NDJSON or CSV in your own pipeline first; that conversion is your responsibility and part of your reconciliation.
Dashboards, saved searches, correlation searches, notable-event state, case history from another SIEMNot importable. Only rows you export as data (and rules, below) can be loaded.
Sigma beyond a converter's supported subsetConvert with an OpenETL Sigma converter, which should fail with a translation error; port the rule as a governed rule via POST /v1/rules. No full Sigma conformance claim.
OTLP/protobuf and OTLP partial-success repliesOTLP/HTTP JSON only.
A TAXII server, or push-style TAXIIRelata serves no TAXII endpoints; polling is an OpenETL source.
Drop-in wire compatibility with the retired PostgreSQL, S3, Flight, Neo4j and Bolt listenersUse HTTP/JSON or the SDKs; see protocol retirement.
Source event time as engine valid time through /v1/ingest/autoThe source timestamp is stored as an ordinary column. Valid and system time are set by the engine. Decide how queries select time before cutover.
Type enforcement on the schemaless route/v1/ingest/auto infers column types from the batch and does not reject later rows with a different type. Detect this in the inventory.
Idempotent re-load/v1/ingest/auto has no idempotency key; resending a batch appends duplicates.
Nested objects and arrays as typed columnsStored as compact JSON text. Flatten fields you filter on.
A universal write fence, cross-node rollback, reverse migrationNone. Fence in the source system; roll back by discarding the destination or with an application-owned reverse transfer.
Multi-tenant, clustered or HA targetsEnterprise scope.

Paths, formats, limits and errors

Limits: 64 MiB per ingest request body, 128 MiB for OTLP and document bodies, 1 MiB for query bodies. There is no documented per-request row cap; size batches well under the body limit (the dry run used 100 rows). Writes pass the shared ingest queue; a full queue returns 429 with Retry-After and nothing is queued. All routes need a registry token and a purpose. Errors are RFC 9457 problem details.

SourceInputRoute or toolBehaviour
Event and log recordsJSON rows with object_type, rows, purposePOST /v1/ingest/autoCreates the type on first write; infers text, int, float, bool per field; nested values become JSON text; adds created_by, _pk and automatic embedding columns. 401 bad token, 422 body shape, 413 too large, 429 queue full.
Records into registered typesJSON objects with _type, on_conflict of upsert, skip or error, optional queuedPOST /v1/ingest/bulkThe type must exist. A missing _type rejects the whole request with 400 missing-type. Synchronous unless queued: true; poll GET /v1/ingest/tasks/{task_id}. A queue acknowledgement is not a durable-write confirmation.
CSV, JSON array, NDJSONtext/csv, application/jsonPOST /v1/ingest, relata ingest <file.csv> --type TRegistered type required.
OTLP traces, logs, metricsOTLP/HTTP JSONPOST /v1/{traces,logs,metrics}One generic type per signal; needs RELATA_OTLP_ENABLED.
Threat intelligenceSTIX 2.1 bundlePOST /v1/ingest/import?format=stixA supported object subset; unsupported objects are skipped and counted.
Detection rulesSigma YAML (subset)OpenETL conversion, then POST /v1/rulesTranslation errors come from the converter.
Export for reconciliation or rollbackn/aGET /v1/export?type=T&format=csv|json|jsonl|stixReads current visible rows.

Procedure

Use a disposable destination for steps 2 to 5. Keep the source intact and read-only until the acceptance window closes.

  1. Inventory the source. Record counts per source and type, the unique-key field and its distinct count, per-field value types (flag mixed types, nested values, nulls), the time field and format, and file or export checksums. Stop if the key is not unique.
  2. Convert to a supported format. Produce JSON rows with one stable key per record, deterministically so it can be re-run.
  3. Dry run. Start a throwaway data directory with no other writer and load in bounded batches, recording each acknowledgement. Do not retry a batch blindly: a repeat through /v1/ingest/auto duplicates rows. After any ambiguous failure, discard the destination and start again.
  4. Reconcile. Compare, and record: source count against SELECT COUNT(*); the key sets (missing and unexpected); a per-record checksum over the mapped fields (nested value to compact JSON text, nulls omitted) against SELECT * pages with the engine's added columns excluded; and a combined set checksum. Page with ORDER BY key LIMIT n OFFSET m. Investigate every difference; stop on any unexplained one.
  5. Failure rehearsal. Send a malformed batch and a request with an invalid token; confirm the count is unchanged and the documented status codes return.
  6. Fence and cut over. Announce a maintenance window. Stop or fence source writers with the source system's controls and record the final watermark. Load the final delta into the production destination (a fresh data directory is safest; do not re-run batches into a populated type), reconcile again, stop the server, run relata backup and relata verify-backup, restart, and only then point readers and writers at Relata.
  7. Rollback. Before any production write is accepted: stop the destination, discard its data directory and resume the source, which was never modified. After production writes are accepted there is no general reverse migration. Export what you need with GET /v1/export, reverse it with your own tooling and reconcile that transfer the same way. Keep the source and a verified backup until the acceptance window ends.

A backup taken after production writes begin is a restore point, not a reverse migration. See Backup and restore, including the open qualification caveat.

Evidence

A script in the source repository loads 1,000 synthetic authentication events (including a nested object, an array, an explicit null and a mixed-type value) through POST /v1/ingest/auto in ten batches into a disposable server. It asserts equal counts, key sets and checksums, the 400 and 401 failure paths leaving the count unchanged, and a rollback that leaves the source file byte-identical. A second script extends it with backup, verify, corrupted-artefact refusal, simulated loss, restore, re-reconcile and rollback. Both passed on one macOS debug binary, once each.

Not shown: a customer or SIEM-product source, more than 1,000 records, concurrent writers, engine valid-time mapping, crash or disk-full behaviour during a load, Sigma, STIX or OTLP source paths, Linux, or the release build. The label stays "rehearsed, not qualified" until an operator-supplied real export is migrated by someone other than the author against a release-candidate binary.

Supported-paths matrix

For the retired protocols, Kafka rows, prepared documents, package upgrades and local garbage collection, use this common sequence: inventory; choose a documented path; isolated dry run; reconcile counts, ids, selected fields, tenant and policy visibility and provenance; fence and cut over in a maintenance window; keep the source and a verified backup until acceptance ends. The database supplies no universal migration fence and no cross-node rollback. State irreversible steps before starting, and retain the evidence (revisions, hashes, inventory, commands, counts, checksums, refusals, the final source watermark and the rollback decision).

PathBoundary
Retired public protocols to HTTP/JSON and the SDKsMap each operation, adapt semantics, validate filters, credentials, pagination, acknowledgements and retention. Application-side differences need an app-owned reverse procedure.
Prepared-document ingestionKeep preparation in your adapter; map content, model version, ontology links, ACLs and provenance to governed rows; inspect task receipts for terminal or uncertain states. Receipts do not reconstruct queued input or certify indexing.
Legacy Kafka rowsOne broker, tenant, source, topic, partition and data directory, into a fresh destination. No keyless deduplication, multi-partition, disk-loss or exactly-once guarantee.
Legacy local layout or package upgradePreserve a verified pre-upgrade backup and the old binary. A narrow two-row fixture is an example, not qualification of other layouts.
Offline native local blob garbage collectionLocal shared filesystem, all owners stopped, minimum grace, verified orphan uploads. Cloud, live and old-scope collection are unsupported.