Observability#

RelataDB emits structured logs, Prometheus metrics, OpenTelemetry traces, and health/readiness probes from a single binary with no external dependencies. Every layer is opt-in except logging, which is always on.

Structured logging#

Set log format to json in production so log shippers can parse fields directly:

RELATA_LOG_FORMAT=json RELATA_LOG_LEVEL=info relata serve

A JSON log line carries ts, level, target, msg, and request-scoped fields (request_id, tenant, principal) when a request is in context. Feed into any log shipper (Loki, Elasticsearch, CloudWatch) without a parsing plugin.

Health and readiness probes#

Wire these into your load balancer and container orchestrator:

# Liveness — always 200 if the process is running
curl http://localhost:9090/health
 
# Readiness — 503 (problem+json) if any of 12 conditions fail
curl http://localhost:9090/health/ready
 
# Profile, role, query quota
curl http://localhost:9090/status
 
# Build info: version, git SHA, build time, profile
curl http://localhost:9090/version

The readiness response (200):

{
  "status": "ready",
  "profile": "server",
  "node_id": "node-1",
  "queue_depth_pct": 0,
  "replication_lag": 0,
  "uptime_secs": 642,
  "node_count": 1,
  "license_tier": "server"
}

When a condition fails, the endpoint returns 503 with an RFC 7807 application/problem+json body whose type names the failing check. Common failure reasons:

Route traffic only to nodes returning 200 from /health/ready.

Prometheus metrics#

/metrics serves operational counters in Prometheus text format. Key metrics:

By default /metrics requires a bearer token (fail-closed). For Prometheus scrapers that authenticate at the network layer (NetworkPolicy, mTLS sidecar):

RELATA_METRICS_PUBLIC=true relata serve

Prometheus scrape config:

scrape_configs:
  - job_name: relatadb
    static_configs:
      - targets: ["relata:9090"]
    # Remove bearer_token if RELATA_METRICS_PUBLIC=true
    bearer_token: "<RELATA_BEARER_TOKEN>"
    metrics_path: /metrics
    scrape_interval: 15s

Alert on relata_audit_chain_valid == 0 — that is a security event requiring immediate investigation.

OpenTelemetry traces#

Set RELATA_OTLP_ENDPOINT to export spans. When the variable is unset, OpenTelemetry is fully disabled — zero overhead, no threads, no allocations.

RELATA_OTLP_ENDPOINT=http://otel-collector:4318/v1/traces \
RELATA_OTLP_SAMPLE_RATIO=0.1 \
relata serve

Use 1.0 temporarily when debugging a specific request flow; drop back to 0.01 for steady-state production to avoid overhead.

docker-compose with OpenTelemetry Collector#

A working local observability stack:

services:
  relata:
    image: ghcr.io/relatadb/relata:latest
    environment:
      RELATA_PROFILE: server
      RELATA_BEARER_TOKEN: "change-me"
      RELATA_LOG_FORMAT: json
      RELATA_LOG_LEVEL: info
      RELATA_OTLP_ENDPOINT: "http://otel-collector:4318/v1/traces"
      RELATA_OTLP_SAMPLE_RATIO: "0.1"
      RELATA_METRICS_PUBLIC: "true"
    ports:
      - "9090:9090"
    depends_on:
      - otel-collector
 
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    volumes:
      - ./otel-collector.yaml:/etc/otel-collector.yaml
    command: ["--config=/etc/otel-collector.yaml"]
    ports:
      - "4318:4318"   # OTLP/HTTP receiver
      - "8888:8888"   # Collector metrics
 
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yaml:/etc/prometheus/prometheus.yml
    ports:
      - "9091:9090"

Minimal otel-collector.yaml:

receivers:
  otlp:
    protocols:
      http:
        endpoint: "0.0.0.0:4318"
 
exporters:
  logging:
    verbosity: detailed
 
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [logging]

Minimal prometheus.yaml:

scrape_configs:
  - job_name: relatadb
    static_configs:
      - targets: ["relata:9090"]
    metrics_path: /metrics
    scrape_interval: 15s

Correlation IDs#

Every request gets an auto-generated X-Request-ID (UUID v7, serve.rs:13248). Pin your own to trace a specific request end-to-end:

curl -H "X-Request-ID: $(uuidgen)" \
  -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \
  http://localhost:9090/query \
  -d '{"sql":"SELECT * FROM Person LIMIT 1"}'

The server stamps the ID on error responses. RFC 7807 application/problem+json bodies carry request_id so a user-visible error can be traced through logs, spans, and the audit chain without a session replay.

Audit chain verification#

# Quick check
curl -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \
  http://localhost:9090/audit/count
# { "entries": 1248, "chain_valid": true }
 
# Deep check: chain + WAL + object-store config
relata check

chain_valid: false means an audit entry was modified or deleted. Treat it as a security event: isolate the node, preserve the WAL directory, and open an incident.

The relata_audit_chain_valid Prometheus metric exports this as a gauge — alert on == 0.

Profiling (CPU / heap)#

Production profiling is off by default and admin-gated — a CPU profile leaks workload shape and resolved symbol names, so it is never silently on.

RELATA_PPROF_ENABLE=true RELATA_ADMIN_TOKEN=changeme relata serve
 
# capture 15s of CPU as a pprof protobuf
curl -H "Authorization: Bearer changeme" \
  "http://localhost:9090/debug/pprof/profile?seconds=15" -o cpu.pprof

/debug/pprof/profile returns Google's standard pprof protobuf (application/octet-stream, cpu.pprof) — not a server-rendered SVG. Render it with the standard toolchain:

go tool pprof -http=:8080 cpu.pprof   # interactive flamegraph / top / source view
# or drop cpu.pprof into https://speedscope.app for a no-install browser view

GET /debug/pprof/heap reports the coarse memory counters the store already tracks (implemented: false — a real per-allocation-site heap profile needs a jemalloc build with --enable-prof; the default allocator is mTLS mimalloc). Bounds: ?seconds= is clamped to [1, 30]; only one profile may run per process at a time (a second gets 429).

Operational debug endpoints#

# Get engine stats without parsing Prometheus format
curl -H "Authorization: Bearer $RELATA_BEARER_TOKEN" \
  http://localhost:9090/metrics.json | jq .

MCP observability tools#

The MCP surface exposes operational tools for agent runbooks:

See also#

• Configuration — logging and OTLP env vars
• Auth & Security — audit hash chain, security events
• Backup & Restore — WAL durability model