LLM & Embedding Configuration#

Relata has two separate model-touching surfaces, and they are configured independently. This page is the single canonical reference for both — consolidating settings that were previously scattered across Configuration, Environment Variables, Search, and Agent Memory.

Crucial invariant (since v1.1): the ingest hot path never calls a model. Ingest is a pure validate → WAL → store loop bounded by disk I/O. LLM/embedder calls are either caller-supplied (vectors come in the row payload) or asynchronous (the media-worker drain populates _emb_* off the request thread). The built-in CPU embedder is query-side only.

LLM endpoint (natural-language query)#

Point Relata at any OpenAI-compatible /v1/chat/completions endpoint to enable the nl_query and interpret MCP tools (natural-language → governed SQL → execute, with a deterministic fallback when unset).

# Ollama (local)
export RELATA_LLM_URL=http://localhost:11434/v1/chat/completions
export RELATA_LLM_MODEL=llama3.1
 
# vLLM (self-hosted GPU server)
export RELATA_LLM_URL=http://vllm-host:8000/v1/chat/completions
export RELATA_LLM_MODEL=meta-llama/Meta-Llama-3.1-8B-Instruct
 
# LM Studio (local)
export RELATA_LLM_URL=http://localhost:1234/v1/chat/completions

Every NL-translated query still runs through the governed path (ACL, PURPOSE, cell masking, tenant scoping). The deterministic fallback when RELATA_LLM_URL is unset keeps the tool working offline.

Embedder sidecar (vector embeddings)#

Vector search on text, image, audio, and video needs _emb_* fields. Since v1.1 you have two ways to populate them:

1. Pre-computed (recommended for throughput) — include _emb_text / _emb_image / _emb_audio / _emb_video (float arrays) directly in the row payload at ingest time. No model call is made.
2. Sidecar + media-worker drain — set RELATA_ACCEL_ENDPOINT to an external embedder; the media-worker drain cycle populates _emb_* asynchronously after the write returns. This is the only path that calls the embedder automatically, and it runs off the request thread.

The built-in CPU embedder (128-dim, deterministic) is query-side only since v1.1: recall() uses it to embed the search query when no sidecar is configured. It is not invoked on ingest.

Quick start#

# Point the server at a sidecar
export RELATA_ACCEL_ENDPOINT=http://localhost:8200
cargo run -p relata-cli -- serve

The server probes the sidecar on startup and logs the model tag.

Sidecar API contract#

The sidecar must implement HTTP endpoints returning JSON. The server waits up to 30 seconds per batch call. Empty "embeddings": [] is the correct response for a modality the sidecar doesn't support (the row is stored without that vector rather than failing).

MODALITY → endpoint mapping#

SIMILAR TO … MODALITY <m> and similar_multimodal(modality="…") route to the corresponding _emb_* field:

Environment variables#

Model tag and re-indexing#

The server derives a model tag from the first successful /embed response (include a "model" field to set it explicitly; otherwise "default"). The tag namespaces the HNSW index — changing the model after data has been ingested requires a re-index pass or a fresh store.

Error handling#

• Non-2xx → server logs a warning and continues without that embedding; the row is stored text-only (still BM25-searchable).
• Connection refused / timeout → circuit breaker opens after 3 consecutive failures; server logs "embedder circuit open" and stops calling the sidecar for 60 s.

Reference sidecar (Python)#

A minimal sidecar using sentence-transformers for text and open_clip for images:

# sidecar.py  —  pip install flask sentence-transformers open_clip_torch Pillow
from io import BytesIO
import torch
from flask import Flask, request, jsonify
from sentence_transformers import SentenceTransformer
import open_clip
 
app = Flask(__name__)
text_model = SentenceTransformer("all-MiniLM-L6-v2")          # 384-dim text
clip_model, _, clip_preprocess = open_clip.create_model_and_transforms(
    "ViT-B-32", pretrained="openai")
clip_model.eval()
 
@app.post("/embed")
def embed():
    texts = request.json["texts"]
    vecs = text_model.encode(texts, normalize_embeddings=True).tolist()
    return jsonify({"embeddings": vecs, "model": "all-MiniLM-L6-v2"})
 
@app.post("/embed-image")
def embed_image():
    from PIL import Image
    items = request.json["items"]
    embeddings = []
    for byte_ints in items:
        img = Image.open(BytesIO(bytes(byte_ints))).convert("RGB")
        tensor = clip_preprocess(img).unsqueeze(0)
        with torch.no_grad():
            vec = clip_model.encode_image(tensor)
            vec = vec / vec.norm(dim=-1, keepdim=True)
        embeddings.append(vec[0].tolist())
    return jsonify({"embeddings": embeddings, "model": "ViT-B-32:openai"})
 
# Return empty lists for modalities you don't support:
@app.post("/embed-audio")
def embed_audio(): return jsonify({"embeddings": []})
@app.post("/embed-video")
def embed_video():  return jsonify({"embeddings": []})
 
if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8200)

FROM python:3.12-slim
RUN pip install --no-cache-dir flask sentence-transformers open_clip_torch Pillow
COPY sidecar.py /app/sidecar.py
CMD ["python", "/app/sidecar.py"]

Verify end-to-end#

Option A — caller-supplied (no sidecar): send the vector in the row payload; _emb_text is honoured verbatim.

curl -s -X POST http://localhost:9090/ingest \
  -H "Content-Type: application/json" \
  -d '[{"_type":"Note","body":"The HNSW index is seeded","_emb_text":[0.12,0.34,0.56,0.78]}]'

Option B — sidecar + media-worker drain: start the sidecar, set RELATA_ACCEL_ENDPOINT, ingest a media row; the worker embeds it asynchronously (typically sub-second).

docker run -p 8200:8200 relata-sidecar &
export RELATA_ACCEL_ENDPOINT=http://localhost:8200
cargo run -p relata-cli -- serve &
 
curl -s -X POST http://localhost:9090/ingest/media?modality=image \
  -H "Content-Type: application/json" \
  -d '{"_type":"Photo","image_b64":"<base64-bytes>"}'
sleep 2  # wait for the drain cycle
curl -s http://localhost:9090/query \
  -H "Content-Type: application/json" \
  -d '{"query":"SELECT _emb_image FROM Photo LIMIT 1"}'

If _emb_image is non-null after the drain window, the sidecar is working. If it stays null, check the server log for "embedder circuit open" or "embed timeout".

Embedding model migration#

Changing the sidecar model after data has been ingested changes the model tag and the vector dimension, which breaks similarity search over existing rows. The migration story: re-index (relata embed --type=<T>) or start a fresh store. See the source repo's embedding-model-migration guide for the full procedure.

See also#

• Search & Retrieval — HYBRID_SEARCH, SIMILAR TO, vector operators
• Vector Index Parameters — HNSW / DiskANN tuning
• Agent Memory — recall runs on the same BM25 + vector pipeline
• Configuration — every other runtime knob
• Environment Variables — the full env-var reference