Local Chat Is Not Data Residency: RAG, Embeddings, and Vector Stores

The companion note on this site covers local and peer-to-peer inference: Ollama, llama.cpp, Petals, model cards, and why residency is not a slogan. This note goes one layer deeper.

Most “AI at work” stacks also embed documents, store vectors, and retrieve chunks into the prompt — retrieval-augmented generation (RAG). If the embedding hop or the vector index leaves the building, a local chat UI has not bought residency. It has bought a familiar interface over a still-leaky trust model.

What RAG actually moves

Lewis et al.’s 2020 RAG paper (arXiv:2005.11401) remains the clearest systems statement: a parametric generator paired with a non-parametric memory — a dense vector index over document chunks — accessed by a neural retriever. Their Wikipedia setup split articles into ~100-word chunks, embedded each chunk, and used a MIPS/FAISS-style index for nearest-neighbour retrieval before generation.

Production stacks still follow the same spine:

1. Ingest — parse files, split into chunks (by tokens, paragraphs, or sliding windows), optionally keep metadata (source path, page, ACL tags).
2. Embed — map each chunk to a fixed-length float vector with an embedding model.
3. Index — store vectors (and usually the raw text) in a vector store that can answer approximate nearest neighbours.
4. Retrieve — embed the user query with the same embedding model, pull top-k chunks, stuff them into the generator’s context.
5. Generate — the local (or remote) LLM answers conditioned on those chunks.

The residency claim fails at whichever step first phones home. A laptop that runs llama-server for chat while posting every PDF to a hosted embedding API still exports the corpus — often more of it than a single chat prompt would.

Frame: Inference residency is “prompts and completions stay here.” Retrieval residency is “chunk text, embeddings, and the index stay here too.” Marketing usually sells the first and ships the second as a default cloud sidebar.

Local embeddings: concrete mechanisms

Embedding models are encoders trained to put related text near each other in vector space — not chat models with the temperature turned down. Ollama’s docs put typical lengths around 384–1024, with L2-normalised outputs from /api/embed, and a production rule that is easy to violate: use the same embedding model for indexing and querying.

Two local runtimes expose this over HTTP:

• Ollama — POST /api/embed with model and input (string or array), optional truncation/dimensions, batchable; bits stay on localhost if the daemon does (capabilities, API).
• llama.cpp llama-server — OpenAI-compatible chat, completions, and embeddings. Flags --embedding / --embeddings restrict to dedicated embedding GGUFs; --pooling (none, mean, cls, last, rank) controls how token states collapse. OpenAI-shaped route: POST /v1/embeddings. Native /embedding / /embeddings cover non-OAI cases, including pooling none for token-level outputs (server README).

On constrained SA / Global-South hardware, residency often means a small embedding GGUF that fits CPU RAM (or a modest GPU) beside the index, with a larger chat model optional. Quantisation here is fit, not fashion. Pin the embedding artefact you indexed with: swapping models mid-corpus silently corrupts nearest-neighbour geometry.

Where vectors live: on-disk stores versus someone else’s disk

A vector store is just durable nearest-neighbour search plus metadata. The residency question is whether that durability is local filesystem or a remote multi-tenant service.

Primary docs make the local path explicit:

• Chroma’s PersistentClient(path=...) saves and reloads database files on the machine you choose; the default path is .chroma if you omit one. The same product line also offers CloudClient — which is fine when you intend cloud, and a trap when a tutorial’s “quickstart” quietly selects it (Chroma clients).
• sqlite-vec is a SQLite extension for float/int8/binary vectors in vec0 virtual tables, written in pure C with no heavyweight deps, aimed at “runs anywhere SQLite runs” — including lean Linux boxes and constrained devices (README). That is residency as a single encrypted disk file if your OS and backup story are sound.
• LanceDB documents connecting to a local path in OSS, creating tables with a vector column, and running similarity search (with optional filters) against that on-disk Lance layout — distinct from Enterprise db:// remote URIs (tables docs).

None of these erase operational risk. An unencrypted laptop with a local index of clinic notes is still a laptop. Residency removes provider exposure for that index; it does not invent access control, disk encryption, or malware resistance.

Chunking and retrieval without mystique

Chunking is a systems trade-off: too large and the embedding averages mixed topics; too small and discourse structure vanishes; overlap windows reduce boundary loss at the cost of index size. At query time the stack embeds the question, searches (often cosine similarity on normalised vectors), takes top-k, optionally reranks (llama-server exposes rerank with a reranker model and --pooling rank), then stuffs surviving chunks into the chat template.

Failure modes are boring and serious: wrong chunk → confident wrong answer; over-broad ACL → one user retrieves another’s documents; stale index → yesterday’s policy PDF still “wins.” Local does not fix hallucination. It fixes where the wrong answer’s source text was processed.

Same API shape ≠ same trust model

Here is the confusion that ships in half the demos.

OpenAI’s embeddings guide describes hosted POST /v1/embeddings with input, model, and optional encoding_format / dimensions — requests that leave your network for their infrastructure (guide). llama.cpp’s server deliberately mirrors that shape: same path family, same JSON fields, even a Bearer token clients expect. Ollama’s /api/embed is a sibling local pattern.

Compatibility is a gift for developers and a hazard for threat models. Point base_url at http://127.0.0.1:8080/v1 and bytes can stay local. Point the same SDK at https://api.openai.com/v1 (or any hosted embedding/vector SaaS) and the knowledge base can leave in five familiar lines. The UI may still say “Local LLM.” The network capture will not.

Treat OpenAI-compatible local servers as wire-format adapters, not as proof the trust boundary matches a frontier API. Ask: which host resolves? Does the vector URI look like a filesystem path or https://…? Does the framework’s default “memory” backend phone home?

Why this matters in SA and the Global South

POPIA-era data protection, cross-border transfer friction, and metered bandwidth all push the same way: do not ship the corpus if you do not have to. A newsroom archive, clinic protocol set, municipal tender library, or civic casefile should not become residue on a foreign GPU farm because someone installed a pretty chat skin.

Civil-liberty stakes are concrete. Embeddings of investigative notes, attorney–client drafts, or organiser sheets are not “just vectors” — they are a lossy but recoverable shadow of the source. Local embedding plus an on-disk index keeps that shadow inside a jurisdiction you can reason about. Cloud RAG defaults invert the burden: trust contracts and subprocessors for every ingest batch, not only the final reply. When the WAN is ugly, a local encoder and SQLite/Lance/Chroma path still answers from yesterday’s index; remote embeddings do not.

What this note deliberately does not do

It does not rank products, cloud credits, or GPUs. It does not give investment, token, custody, or trading advice. It does not claim local RAG is accurate, uncensored, or immune to prompt injection, poisoned documents, or insider misuse. It does not prescribe a chunk size, embedding model ID, or compliance checkbox for any organisation. It does not teach how to evade lawful process. It does not treat OpenAI-compatible APIs as endorsement of any vendor.

Educational takeaway: local generation without local retrieval is cosplay residency. Embed on hardware you control, store vectors on disk you control, keep the query path on the same trust island — and treat “/v1/embeddings works” as a wiring detail, not a security property.

Related on CryptoDecentral

• Decentralised / local AI pillar
• Local and Peer-to-Peer AI: Data Residency Without the Hype
• Network metrics

Further reading (primary)

• Lewis et al. — Retrieval-Augmented Generation (arXiv:2005.11401)
• Ollama — Embeddings · POST /api/embed
• llama.cpp HTTP server README (embeddings, pooling, OpenAI-compatible routes)
• Chroma clients — PersistentClient vs CloudClient
• sqlite-vec
• LanceDB — basic table operations / local connect
• OpenAI — embeddings guide (contrast: hosted /v1/embeddings trust model)