Check yourself: local AI, tools, and residency

Questions and answers, readable without JavaScript.

1. If a model runs locally, does that mean nothing leaves the machine?

   • No. Once tools, web search, function calling, or MCP servers are in the loop, inference residency is not action residency
   • Yes, local weights close the trust boundary
   • Yes, unless the model is over 7B
   • Only cloud models can make network calls
   Answer 1

   No. Once tools, web search, function calling, or MCP servers are in the loop, inference residency is not action residency

   Local sampling can keep prompts and completions on hardware you control. Tools and MCP move side effects — filesystem reads, HTTP, subprocesses — off that closed box.

   Read the note → · Inference Residency Is Not Action Residency: Tools, MCP, and Local LLMs

2. In a RAG stack, what else has to stay local for the residency claim to hold?

   • Only the chat window
   • Only the PDF filenames
   • The model card on a public hub
   • The embedding step and the vector index, not just the generator
   Answer 2

   The embedding step and the vector index, not just the generator

   If the embedding hop or the vector index leaves the building, a local chat UI has not bought residency. Retrieval residency means chunk text, embeddings, and the index stay here too.

   Read the note → · Local Chat Is Not Data Residency: RAG, Embeddings, and Vector Stores

3. Is a model tag a guarantee of which engine runs it?

   • Yes. The tag names the kernel
   • Yes on Linux, no on Mac
   • No. Ollama's v0.40.0-rc0 made supported Apple Silicon architectures run on MLX by default under the same pull/run tags
   • Only if you also hash the GGUF
   Answer 3

   No. Ollama's v0.40.0-rc0 made supported Apple Silicon architectures run on MLX by default under the same pull/run tags

   The runner note's thesis: a model tag is not a runner contract. On that pre-release, supported Apple Silicon architectures automatically run on MLX unless you check which runtime loaded.

   Read the note → · A Model Tag Is Not a Runner Contract: Ollama’s MLX Default and What Still Differs

4. What did Ollama 0.40.2 do to models downloaded with earlier versions?

   • Deleted them
   • Upgraded them in the background the first time you run them, and kept the original copy as a backup
   • Required a new ollama pull before any run
   • Moved them to a cloud runner
   Answer 4

   Upgraded them in the background the first time you run them, and kept the original copy as a backup

   v0.40.2 upgrades those models in the background on first run for llama.cpp compatibility, and keeps the original copy on disk as a temporary backup. That is not a pull you chose.

   Read the note → · A Background Upgrade Is Not a Verified Pull: Ollama 0.40.2–0.40.3 and On-Disk Model Migration

5. What did Ollama 0.40.3 change about that background upgrade?

   • It made the upgrade mandatory on every request
   • It deleted every backup
   • It paused background model upgrade because of per-request overhead, including for embeddings
   • It turned MLX off worldwide
   Answer 5

   It paused background model upgrade because of per-request overhead, including for embeddings

   v0.40.3 paused the background model upgrade for embedding and other request performance. The pull request says it temporarily removes that migration until a future release.

   Read the note → · A Background Upgrade Is Not a Verified Pull: Ollama 0.40.2–0.40.3 and On-Disk Model Migration

6. What is llama.cpp GBNF for, compared with tool calling?

   • GBNF constrains sampling locally so the output matches a grammar. Tool calling asks the host or an MCP server to act
   • GBNF phones a remote tool, tool calling stays inside the decoder
   • They are the same HTTP call
   • GBNF is a vector index
   Answer 6

   GBNF constrains sampling locally so the output matches a grammar. Tool calling asks the host or an MCP server to act

   Grammar-constrained decoding needs no network hop for the structure itself. Tool calling is a request for a side effect in the world. One is inference residency; the other is action residency.

   Read the note → · Inference Residency Is Not Action Residency: Tools, MCP, and Local LLMs

7. Does a successful load prove the weights are the bytes you think they are?

   • Yes. A green loaded status checks every tensor
   • Yes, if the filename ends in .gguf
   • Only when the chat reply looks fluent
   • No. An embedded GGUF can still map the wrong tensor bytes if padding is taken from the outer file instead of the GGUF start
   Answer 7

   No. An embedded GGUF can still map the wrong tensor bytes if padding is taken from the outer file instead of the GGUF start

   The GGUF note's thesis: successful load is not proof of correct weights when the file is embedded and the loader pads from the wrong origin. b11019 fixed alignment relative to the GGUF's own start.

   Read the note → · Successful Load Is Not Proof: Embedded GGUF Alignment and Wrong Tensors

8. What does local inference remove, and what does it leave in place?

   • It removes malware and shoulder-surfing
   • It removes provider exposure for that session. It does not invent a secure operating system
   • It removes the need to read a model card
   • It guarantees the weights were trained without logging
   Answer 8

   It removes provider exposure for that session. It does not invent a secure operating system

   When inference is local, there is no third-party training clause for that session. A compromised laptop with a local model is still a compromised laptop. Residency is not a secure OS.

   Read the note → · Local and Peer-to-Peer AI: Data Residency Without the Hype