Local-First AI Apps: Patterns and Pitfalls

Local-first AI works inside a well-defined envelope: models of 8B parameters or under, tasks that survive an occasional wrong answer, and hardware you probe at runtime instead of assuming. Inside the envelope you get zero marginal inference cost, offline operation, and data that never leaves the device. Outside it — long-context reasoning, multi-step agents, generation where quality is the product — the honest architecture is local-by-default with explicit cloud escalation. Most local-first failures I've seen come from pretending the envelope is bigger than it is.

What runs on user hardware, by task

Embeddings are the underrated row. A 137M-parameter model like nomic-embed-text runs anywhere, indexes a user's entire document library in minutes, and gives you semantic search with no envelope anxiety at all. If you're deciding what to make local first, start there, not with chat.

The other consistent winner is structured extraction. Small models are unreliable prose writers but decent form-fillers, and grammar-constrained decoding (llama.cpp grammars, Ollama's format parameter) removes the malformed-JSON failure mode entirely — see structured output from LLMs for the technique.

Probe the machine, don't assume it

"Runs on user hardware" is a distribution, not a fact. The same feature is instant on an M3 and unusable on a 2019 dual-core. Detect, then measure — this is the pattern for finding a local Ollama instance without hanging the UI:

async function pickBackend(): Promise<"local" | "cloud"> {
  try {
    const r = await fetch("http://127.0.0.1:11434/api/tags", {
      signal: AbortSignal.timeout(400),
    });
    if (r.ok) {
      const models = (await r.json()).models ?? [];
      if (models.some((m) => m.name.startsWith("qwen3:8b"))) return "local";
    }
  } catch {
    /* not running */
  }
  return "cloud";
}

Detection alone isn't enough. On first run, time a 50-token generation and cache the result. My thresholds: under 7 tok/s decode, the device gets local inference for background jobs only (overnight tagging, index building) and cloud or nothing for interactive features; over 15 tok/s, interactive chat is fine. Re-probe after model updates, because a quantisation change moves the number — quantisation trade-offs are half of what makes the envelope stretch.

Derived data: recompute, don't sync

The classic sync mistake is treating AI outputs as user data. Split your data model in two. Raw user content (notes, photos, files) syncs normally. Derived artifacts — embeddings, summaries, auto-tags — are a cache keyed by (content_hash, model_id, prompt_version), and caches get rebuilt, not synced.

The reason is stricter than tidiness: a vector index is only valid for exactly one embedding model and version. Two devices running different models produce vectors in incompatible spaces, and syncing them silently corrupts search. When you upgrade the embedding model, re-embed in the background and keep serving the old index until the new one is complete — a blue-green deployment happening on someone's laptop.

Summaries and tags are cheaper to get wrong but the same rule applies: recomputing 2,000 note summaries with a 3B model is a few minutes of background work, while designing conflict resolution for synced AI output is a permanent tax.

Degrade to something useful

Every AI feature needs a defined non-AI fallback, decided at design time. Semantic search falls back to SQLite FTS5 keyword search. Auto-tagging falls back to rules on filename and date. Summaries fall back to the first 300 characters and an honest label. The acceptance test I use: delete the model and disable networking — the app should still be a good app, just less clever. If a feature has no acceptable fallback, it was never a local-first feature; it's a cloud feature you were hoping to run locally.

Where local-first breaks

Long context. KV cache is the wall before quality is. An 8B model with grouped-query attention costs about 131KB of cache per token at fp16 — a 32k context conversation is roughly 4.3GB on top of the 5GB of weights, which evicts an 8GB GPU. Prefill on CPU is worse: pushing 32k tokens through a laptop takes minutes, not seconds.

Multi-step agents. Error compounds per step. A small model that gets each tool call right 90% of the time completes a 10-step chain about 35% of the time. Local agents want three steps or fewer, with validation between steps.

Quality-as-product generation. User-facing prose, production code, anything the user will paste under their own name. The gap between an 8B local model and a frontier API is small on extraction and large on this.

Thermals and battery. Sustained decode pins the GPU. On laptops, schedule background inference for AC power only — users notice a hot lap faster than they notice good auto-tags.

Escalate loudly, not silently

When you do add cloud escalation, the privacy promise is the product, so breaking it quietly is worse than not making it. Escalation should be per-feature consent, not one buried global toggle; the UI should show which requests left the device; and the log should record exactly what was sent. "Local-first" apps that silently post telemetry or fall back to cloud inference get caught, and the audience that chose a local-first app is precisely the audience that checks — self-hosters run the same audit on every LLM runner they deploy.

What I'd do

Ship v1 with local embeddings plus semantic search (works everywhere), one small model for extraction and tagging with a rules fallback, and no local chat. Probe hardware on first run, gate interactive features at 10 tok/s, and key every derived artifact by content hash and model version so recompute is always safe. Add an 8B chat model only for devices that pass the probe, and cloud escalation only with per-feature consent and a visible indicator. The apps that fail are the ones that promise a frontier experience on a 2018 laptop; the ones that work pick features where a small model is genuinely enough.