Tracing and Observability for LLM Apps

Log lines cannot debug a six-step agent. What you need is traces: one trace per user request, a span for every model call and every tool call, with token counts, cost, and the exact rendered prompt attached to each span. That's the entire architecture, and a self-hosted Langfuse instance gets you there in an afternoon for the price of running Postgres. The teams still grepping application logs for "why did it say that" are missing structure, not data.

Why logging fails here specifically

An LLM request isn't an event, it's a tree. One user message fans out into a retrieval call, two model generations, four tool executions, and maybe a subagent — and the question you'll ask during an incident is never "what happened at 14:32" but "show me everything about this request: what was retrieved, what exact prompt went out, which prompt version, what came back, what did it cost". Flat logs make you reassemble that tree by hand from interleaved lines. Traces are the tree.

The trace model

One trace per user request (grouped by session where relevant), three kinds of spans, each with the attributes you'll actually query:

Two attributes earn their keep within a week. Prompt version on every generation span is what turns "quality dipped Tuesday" into a one-line query — it's the runtime half of treating prompts like code. Cached-token counts are how you notice a prompt edit silently broke your prefix cache before finance does.

Instrumentation is a decorator, not a project:

from langfuse import observe, get_client

@observe()
def answer(question: str) -> str:
    docs = retrieve(question)              # child span
    completion = generate(question, docs)  # generation span w/ usage + cost
    get_client().update_current_trace(
        metadata={"feature": "qa", "prompt_version": PROMPT_HASH},
    )
    return completion

If you'd rather not marry a vendor SDK, OpenTelemetry's GenAI semantic conventions (opentelemetry.io) standardise the same span attributes, and most LLM observability backends — Langfuse included — ingest OTLP. Instrument once, keep the backend swappable.

Cost per request is the killer feature

Compute cost from the returned usage fields on every response — actual input, output, and cached token counts times your price sheet — never from client-side estimates, which drift the moment a provider changes tokenisers or you enable caching. Attach it to the span, aggregate by feature and by user.

This one aggregation catches, in my experience, more real problems than any quality metric: the agent loop that occasionally runs 40 turns instead of 4, the cache-hit ratio that fell off a cliff after a prompt edit, the retrieval change that doubled context size, the single integration user costing $30/day. Per-feature daily cost with a step-change alert is the first dashboard to build, and it's also the data source that makes cost engineering something you do from measurements instead of invoices.

Sampling: metadata always, payloads selectively

Full prompts and completions are bulky — an agent-heavy app generates gigabytes a week — and they're where the privacy risk lives. Sample in layers:

• 100% of metadata: every trace, every span, tokens, cost, latency, status. This is cheap and it's what your aggregates are built on. Never sample this layer.
• Sampled payloads: full prompt/completion text for 10–20% of normal traffic, adjusted to your volume.
• Always keep the interesting ones: errored traces, validation retries, refusals, p99 latency, and traces flagged by user feedback. These are the ones you'll actually open.

Retention follows the same split — payloads for 30–90 days, metadata aggregates indefinitely. Payloads are also the reason self-hosting is the right call for anything sensitive: they contain whatever your users typed, and shipping that to a third-party SaaS is a data-processing decision someone should be making deliberately, not defaulting into via an SDK key.

Self-hosted options, honestly compared

Langfuse is the mature default: docker compose up gives you the v3 stack (Postgres for transactions, ClickHouse for analytics, Redis and blob storage alongside — see langfuse.com/docs), comfortable on a 4GB VM at small-team volume, with prompt management and eval scoring in the same UI as traces. Arize Phoenix is lighter and excellent for retrieval debugging; a plain OTel collector into your existing Grafana stack works if you already live there and only want metrics, though you'll miss the prompt-diff and trace-reading ergonomics that make LLM-specific tools worth running. Whichever backend: put it on your own box, next to the rest of your monitoring stack, and treat it as production infrastructure — it's the system you'll be staring at during every incident.

Alerts that map to real incidents

Four, tuned to page rarely: daily cost per feature (step change vs 7-day baseline), error-plus-refusal rate, p95 time-to-first-token, and — once you have calibrated evals — the nightly eval score on sampled production traces, written back to the traces it graded. That last loop is the payoff of doing observability properly: quality regressions page you with the offending traces already attached.

What I'd do

Afternoon one: Langfuse via compose, @observe on your entry points, usage-based cost on every generation span, prompt hash in metadata. Week one: the four alerts and the payload-sampling policy. Then stop — resist the observability-platform shopping spree until you've spent a month actually reading traces, because reading traces is the habit that finds bugs; the tooling above is just what makes them readable.