Anthropic and OpenAI both sell batch inference at a flat 50% discount with a 24-hour completion window, and in practice most jobs finish in well under an hour. If any of your workload is triggered by a cron job rather than a click and it's running through the real-time API, you are paying double for latency nobody is waiting on. The catch is architectural: batch is not "the same call, cheaper" — it's a different system with different failure modes, and teams that treat it as a drop-in swap build the pipeline twice.
What the batch APIs actually promise
You upload a file of requests (JSONL, each line carrying your own custom_id), the provider processes them within 24 hours, and you download per-line results where any individual line can fail independently. Rate limits are separate from and far larger than your real-time quota — batch is how you push 100k requests through without touching the limits your product depends on. There is no ordering guarantee, no streaming, and the 24-hour window is a ceiling, not an SLA: plan for the ceiling, enjoy the usual sub-hour reality. Result files also expire — on the order of days to weeks depending on provider — so downloading and archiving them belongs in your reconciliation step, not in an on-demand fetch three months later. Discounts stack with caching only unreliably (you don't control scheduling, so prefix locality is luck) — price your batch math at the flat 50% and treat cache hits as a bonus.
What belongs in batch
The test is simple: does a human see the result in this session? If not, it can wait an hour, and it belongs in batch. Concretely: embeddings backfills and re-indexing, enrichment and classification of historical records, nightly digests and summaries, eval suite runs, distillation data generation, content moderation sweeps, and report generation. In several products I've looked at, 30–60% of total token volume was cron-shaped work like this — which means a flat 50% discount on a third to half of the bill for zero model or quality change, usually the second-cheapest win in the cost-lever ranking after caching.
The system you actually have to build
Real-time inference is a function call. Batch is a data pipeline, and it needs pipeline parts:
Chunking. Split work into jobs of 1k–50k items. One giant job means one giant blast radius when something's wrong with the prompt; per-job metadata (prompt version, model, source query) makes reprocessing sane.
Idempotency. custom_id is your join key back to source rows. Derive it deterministically (row ID + prompt version), so resubmitting a job or re-running reconciliation can never double-apply results.
Reconciliation. After download: match results to rows, validate outputs (schema, label membership), requeue individual failures once into a follow-up job, and dead-letter what fails twice. Expect a small percentage of per-line failures as normal operation, not incident.
State tracking. A jobs table — submitted, in-flight, downloaded, reconciled — plus a poller. Without it, the first partial failure turns into an afternoon of comparing files by hand.
The retry/dead-letter discipline is the same muscle as real-time API resilience; the difference is that batch failures are cheap and calm if you built the reconciliation step, and chaotic if you didn't.
Self-hosted batch: the same discount, different mechanism
The economics replicate on your own hardware. A GPU serving interactive traffic runs at whatever utilisation your users generate — often 10–30% — while the same card running offline inference with saturated continuous batching delivers 2–5× the tokens per second it manages under latency-constrained serving. vLLM's offline engine (or even a nightly llama.cpp run on a homelab card) turns idle overnight hours into the batch tier: embeddings, enrichment, and eval runs scheduled at 2am cost you electricity. The scheduling logic — queue, chunk, reconcile — is identical to the hosted case, which is a good argument for building it once behind an interface.
Hybrid designs that pay
| Pattern | How it works | Where it fits |
|---|---|---|
| Precompute + fallback | Batch generates likely answers nightly; real-time covers misses | Recommendations, FAQ-ish support |
| Deadline queue | Requests carry a deadline; anything with >1h slack rides the next batch | Enrichment triggered by user actions |
| Tiered SLA | Free tier waits for batch, paid tier gets real-time | AI features with a free tier |
| Shadow evals | Every prompt change runs the eval set via batch before deploy | CI for prompts |
The precompute pattern is the sleeper: if 60% of tomorrow's requests are predictable from today's data, you can serve them from a table at batch-half-price economics and real-time-zero latency, which also quietly fixes your p99 latency for those requests.
Two systems, one decision rule
Route by who's waiting. A human in the loop: real-time path — streaming, hedging, tight timeouts, full price. A cron job, a backlog, a pipeline: batch path — chunked jobs, reconciliation, half price. Resist the middle path of "real-time API called from a worker queue at full price", which combines the cost of one system with the architecture of the other; it's the most common shape I see and it's strictly dominated. Build the jobs table and the reconciler the first week you have any batch workload at all — every batch use case you add afterwards inherits it for free, and the 50% discount starts compounding from day one.