Verdict
For most teams, provider batch APIs win more often than the self-hosting instinct suggests: all three major providers discount 50% with a 24-hour completion window, and at the small-model end the discounted prices land in the same band as a well-utilized rented GPU - gpt-5.4-nano batch input works out to about $0.10 per million tokens against the roughly $0.04-0.09 per million total tokens a rented H100 running an 8B model at high batch achieves at 50-100% utilization, and the gap closes entirely once real-world utilization, retries, and orchestration time are priced in. Self-hosting wins in three specific regimes: you need weights the providers do not serve (fine-tunes, open models with modified sampling), your data cannot leave your infrastructure, or you run sustained 70B-class volume where Sonnet-class batch pricing ($1.50 input / $7.50 output per million) leaves real room under it. If your jobs are occasional and small, a third option beats both: just call the synchronous API and skip the batch plumbing entirely.
The provider side is unusually uniform and verifiable. OpenAI's Batch API is 50% off standard pricing with a 24-hour completion window, a 50,000-request cap per batch file, and a 200 MB input file limit (developers.openai.com/api/docs/guides/batch; the Batch API announcement and FAQ on community.openai.com and help.openai.com). Anthropic's Message Batches API is also 50% off with a 24-hour window but allows up to 100,000 requests per batch (platform.claude.com/docs batch-processing docs), putting Claude Sonnet-class batch pricing at $1.50 per million input tokens and $7.50 per million output. Gemini's Batch API is likewise 50% off synchronous pricing with a within-24-hours target (per leanlm.ai's cross-provider batch roundup; Google's own per-model batch table was not independently re-verified for this page, so treat exact Gemini per-model figures as check-before-you-commit). The self-host side: RunPod lists H100 80GB from $1.99/hr Community Cloud and $2.39/hr Secure Cloud, with RTX 4090 from $0.34/hr (runpod.io pricing; corroborated by getdeploying.com and intuitionlabs.ai trackers), and Vast.ai marketplace H100s run roughly $1.60-1.87/hr on-demand with interruptible/spot listings around $0.57-1.03/hr (getdeploying.com and intuitionlabs.ai H100 price trackers; Vast.ai is a marketplace, so listings move). Published vLLM measurements put Llama 3.1 8B total throughput on a single H100 around 12,500 total tokens/sec at high batch (Microsoft's Azure HPC benchmark writeup on techcommunity.microsoft.com and aimultiple.com's inference-engine roundup both report ~12,553 tok/s for optimized vLLM; Simplismart's deployment writeup measured 8,674 total tok/s in a less aggressive configuration). At 12,500 tok/s an H100-hour processes ~45 million total tokens, so a $1.99/hr GPU at full utilization lands near $0.044 per million total tokens - and near $0.09 at 50% utilization, which is what real pipelines achieve. That number is the whole decision. OpenAI's current small-model tier is the gpt-5.4 family (developers.openai.com/api/docs/pricing lists gpt-5.4-nano at $0.20/M input and $1.25/M output, gpt-5.4-mini at $0.75/$4.50; corroborated by benchlm.ai and pricepertoken.com trackers), so 50%-off batch rates are roughly $0.10/M input for nano-class and $0.375/M for mini-class: a saturated H100 sits at rough parity with nano-class batch on raw token math, clearly under mini-class, and far under Sonnet-class batch rates - but only when you hold utilization high on a model whose quality actually matches your task. One freshness warning that is itself a finding: these prices moved a full generation between the GPT-5 and GPT-5.4 families, so any break-even spreadsheet built on last year's per-token figures is already wrong - rerun the math against the vendor's current sheet before committing either way.
Decision Table
| Criterion | Edge | Explanation |
|---|---|---|
| Headline discount and SLA | Tie | All three providers converged on the same offer: 50% off, results within 24 hours. OpenAI Batch (openai.com/api/pricing), Anthropic Message Batches (platform.claude.com/docs), and Gemini Batch (per leanlm.ai's cross-provider comparison) are indistinguishable on the headline terms. The differences that matter live one level down - request caps, file limits, and queue behavior - so the discount itself should never be the deciding factor between providers. |
| Effective cost at the small-model end | Provider batch APIs (OpenAI Batch API, Anthropic Message Batches, Gemini Batch API) Best edge | Closer than either camp admits, and the tiebreak is ops. Current nano-class batch pricing - gpt-5.4-nano at $0.20/M input and $1.25/M output list price (developers.openai.com/api/docs/pricing, corroborated by benchlm.ai and pricepertoken.com), so roughly $0.10/$0.625 at the 50% batch discount - sits at rough parity with a rented $1.99/hr H100 pushing ~12,500 total tok/s through an 8B model (~$0.044/M total tokens at 100% utilization, ~$0.09 at a realistic 50%; throughput per techcommunity.microsoft.com and aimultiple.com). On raw token math a saturated GPU edges ahead, especially on output-heavy synthetic-data runs. But once you add orchestration time, retries, idle spin-up, and the fact that the batch side hands you a frontier-lab small model instead of an open 8B, input-heavy workloads at realistic utilization tip to the batch API - by less than the old GPT-5-generation prices suggested, which is why you should rerun this row's math against the current price sheet. |
| Cost at sustained 70B-class volume | Self-hosted offline batch inference (vLLM on rented GPUs) Best edge | Sonnet-class batch pricing is $1.50 input / $7.50 output per million tokens (platform.claude.com/docs batch pricing table). A 70B-class open model served with vLLM in FP8 on rented H100s has meaningful room under that output price if - and only if - you keep the GPUs saturated: the hardware cost per token scales with utilization, and an idle reserved GPU still bills. Teams running tens of billions of offline tokens monthly on a task where an open 70B matches the frontier model's quality are the canonical self-host winners. Benchmark your own model-task pair first: published 70B throughput numbers vary widely by quantization and hardware, so we deliberately do not print a single 70B $/M figure here. |
| Model control: fine-tunes, open weights, custom sampling | Self-hosted offline batch inference (vLLM on rented GPUs) Best edge | Batch APIs serve the provider's catalog, full stop. If your offline pipeline needs a fine-tuned checkpoint, a model the providers do not host, logprobs-heavy custom decoding, or guaranteed reproducibility pinned to exact weights, self-hosted vLLM is the only option. This is frequently the real reason teams self-host batch - the cost argument gets made afterward. |
| Data residency and compliance | Self-hosted offline batch inference (vLLM on rented GPUs) Best edge | Self-hosted batch keeps prompts and outputs inside infrastructure you control - decisive for regulated data that cannot transit a US provider, or for contracts that forbid third-party processing. Provider batch files sit in the provider's storage for the duration of the job (OpenAI's flow uploads a JSONL file that persists until you delete it). If your compliance posture already forbids the synchronous API, batch inherits the same prohibition. |
| Throughput ceilings and job-size limits | Self-hosted offline batch inference (vLLM on rented GPUs) Best edge | OpenAI caps batches at 50,000 requests per file and 200 MB, with enqueued-token limits tied to your usage tier (community.openai.com Batch API thread); Anthropic caps at 100,000 requests or 256 MB per batch, whichever comes first (platform.claude.com/docs batch-processing). Big jobs mean sharding, per-tier queue management, and 24-hour worst-case turnaround per shard. A self-hosted vLLM fleet has no per-job caps - throughput is whatever you rent, and you can finish a giant corpus overnight by scaling out on Vast.ai interruptible instances (listings around $0.57-1.03/hr per H100) instead of waiting on a provider queue. |
| Ops burden and failure modes | Provider batch APIs (OpenAI Batch API, Anthropic Message Batches, Gemini Batch API) Best edge | Batch APIs turn inference ops into a file upload and a poll loop; the provider owns GPU failures, OOMs, and retries. Self-hosted batch means driver and CUDA version management, spot-instance preemption handling, checkpointing partially completed shards, and debugging vLLM OOMs at 3 a.m. - real engineering time that never appears in the naive $/M-token math. If your team does not already run GPUs in production, the first month of self-hosted batch usually costs more in engineer-hours than it saves in tokens. |
| Occasional small jobs | Tie | A third option wins this row outright: the plain synchronous API. For a one-off eval of a few thousand requests or a corpus under a few million tokens, the 50% discount saves single-digit dollars while costing you batch-file plumbing, a polling loop, and up to 24 hours of waiting - and spinning up a GPU for it is even less defensible against per-hour minimums and setup time. Sub-$20 jobs should just run synchronously with concurrency; neither batch architecture earns its complexity there. |
Choose Provider batch APIs (OpenAI Batch API, Anthropic Message Batches, Gemini Batch API) if...
- Teams whose offline tasks are handled by nano/mini-class models, where batch pricing sits at parity with the self-hosted $/M floor before you have paid a single engineer-hour of GPU ops.
- Teams without existing GPU ops experience - the 24-hour SLA costs patience, not engineer-hours.
- Workloads needing frontier-model quality (evals judged by a top model, complex extraction) where no open model substitutes.
- Spiky, unpredictable offline volume: you pay per token, never for idle GPUs.
Choose Self-hosted offline batch inference (vLLM on rented GPUs) if...
- Pipelines built on fine-tuned or open-weight models the providers do not serve.
- Data-residency-constrained workloads where prompts cannot transit a third-party API.
- Sustained, saturating 70B-class volume with a quality bar an open model meets - the only regime where the $/M math clearly favors GPUs.
- Giant single jobs where provider per-batch caps would force painful sharding, and spot-market GPUs (Vast.ai interruptible H100s around $0.57-1.03/hr) can be scaled out overnight.
- Output-heavy generation workloads (large-scale synthetic data) where batch output prices ($0.625/M even at nano class) are a multiple of a saturated GPU's ~$0.04-0.09/M total-token cost.
Decision rules
- 01If your task is served well by a small frontier model (nano/mini class), default to the provider batch API - at roughly $0.10-0.375 per million batch input tokens (50% off gpt-5.4-nano/mini list prices) it sits at parity-or-better with a realistically-utilized rented H100 running an 8B open model (~$0.04-0.09/M total tokens) once orchestration and idle time are counted, and you get a frontier-lab model.
- 02If you need fine-tuned weights, a model providers do not serve, or data that cannot leave your infrastructure, self-host - the cost comparison is irrelevant because batch APIs cannot do the job at all.
- 03If you run sustained 70B-class offline volume (tens of billions of tokens monthly) and an open model matches your quality bar, price a saturated vLLM fleet against Sonnet-class batch ($1.50/$7.50 per M) - self-hosting wins only above roughly 60-70% sustained GPU utilization, so measure your real utilization, not your peak.
- 04If a single job exceeds OpenAI's 50,000-request/200 MB file cap or Anthropic's 100,000-request batch cap by a large factor, self-hosted throughput you control may beat sharding across provider queues with 24-hour worst-case turnaround per shard.
- 05Confirm Gemini's exact per-model batch prices on ai.google.dev before building on them - the 50%/24h shape is consistent across roundups, but per-model figures change and were not independently re-verified for this page.
- 06For occasional jobs under a few million tokens, skip both: run the synchronous API with client-side concurrency and move on.
Migration Notes
- Moving from batch API to self-hosted: build utilization tracking first. The break-even math quoted everywhere assumes 100% utilization; real pipelines idle between shards, and every idle hour moves the break-even against you.
- Moving from self-hosted to batch APIs: reshape your pipeline around asynchronous completion - code that assumed streaming or synchronous results needs a submit/poll/collect structure, and per-file request caps mean a sharding layer for large corpora.
- Either direction, keep the batch-job format provider-agnostic (a JSONL of request objects): OpenAI, Anthropic, and a vLLM offline script can all be fed from the same intermediate representation with thin adapters.