Batch API in Practice: Where Half-Price Applies and What Bites First

The Batch API is the official way to trade waiting for price: OpenAI describes a 50% cost discount compared with synchronous APIs, a separate pool of significantly higher rate limits, and a 24-hour turnaround. Those three properties point at the same decision. Can the work wait? Is there enough of it to matter? And is the synchronous path already blocked by rate limits? If the answers are yes, yes, and yes, batch is the right channel. If any answer is no, the discount is not the constraint you think it is.

The traps are not in the API call. They are in the engineering assumptions teams bring to it: that output arrives in input order, that a finished batch means every request succeeded, and that an expired batch means the requests failed. Each of those is false, and each one silently corrupts a pipeline that assumes it.

Everything below follows the official Batch guide, checked on 2026-10-29. Verify limits and statuses against that page before building on them.

When batch is the right tool

The guide names the canonical cases: running evaluations, classifying large datasets, and embedding content repositories. What they share is that no human is waiting on any individual response.

Three signals make the decision concrete.

First, nobody needs the answer immediately. A nightly reclassification, an evaluation sweep, or a backfill of embeddings can absorb the 24-hour window without anyone noticing.

Second, the volume is large enough that the discount changes the budget line. A handful of requests belongs on the synchronous path, where the engineering is simpler.

Third, the synchronous path is already constrained. This is the signal teams underrate: batch rate limits are separate from existing per-model rate limits, and the guide states plainly that using the Batch API will not consume tokens from your standard per-model rate limits. If a workload is being throttled during the day, moving it to batch does not just cost less; it stops competing for the same headroom.

The cases that do not fit are equally clear. Interactive features, streaming responses, and anything where a user is watching a cursor are wrong for this channel. The worker rejects requests that set stream=true, which is the API telling you the same thing.

Building an input file that survives contact with production

A batch starts as a .jsonl file. Each line is one request with a unique custom_id, a method, a url, and a body whose parameters match the underlying endpoint. The window is declared at creation time through completion_window, and the only value the service currently accepts is the 24-hour one; each batch completes within that window, often sooner.

Two rules deserve emphasis because they cause most of the rework.

The custom_id must be unique, and it should be something you can join on later. The impulse is to use line numbers, which is exactly the assumption the output order will break. Use a stable business key instead, such as a tenant plus a row identifier, so results can be mapped back into your system without a lookup table that only exists in someone's script.

And each input file can include requests to a single model only. A job that spans several models becomes several files and several batches, which changes how you track progress and how you retry.

The limits that actually bind

The per-batch limits are the ones you hit while constructing work: a single batch may include up to 50,000 requests, and a batch input file can be up to 200 MB. Embeddings carry an additional restriction, with a maximum of 50,000 embedding inputs across all requests in the batch.

Queued prompt tokens are limited per model, and the current values live on the platform settings page rather than in the guide, which is a hint that they change. The creation rate is the third constraint: up to 2,000 batches per hour. The guide's advice for higher volume is not to create batches faster but to increase the number of requests per batch, which is also the cheaper shape operationally.

There is no output-token limit at present. That matters for output-heavy jobs, where the input file is small but the responses are long.

Trap one: output order is not input order

The output file contains one response line for every successful request line in the input file, and the guide is explicit that the line order may not match the input order. Any code that zips input and output by position is wrong, and it will be wrong intermittently, which is worse than consistently wrong.

The fix is the custom_id, which is present on every output line. Treat the output as a set keyed by that identifier, not as a sequence. This is also why the identifier choice from the previous section matters: a business key makes the join trivial, while a generated identifier forces you to keep a mapping table for every batch.

Trap two: failures live in a different file

A batch that reports success at the batch level can still contain failed requests. The guide says failed requests have their error information written to an error file, retrieved through the batch's error_file_id.

So "the batch finished" is not the same as "every request worked". A pipeline that downloads only output_file_id will silently lose rows, and the loss is invisible unless someone reconciles counts. Check both files, and compare the number of output lines plus error lines against what you submitted.

The request_counts field on the batch object is the cheapest reconciliation: total, completed, and failed. If those numbers do not add up the way you expect, stop and read the error file before treating the results as complete.

Trap three: the statuses mean different things

The status list is short, and the distinctions matter:

• validating: the input file is being validated before the batch can begin.
• failed: the input file failed the validation process.
• in_progress: the file validated and the batch is running.
• finalizing: the batch completed and results are being prepared.
• completed: results are ready.
• expired: the batch could not be completed within the 24-hour window.
• cancelling: cancellation is in progress and may take up to 10 minutes.
• cancelled: the batch was cancelled.

Two readings are commonly wrong. First, failed at the batch level is a validation failure, not a statement that individual requests failed; per-request failures appear in the error file. Second, expired is not the same as "the requests failed". The guide describes unfinished requests being cancelled while responses to completed requests remain available in the output file, and expired requests are written to the error file with a batch_expired code so you can retrieve them by custom_id.

The practical response to expired is to split the work into smaller batches and resubmit the expired portion, not to re-run everything. And while a batch is cancelling, do not create a replacement; wait for cancelled so you do not end up with two batches processing the same rows.

Which endpoints can be batched

The supported list is broader than chat alone: /v1/responses, /v1/chat/completions, /v1/embeddings, /v1/completions, /v1/moderations, /v1/images/generations, and /v1/images/edits. Image generation and editing are easy to forget, and they are often the largest jobs a team runs.

Moderation has its own input shape: include an input field in every request body, and note that the worker rejects stream=true there as well.

Batch and caching are not alternatives

Synchronous traffic and asynchronous traffic need different optimizations. On the synchronous path, the lever is prompt caching: reuse a stable prefix and choose the lifetime that matches the reuse window, as the twelve-hour caching analysis describes. On the asynchronous path, the lever is batch pricing.

The order of operations matters: decide whether the workload can be asynchronous first, send it to batch if it can, and only then optimize the synchronous path that remains. The broader sequencing of cost work is in the cost reduction order of operations. Applying caching logic to a workload that should have been batched is wasted effort, and batching a workload that needs sub-second responses is a product decision disguised as a cost decision.

The operational benefit that is not price

Because batch uses a separate pool, moving daytime batch-shaped work into it frees synchronous headroom. For teams whose daytime traffic is throttled, that effect can matter more than the discount: the evaluations and backfills stop competing with user-facing requests for the same limit.

This is also the argument for scheduling. A nightly batch that finishes within its window costs less and keeps the daytime path clear, while the same job run synchronously at peak would consume the headroom that interactive features need.

The checklist

1. Confirm the work can absorb the 24-hour window and does not need streaming.
2. Check whether the synchronous path is rate-limited; if so, batch is doubly justified.
3. Split input files by model, one model per file.
4. Use a stable business key as custom_id, and store the mapping.
5. Keep each batch within 50,000 requests and each file within 200 MB, and remember the embedding input cap.
6. Reconcile output_file_id against error_file_id instead of trusting the batch status alone.
7. Treat expired as partial completion: resubmit the expired custom_ids in smaller batches, and wait out cancelling.
8. Log the batch ID, the endpoint, and the counts, with the field discipline in the logging guide, and verify the first production batch against the testing guide before scheduling it nightly.

Batch is a pricing channel with a different failure surface, not a faster endpoint. The discount is real, the window is fixed, and the two files are the source of truth. Read the errors before you trust the results, and the rest of the engineering is ordinary queue work. When a batch does fail in a way you did not expect, the failure triage map is the place to start.

All figures and behaviors above follow the official guide as checked on 2026-10-29; confirm them against the current documentation before relying on them.