Long Conversations Without a Context Overflow: State, Compaction, and What Still Gets Billed

Two facts about long conversations surprise most teams, and both change the design. The first is billing: chaining turns with previous_response_id does not make the earlier input free. The official documentation is explicit that even when using previous_response_id, all previous input tokens for responses in the chain are billed as input tokens. The second is pruning: whether you should trim history depends on how you continue the conversation, and the advice is opposite for the two patterns.

Everything else follows from separating two ideas that sound similar. State is about who remembers the conversation. Context is about how much fits in a single request. Managing the first does not raise the second.

This article follows the official conversation state guide and compaction guide, checked on 2026-11-02.

State and context are different limits

The context window is the maximum number of tokens that can be used in a single request, and that number includes input, output, and reasoning tokens. It is a per-request ceiling, not a per-conversation allowance.

State management answers a different question: where does the history live between requests? You can hold it yourself and resend it, let the platform persist it as an object, or chain responses by identifier. None of those choices changes the ceiling. A conversation that has grown beyond the window will still overflow, whether the history arrives as an input array or is referenced by an ID.

That distinction explains why "we use server-side state" is not a capacity plan, and why "we resend everything" is not automatically wasteful. They are different trade-offs along a different axis.

Three ways to hold the conversation

The manual approach puts prior turns in the input array. It is fully stateless, easy to replay, and easy to audit, because the request contains exactly what the model will see. The cost is that you own the bookkeeping and you resend whatever you include.

The Conversations API persists state as a long-running object with a durable identifier, which can be reused across sessions, devices, or jobs. It is the right shape when the conversation itself is a product object that should outlive a single process.

previous_response_id chains responses: each new request references the prior response and sends only the new user message. It is the lightest client-side bookkeeping of the three, because the chain is carried by the identifier.

The choice is not about which is modern. It is about two questions: may the platform retain this conversation, and do you need precise control over what is sent on each turn? The Responses versus Chat Completions comparison is the place to start if the request shape itself is still undecided.

The billing fact that changes the math

Chaining with previous_response_id removes the need to resend history in the request body. It does not remove the history from the bill. The guide states that all previous input tokens for responses in the chain are billed as input tokens in the API.

So the intuition "using an ID means less input, therefore lower cost" is wrong on the second step. The tokens are still input tokens; they are simply supplied by the platform from stored state rather than by your JSON.

What actually reduces cost on long conversations is a different pair of levers. Prompt caching makes repeated prefixes cheaper to read, which is why a stable prefix and a sensible cache lifetime matter, as the caching analysis describes. Compaction reduces the number of tokens carried into the next window at all. Neither is a billing trick; both change what gets sent.

Turning on server-side compaction

Compaction is enabled in the same Responses create request, by setting context_management with compact_threshold. When the rendered token count crosses the configured threshold, the server runs a compaction pass.

Three details matter operationally. No separate /responses/compact call is required in this mode; the compaction happens inside the normal request. The response stream includes an encrypted compaction item. And that item is opaque and not intended to be human-interpretable, so do not build UI or logs that assume it is a readable summary of the conversation.

The compaction item carries forward key prior state and reasoning into the next run using fewer tokens. That is the mechanism: not a text summary a human wrote, but an encrypted artifact the model can consume.

The ZDR-compatible path

Teams often assume that long conversations and no-retention are mutually exclusive. The guide gives the official path: server-side compaction is ZDR-friendly when you set store=false on your Responses create requests.

That combination is the one to remember for regulated workloads: compaction for context size, store=false for retention posture. It does not make the conversation stateless in the sense of client bookkeeping, because you still have to carry the compaction item forward, but it addresses the retention question directly.

Two chaining patterns, opposite pruning advice

Here is where the second surprise lives.

If you chain statelessly with an input array, you append the response output, including compaction items, to your next input array. Then the latency tip applies: after appending, you can drop items that came before the most recent compaction item, because the latest item carries the necessary context. That keeps requests smaller and reduces long-tail latency.

If you chain with previous_response_id, you pass only the new user message each turn, and you should not manually prune. The platform is managing the chain; trimming it yourself conflicts with that management.

Getting these backwards is a common failure. Pruning a previous_response_id chain can remove context the platform expects, while refusing to prune a stateless array keeps every stale item in the request and inflates both latency and input tokens.

When you want compaction on demand

There is also a standalone compact endpoint for explicit control. It is fully stateless and ZDR-friendly: you send a full context window of messages, tools, and other items, and it returns a new compacted window to pass to your next /responses call.

Two rules come with it. The window you send must still fit within the model's context window, so this is not a way to compress something that is already too large. And you should not prune the output: the returned window is the canonical next context, and it generally contains more than just the compaction item, including retained items from the previous window.

That makes the endpoint a good fit for scheduled tidying or for compressing at a deliberate point in a workflow, rather than on every turn.

Pitfalls worth designing around

Replay the output array. For stateless reasoning-model requests, preserve every item in the response's output array. The Responses API returns encrypted reasoning items by default, and replaying the complete output keeps reasoning items and assistant phase values intact. Dropping items to "clean up" silently removes context the next turn may need.

Watch the phase field. The guide carries a troubleshooting note for cases where a model treats an intermediate update as the final answer: verify that your integration preserves the assistant message phase field. This is exactly the kind of bug that looks like a model quality problem and is actually a state-reconstruction problem.

Do not treat the compaction item as documentation. It is encrypted and opaque. If you need an audit trail, log the request metadata and the decisions, not a decoding of the item, following the field discipline in the logging guide.

Re-run regression after changing the pattern. Switching between stateless arrays and ID chaining changes what the model sees on each turn. Validate with the same discipline as a prompt change, as the testing guide describes, and use the failure triage map when a long conversation starts producing odd answers.

Decision checklist

1. Decide whether the platform may retain the conversation; if not, use store=false.
2. Decide whether you need precise control over each turn's input; if so, prefer the stateless array.
3. Set compact_threshold with output headroom in mind, since the context window covers input, output, and reasoning.
4. Write the chaining pattern into the integration contract: stateless array or previous_response_id.
5. For stateless chains, keep the latest compaction item and drop what came before it.
6. For previous_response_id chains, do not prune manually.
7. Use /responses/compact when you want compaction at a deliberate moment, and pass its output through unpruned.
8. Remember that an ID does not reduce billed input tokens; budget for the chain, or use caching and compaction to change the token count.

Long conversations are a state problem and a capacity problem at the same time, and the two have different tools. Pick where the history lives, decide how much of it travels, and be explicit about which of those two decisions you are making. The bill will reflect the tokens either way.

All parameters and behaviors above follow the official guides as checked on 2026-11-02; confirm them against the current documentation before relying on them.