Streaming in Production: SSE Events, Idle Timeouts, and Usage Accounting

LLM API streaming is the easiest feature to add and one of the easiest to get wrong. In a demo it is a single flag, stream: true, plus a loop that prints tokens. In production it becomes four separate problems: the event model (how you parse without brittleness), the timeout policy (how long a silence means a dead stream), usage accounting (how you get token counts when the response arrives in pieces), and what happens when a stream breaks halfway through an answer. Get any of the four wrong and the symptoms are familiar: answers that hang, invoices that do not match dashboards, or duplicated generations after a hiccup. This piece is the streaming chapter of the builder series, following the structured outputs and tool calling guides.

The LLM API streaming event model, in two shapes

Both major providers stream over Server-Sent Events, and both layer semantic event types on top of the transport, but the shapes differ in ways that change your parser.

OpenAI. By default, the API generates the model's entire output before sending it back in a single response; streaming lets you start printing or processing while generation continues. The official streaming guide focuses on HTTP streaming (stream=true) over SSE. A separate WebSocket transport exists for persistent connections with incremental inputs via previous_response_id; that is a different transport choice, not a parser detail. On the HTTP path, the Responses API uses semantic events: every event is typed with a predefined schema so you can listen only for what you care about. Events you will handle in almost every integration include response.output_text.delta for incremental text, response.completed for the completed response, error for in-stream errors, and response.created. Some lifecycle events are emitted once, others repeatedly as the response is generated. If streamed output is user-visible, also read the Moderation risk section in that guide as part of your launch review.

Anthropic. Each event carries an SSE event name such as event: message_stop, and the matching type inside its data. The streaming documentation defines the full flow: message_start arrives first with an empty content; then, for each content block, a content_block_start, one or more content_block_delta events, and a content_block_stop; then one or more message_delta events carrying top-level changes to the final message; and a final message_stop. Each content block carries an index corresponding to its position in the final content array, which matters as soon as a response mixes text, tool calls, and thinking: the delta types are text_delta, input_json_delta, and thinking_delta.

Parsing server-sent events from LLM APIs: four traps

The transport is simple, which is why the failure modes are subtle. Four traps account for most broken parsers:

1. Treating pings as content. Anthropic's docs state that event streams may include any number of ping events. A parser that assumes "every event carries something I need" will either crash or emit garbage. Handle pings explicitly: skip them, optionally count them for keepalive logic.
2. Adding cumulative counters. The token counts in the usage field of the message_delta event are cumulative. If you sum each occurrence, you double- and triple-count. The correct habit is to overwrite: keep the latest value and use it at the end. This is the single most common source of token dashboards that disagree with provider consoles.
3. Rejecting unknown event types. Per the versioning policy, new event types may be added, and code should handle unknown types gracefully. A parser with a strict allowlist of event names will break on a vendor update. Give every dispatch a default branch that logs the unknown type and continues.
4. Treating HTTP 200 as success. Errors can arrive inside the event stream. Anthropic's docs give the canonical example: under high usage you may receive an overloaded_error in-stream, which would normally correspond to an HTTP 529 in a non-streaming context. So a successful status line means "the stream opened," not "the answer finished." Your loop must check both: in-stream error events, and whether the completion event (the response.completed event, or the message_stop event) actually arrived.

Streaming timeout handling: idle gaps, not total duration

A stream leaves you with three different clocks, and they need three different policies:

• Connection establishment. How long you wait for the HTTP response headers. This can stay strict, because it measures a request, not a generation.
• Idle time: the gap between events. This is the one that matters in streaming, and it should be measured per gap, not from first byte to last. A long answer that is producing events steadily is healthy even if it runs for minutes; a stream that has gone silent is not.
• Total duration. A hard ceiling as a backstop, set from your own workload data rather than from hope.

Two details from the official docs shape the idle policy. First, pings: since ping events can appear any number of times, treat any event, including pings, as activity that resets the idle clock; do not filter pings out of liveness tracking. Second, tool use: the Anthropic docs note that when using tools there may be delays between streaming events while the model works, because tool arguments arrive as chunked partial JSON assembled key by key. A timeout tuned on plain text turns into spurious disconnects the day tools enter the loop. The practical pattern is to widen the idle threshold for tool-bearing turns, or set a separate threshold for them, and keep the tool-side assembly rules where the tool calling guide puts them.

Usage accounting: Anthropic message_delta usage and OpenAI's completed event

Usage data exists on both streaming paths, but only at well-defined moments, and that shapes the accounting discipline.

On OpenAI's side, the final usage numbers are part of the completed response, so read them from the response.completed event. On Anthropic's side, the usage field of the message_delta event carries the counts, and remember, they are cumulative: take the final occurrence. Nothing before the completion moment is authoritative, which leads to the two rules that keep cost data sane:

1. Only count usage when the completion event arrives. A stream that died mid-answer never delivered its final numbers; if you estimate or extrapolate, mark the call as estimated, not measured. (This is the streaming corollary of the cost hub's rule that only completed work counts.)
2. Keep interrupted streams in their own bucket. Do not merge partial-stream calls into the same aggregate as normal calls, or your per-feature cost comparisons from the cost order of operations start lying to you. Track counts and rates for incomplete streams as their own metric.

One related question this article will not answer from the stream shape: whether a given interrupted stream is billed. That is a question for your own usage records and the provider's billing documentation, not something you can infer from event order.

When a stream breaks: retry or render what arrived

When a stream dies or delivers an in-stream error, there is no universal right answer, only a decision framework:

• Retrying regenerates the answer. That is right for batch and automated pipelines, where the consumer expects one complete result and a partial one is useless. Keep it inside a retry budget, and remember the retryable/non-retryable split: an in-stream overloaded_error is a capacity signal, similar in meaning to its non-streaming HTTP counterpart, while other error types may be policy or request problems that a retry cannot fix, as the error code triage and the retry budgets guide lay out.
• Rendering the partial result is right for user-facing conversational interfaces. The user has already seen tokens; yanking them away and regenerating from scratch is worse than an honest continuation path. The obligation this creates is narrow and important: your system must be able to distinguish a complete answer from a partial one, in the UI and in the data, so nothing downstream treats a truncated response as final.

What not to do is mechanical: do not treat an in-stream error event as if it were an HTTP-level failure with a single retry policy. Look at the event's error type first, then apply the matrix above.

What extra fields a streaming call logs

The logging field list from the observability guide applies unchanged to streaming, with five additions worth wiring in from day one:

• whether a completion event arrived (the single most useful reliability signal on this path);
• how many delta events were received, as a rough shape-of-response metric;
• the in-stream error type, if any, recorded separately from HTTP status;
• the maximum observed gap between events, which is the data you need to tune the idle threshold rather than guess it;
• whether a partial result was rendered to the user, so quality and fulfillment metrics can exclude it.

The streaming checklist

1. Parse semantically: switch on event type, and skip pings explicitly.
2. Overwrite usage counters; never accumulate cumulative fields.
3. Let unknown event types log and continue; do not fail the stream on them.
4. Treat the HTTP status as "stream opened," not "answer finished"; check for in-stream errors and for the completion event.
5. Set the idle timeout per event gap, reset it on any event including pings, and widen it for tool-bearing turns.
6. Read usage only at the completion moment; mark anything else as incomplete, not as data.
7. Decide retry-versus-render by consumer type, one strategy per path, and route in-stream errors through the error-type matrix first.
8. Log the five streaming fields above, including the max inter-event gap.

Streaming is not harder than the rest of a production integration; it is just less tolerant of assumptions. The wire protocol is simple, the four problems are permanent, and the teams that handle them treat "did the stream finish" as a first-class fact everywhere in their stack. When your calls go through a compatible endpoint, the endpoint-level details live in the platform's own documentation.