How to choose an AI model for production is not a question about who tops a leaderboard this week. It is a question about constraints, and you already know most of the answers. Write six things down first, task type, context length, input modality, interaction shape, controllability, and cost-structure sensitivity, then converge on a choice with a small self-test built from your own traffic. This guide covers why benchmark lists cannot make this call, the six model selection criteria, a minimal golden set you can assemble in an afternoon, and a decision table organized by task family and model type.
Why leaderboards cannot make your production model choice
Benchmarks measure an average task under fixed prompts and fixed conditions. Your production request is not that. It has a particular input distribution, a particular prompt structure, and particular tolerance for failure that no public evaluation shares.
The gap shows up in the translation step. A published number tells you how some models scored on somebody's test set; it cannot tell you whether your task gets better or worse, because your task was never in the test. A useful habit when reading any published number: ask what task it measured, which prompt version was used, when it was run, and who controlled the setup. If those questions cannot be answered, the number is decoration, and you are back to your own constraints and your own tests. And the conditions behind any score age quickly: prompts change, versions roll, evaluation harnesses differ between publications, and a score with no visible date or methodology is a story, not evidence. None of that makes evaluation useless. It makes it someone else's evaluation, which is exactly why the last step of any model selection process is your own, run against your own inputs.
The practical consequence: treat leaderboards as signal for discovery, shortlists, and vocabulary, and treat production model choice as a constraint-matching problem you finish yourself. The capabilities that matter at the shortlist stage are documented at the source, and provider model pages are where those specs live, OpenAI's models documentation among them: what a model accepts, what it returns, and what it is designed for.
Six model selection criteria to write down first
Answer these before you open any comparison page. Two of the six, context window and modality constraints, eliminate the most candidates the fastest, so apply them early. Skip a criterion and you will re-litigate the whole choice within a quarter, usually while production traffic is already flowing through the model you picked.
One: task type. Generation, extraction, classification, code, or multi-turn tool calling. How to judge: classify the actual workload, not the demo. A chat feature that mostly extracts fields is an extraction task wearing a chat costume, and its criteria follow extraction.
Two: context length. How big is a typical input, and how big is a worst-case one? How to judge: measure across real requests, not samples chosen by hand. The deciding question is whether a full workload fits in one pass or must be chunked; the moment chunking enters, you have a pipeline problem that no model can absorb quietly.
Three: input modality. Text only, images coming in, or images being generated or edited? How to judge: audit what actually arrives. All current models from the major vendors handle text and image input, as Anthropic's models overview states plainly for its lineup, but generation and editing live at separate endpoints with separate constraints; if your product touches them, treat modality as a hard filter first.
Four: interaction shape. Offline batch or real-time interaction, and how much latency sensitivity comes with it. How to judge: identify who waits for the output and what breaks when they wait longer. A nightly pipeline and a live autocomplete have the same model question and utterly different answers once this criterion is written down.
Five: controllability. Structured output stability, tool-calling reliability, and reproducibility. How to judge: look downstream. If parsers, schemas, or automation consume the output, then stability is a requirement, not a preference, and it outranks capability on most days.
Six: cost-structure sensitivity. Not a number, a shape: are your requests input-heavy or output-heavy, do they repeat, can results be cached, and is the workload batch or interactive? How to judge: read your own traffic mix. Two teams running the same feature can land on different structures depending on whether they send long documents in or get long documents out.
Converge with a minimal golden set
With the six criteria written down, shortlist two or three candidates and settle the choice the way you settle every other production decision: with a small test built from real data.
Pull ten to thirty representative samples from actual traffic, and deliberately include the awkward ones, the longest input, the weirdest format, the case that made someone file a bug. For each sample, write an assertion on the acceptable output shape: this field present, this structure valid, this refusal absent, this citation format respected. Assertions beat vibes because they can fail visibly.
Run every candidate against the same set, then compare two things: how many assertions pass, and, more usefully, how failures distribute. One model failing on formatting while another fails on content tells you which failure your product tolerates. Repeat the run with identical inputs to catch instability and format drift, because a choice that only works on alternate Tuesdays is not a choice. Keep the golden set small enough to maintain: a few dozen well-chosen samples that get rerun whenever anything changes beat a grand evaluation suite that is rebuilt from scratch every quarter, because the rerun is what actually catches drift. This is the same verify-before-production discipline as our compatibility checklist, applied to model behavior instead of endpoint behavior.
The decision table: task families versus model types
The table below maps task families to the type of model to prioritize. Types, not names, because names rotate, versions retire, and the shape of the work stays recognizable longer than any specific model does. Which model for coding vs. long documents is not one question with one answer either; the table splits them because their constraint profiles differ.
| Task family | Priority type | The constraint doing the deciding |
|---|---|---|
| Summarization and rewriting | Lightweight-efficient or long-context | Volume and input size; interactive if users wait |
| Long-document analysis | Long-context | One-pass coverage versus chunk-and-merge pipeline |
| Code generation and review | Reasoning-heavy with strong tooling | Structured output and edit correctness |
| Structured extraction | Controllable structured-output, multimodal when inputs include images | Schema stability is the product |
| Multi-turn tool calling | Reasoning-heavy with reliable tool use | Failure behavior across turns |
| Low-sensitivity batch (drafts, tagging) | Lightweight-efficient | Cost structure and throughput shape, no human in the loop |
When you drop from types to actual model names, use the live inventory rather than memory: GET /v1/models on your platform returns the current lineup, and the platform documentation explains how to read it. Model directories written by the provider, Google's platform models page, for example, are the right place to check what each entry accepts before you spend a golden set on it.
Combine and replace without drama
A resilient arrangement gives each task family at most two candidates: a primary and a spare, both passing the golden set, both reachable through the same integration. Chasing a single winner for every family is how teams end up with a dozen integrations and no idea which one carries production traffic. Write the conclusion down where your team already keeps decisions, with the golden-set result attached; the note is the asset, and it is what makes the next replacement an afternoon instead of an incident.
Replacement follows the same discipline as selection. When a version retires or a candidate stops clearing its assertions, rerun the golden set against the replacement rather than trusting a changelog, and let the failure types decide again. The mechanics of keeping tool-side configuration portable are covered in our setup guide for running one key across coding tools, and the full lifecycle view, notice windows, alias layers, and rollback, is in the model migration playbook.
One page keeps this honest: write the six constraints, pull the samples, write the assertions, run the candidates, read the failure types, record the conclusion. Six columns of notes beat any spreadsheet of scores, because every entry maps back to a requirement you actually have.
Where to go next
- See how model choice plays into a wider workflow. Assigning research, writing, and review to different models is this guide's logic applied across a whole pipeline.
- Follow the product thread. More model-and-platform selection reading lives in the product guides.
Choose by constraint, converge by test, and record why. The leaderboard will move on; your constraints will still be there next quarter.