A team asks an API to recommend suppliers, saves the answer, and adds it to a dashboard labelled "ChatGPT visibility." The request succeeds. The response contains recognisable companies. The dashboard looks credible.
The label is the problem. An API response is an observation about a particular request and its configuration. It does not, by itself, show what someone saw while using ChatGPT search. Confusing the two can lead a product team to optimise the wrong system and a marketing team to report exposure it has never measured.
For developers working with multiple models, this distinction matters beyond brand monitoring. A model name is only part of an application. The supplied context, instructions, retrieval tools, and surrounding product behaviour help determine the answer a user receives.
Define the system before you compare the answers
An API test is easiest to interpret when its inputs are explicit. You can record the requested model, messages, tool definitions, available documents, and supported generation settings. You also control how the application processes the response before displaying it.
A consumer product has its own configuration and interface. Some of that behaviour may be visible; some may not be documented in enough detail to reproduce. Even if two experiences refer to the same model family, a matching prompt does not establish that they used identical context or retrieval.
This is not a reason to abandon API testing. It is a reason to name the result accurately. "Our application returned these suppliers with web retrieval enabled" is a useful observation. "ChatGPT recommends these suppliers to buyers" is a different claim that needs evidence from the named product and a defined set of buyer tasks.
Treat them as separate test programmes. Application tests answer whether your product behaves correctly. Product-observation tests answer what was displayed in an external experience under the conditions you recorded. Neither automatically substitutes for the other.
A generated answer is not proof that a search happened
A model can produce an answer about a company without opening its website during that request. Recognising a name does not establish the freshness of the information, and a URL in ordinary generated text is not itself a retrieval trace.
Where your application needs current information, configure an appropriate retrieval path and inspect its actual use. OpenAI's web-search documentation describes enabling web_search in the Responses API, search-call output items, and URL citation annotations. It also notes that enabling the tool allows the model to decide whether to search based on the prompt.[2]
That means "tool available" and "tool used" should be separate fields in a test record. Save the returned tool-call information rather than inferring a search from confident wording. If a request fails, returns an incomplete result, or does not use the expected retrieval path, preserve that outcome too.
Keep citation annotations when rendering a search-supported answer. OpenAI's documentation requires clearly visible, clickable inline citations when displaying information from its web results to users.[2] Flattening everything into a plain text string can discard the evidence your interface needs to show.
A citation still needs inspection. Check whether the cited page supports the attached statement, rather than awarding a quality point merely because the answer contains a link.
Separate supplier research from visibility measurement
Suppose a team is investigating how people find a geo agency singapore. A development test might ask an API-backed assistant to identify services, explain selection criteria, or compare supplied provider pages. Those tests can reveal whether the application retrieves useful evidence and describes it accurately.
A visibility study asks something else: whether, and how, a provider appears in a specified AI product during a set of relevant user tasks. The observation should distinguish an answer mentioning the brand, a citation linking to its website, and a recommendation with a stated reason. A source used for a factual point is not necessarily a recommended supplier.
If the test prompt explicitly asks about one company, finding that company in the answer says little about its likelihood of appearing in an unbranded discovery task. Branded and unbranded prompts can both be useful, but they belong in different groups.
Do not turn an absent mention into an outage or a definitive verdict on a provider's visibility. Save it as one result within the stated test conditions. Likewise, do not turn a single flattering answer into a claim about every buyer's experience.
Keep a record you can reproduce or honestly describe
For your own application, a useful test record includes:
- the exact user input and any application instructions sent with it;
- the endpoint, requested model identifier, and version information returned;
- the documents or conversation context supplied to the request;
- which tools were available and which actually ran;
- supported settings that could affect the answer, including retrieval filters or location configuration where used;
- the time of the request, returned answer, citations, errors, and retry history;
- the rule used to score the output and any human corrections to that score.
Store enough detail to investigate differences without casually retaining sensitive customer data. Use approved test material, restrict access to logs, and decide how long raw inputs need to be kept.
For observations of an external product, record the product, visible mode, query wording, time, conversation setup, relevant settings you can observe, and the displayed answer. Mark unknown configuration as unknown. Calling a test reproducible does not make hidden product settings available.
Separate failed observations from valid answers that contain no mention. A failed page load or interrupted response is not a clean negative result. Mixing those outcomes can make reliability problems look like changes in brand visibility.
Change one part of the application at a time
When evaluating a new model, hold the evidence pack and review criteria steady. When evaluating retrieval, keep the model and other supported settings stable where possible. If you change the prompt, model, documents, and search configuration together, you are comparing whole workflows rather than isolating one cause.
Both kinds of comparison are legitimate. Label them correctly. A complete workflow comparison helps a team choose what to ship; a controlled component test helps explain why one configuration behaves differently.
Run representative tasks more than once and retain the differences. Record the number of attempts and valid observations when reporting rates. Results from a deliberately selected test set describe that set; they are not automatically estimates of how often all users see a brand.
Avoid treating agreement between models as verification. Several systems can repeat a claim drawn from the same source or share an unsupported assumption. For factual accuracy, the deciding evidence is still the underlying material and the criteria the task was meant to satisfy.
Test the gateway as well as the model
String AI documents a shared API entry point and OpenAI-compatible authentication for supported integrations.[1] That is useful for teams comparing models through a common connection pattern. It should not be read as a promise that every upstream feature, hosted search tool, or response object works identically through every route.
Before relying on retrieval or citation handling, test the exact endpoint and model combination your application will use. Check that the intended tool is supported, the request preserves its configuration, and the response exposes the information your interface needs. Decide what happens when the feature is unavailable: return a clear limitation, use an explicitly configured alternative, or stop the task. Silently falling back to an answer without retrieval changes what the application can claim.
An API evaluation can be rigorous and commercially useful without pretending to reproduce a consumer product. Name the system you tested, preserve its configuration, and keep its evidence attached to the result. That gives developers something they can improve and marketers a measurement they can explain.
Sources
[1] https://www.string.ink/docs
[2] https://developers.openai.com/api/docs/guides/tools-web-search