Choose comparable questions

Start with buyer tasks that can be asked naturally on each selected surface. Keep the task and constraints consistent while acknowledging that interfaces may work differently. Record the exact wording rather than claiming tests were identical when one required a different interaction. Decide whether you are comparing answer usefulness, brand mentions, citations, or referrals. A single league table can conceal these distinctions. Avoid treating one product's displayed order as equivalent to another's source panel, and do not assume every answer provides a ranked list of recommendations.

Record the actual surface

Name the consumer product, mode, or API used. 'Google' and 'ChatGPT' alone may be too broad for a reproducible observation. Keep search settings, location, language, and conversation context where relevant. An API response belongs in an API series, even if its model family is associated with a consumer product. Google notes that its AI surfaces can differ in their models and techniques. Therefore, keep AI Overviews and AI Mode observations separate rather than merging them into one result that hides which experience actually displayed your page.

Use a common annotation guide

Apply the same definitions for confirmed mention, visible citation, relevant recommendation, and factual error. Review uncertain identity matches manually. Count absent features appropriately: if a surface does not expose a comparable source list, mark the field unavailable instead of zero. Hypothetical example: a software buyer task produces a named recommendation on one surface and an explanatory answer on another. The first can score a recommendation event, but the second should be judged on whether it fulfilled the request. A missing recommendation is not necessarily an inferior answer.

Present differences with evidence

Show a small set of matched examples alongside summary counts. Readers can then understand whether differences arise from product fit, source selection, presentation, or measurement limitations. Open cited pages to verify the claims they support. Avoid interpreting citation totals as audience size or engine market share. A sample collected by your team says what happened in that sample. If the comparison spans several days, disclose timing and avoid giving one surface many more attempts without explaining the weighting. Keep failed requests visible in a separate collection-status column.

Use comparisons to choose work

The outcome should guide an action: fix inaccurate facts everywhere, investigate a specific crawler path, improve an underexplained task, or refine the measurement design. Do not create contradictory pages to chase each surface's latest response. A consistent, evidence-backed public record helps buyers across channels. Revisit the same tasks after substantive changes and keep the engine-specific series intact. When a surface changes its interface or collection method, mark the break. This preserves a useful history without pretending that a technically different test continues the exact same experiment. If stakeholders request a winner, specify the criterion first and explain why that criterion matters to the customer decision.

PUT IT TO WORK

Use matched buyer tasks and separate reporting to compare search surfaces responsibly.

Sources & further reading

Official documentation informs this guide. Examples are illustrative, and product interfaces or limits can change.

THE WORKSPACE WE'RE BUILDING

From search signals to a useful first draft.

SERP.so is in development. Join the waitlist for a workspace that connects research, content review, and publishing.

Join early access ↗