Question
Why Claude and Gemini Recommend Different Sources for the Same Query
Chat assistants 4 min read
One explanation you may hear for the difference is that Claude answers only from its training data while Gemini searches the web. That has been out of date since 2025. Anthropic announced web search for Claude on March 20, 2025, and on May 27, 2025 made it “available globally on all Claude plans.” When Claude uses information from the web, Anthropic says, “it provides direct citations so you can easily fact check sources.” Both assistants can now search. They still recommend different sources, for reasons that are more structural than a yes-or-no on web access: they reach the web through different crawlers and permissions, they decide differently when to search, they run on different models, and each varies from one response to the next.
They reach the web through different crawlers
Each company documents its own crawlers, and a site’s robots.txt can treat them differently.
Anthropic’s help page on its crawlers describes three:
- ClaudeBot collects web content that could contribute to model training. Restricting it signals that a site’s future materials should be excluded from training data.
- Claude-SearchBot “navigates the web to improve search result quality for users.” Disabling it prevents Anthropic from indexing the site’s content for search, which Anthropic says “may reduce your site’s visibility and accuracy in user search results.”
- Claude-User visits pages when a user’s question prompts it. Disabling it “prevents our system from retrieving your content in response to a user query, which may reduce your site’s visibility for user-directed web search.”
On Google’s side, the relevant control for Gemini is Google-Extended. Google’s crawler documentation describes it as a robots.txt token that governs whether a site’s content is used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. It has no separate user agent, and Google states that it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.”
The practical consequence: a site can be available to one assistant and not the other. A publisher that blocks Claude-SearchBot but not Google-Extended, or the reverse, can see the difference reflected in which assistant can cite it. Two sites with identical content can end up in different candidate pools purely because of their robots.txt choices.
They decide differently when to search
Having web access does not mean using it for every question. Anthropic describes web search as something that “expands Claude’s extensive knowledge base with real-time insights,” which implies a combination of built-in knowledge and retrieved pages. When an assistant answers partly or wholly from what its model already knows, the sources it recommends come from training, not from a live search.
Neither company’s crawler or product documentation describes the rules that decide when its assistant searches, how many pages it retrieves, or how it ranks them. Two assistants given the same query may therefore search in one case and not the other, or search with different reformulations of the question. That alone can produce different source lists.
They run on different models
Claude and Gemini are built by different companies on separately trained models. Even when both retrieve pages for the same question, each model decides which retrieved pages best support its answer and how to present them. There is no shared selection standard between companies, and neither publishes its weighting. Differences in recommended sources are the expected result of two independent systems, not a sign that one is broken.
Each one varies on its own
The differences between assistants sit on top of variation within each one. A SparkToro study published in January 2026 had 600 volunteers run 12 prompts through ChatGPT, Claude and Google’s AI 2,961 times. The researchers found less than a 1 in 100 chance that ChatGPT or Google’s AI would return the same list of brands in any two responses if asked 100 times, and that seeing two lists in the same order was closer to 1 in 1,000 runs.
If a single assistant rarely repeats its own list, comparing one Claude answer with one Gemini answer says little. Part of the difference you see may be variation that would have appeared within either assistant alone.
What this means for your site
- Check your robots.txt for each company. Decide deliberately about Claude-SearchBot, Claude-User and ClaudeBot, and about Google-Extended. Search visibility and training are separate choices for both companies.
- Do not treat one assistant as a proxy for another. Visibility in Gemini says little about Claude, and the reverse.
- Measure with repetition. Use a fixed set of prompts, run them several times in each assistant, and look at how often your brand or pages appear rather than at any single answer.
- Work on what both can find. Clear, current, well-sourced pages that answer specific questions are useful to any assistant that retrieves them. The parts that differ between engines are mostly outside your control.
Treat each assistant as its own channel. They reach the web through different doors, which robots.txt controls separately; they choose when to search and how to rank what they find by rules neither publishes; and each varies from one answer to the next. Open the doors you want open, keep your pages worth citing, and measure each assistant on its own terms.