Ask ChatGPT About Your Business 100 Times Before You Trust the Answer
Updated: 2 days ago
Your business can show up in ChatGPT today and be missing from the same answer an hour from now. That is normal for these tools, and it changes how AI visibility should be measured. Ask the same question twice and you will often get two different answers, with two different sets of businesses named.
We want to walk through why that happens, because once you see it, a screenshot of an AI answer stops looking like proof.
Key Takeaways
|
A Screenshot Is One Roll of the Dice
Picture this. Whoever handles your marketing sends you a screen grab of ChatGPT. There is your company, named in the answer when someone asked for the best supplier in your category. You are showing up in AI search, they say. Proof it is working.
Then you open ChatGPT yourself an hour later, ask the same question, and your company is nowhere in the answer.
Both results are real answers the tool gave. A picture of one answer captures a single roll of the dice, and these tools roll the dice every time someone asks. Calling that one answer proof is like calling a coin weighted because it came up heads once. One flip is not a measurement.
That captured answer is what a lot of people are selling as evidence right now. Once you see why it proves so little, the whole conversation about AI search changes.
What the Research Shows
Over November and December of 2025, Rand Fishkin of SparkToro and Patrick O'Donnell of Gumshoe.ai ran one of the largest public tests of this question to date. Gumshoe sells AI tracking tools, and Fishkin disclosed that up front. Six hundred volunteers ran twelve recommendation prompts through ChatGPT, Claude, and Google's AI a combined 2,961 times. Each response became a ranked list of the businesses it named, so the lists could be compared.
The odds of any two responses returning the same list of businesses were under one in a hundred. The odds of the same list in the same order were under one in a thousand. Same question, a different answer almost every time. Fishkin's verdict on tools that sell you an AI ranking position was one word: baloney.
Ahrefs found the same pattern from another angle. Comparing Google's two AI answer formats, AI Overviews and AI Mode, on the same searches, it found the cited sources differed 87 percent of the time.
So if your whole picture of AI search comes from someone asking a question once and capturing the result, you are looking at one flip of the coin.
Why This Is Good News for Your Business
The other side of this is the part that matters for you.
Individual answers bounce around. The overall pattern holds.
In a follow-up that used prompts real people wrote, the leading headphone brands, Bose, Sony, Sennheiser, and Apple, appeared in 55 to 77 percent of responses, however the question was worded. Any single answer was unpredictable. Across enough of them, a clear pattern held.
That range comes from a category with a short list of familiar brands. In a crowded field like brand design agencies, the leaders showed up closer to 30 to 40 percent of the time. The bar for showing up consistently is different there, and anyone quoting you headphone-sized numbers for a crowded service category is quoting the wrong benchmark.
What holds in both cases is the shape of it. Being mentioned once means little. Being mentioned at a rate you can measure, across many versions of the question, means a lot. That rate is the number worth watching, and a single captured answer can never show it to you.
Measure AI Visibility as a Rate
You measure it the way you would measure anything that varies: take lots of samples, not one.
Ask the key question many times, across the different ways a real buyer would phrase it, and track how often your business comes up. That gives you a rate, a real number you can watch over time and improve.
For scale, the SparkToro study ran each question 60 to 100 times per tool to get a sample worth drawing conclusions from. That is the order of magnitude real measurement takes. A screenshot is one moment. A baseline is a measurement you can stand on.
One more wrinkle. When the researchers asked people to write their own prompt for the same need, almost no two looked alike. So sampling one perfectly worded question many times still misses most of how buyers ask. Real measurement varies the wording too.
If you want the longer version of how we run that measurement, it is on our answer engine optimization services page.
What to Ask Whoever Handles Your Marketing
You do not need to run any of this yourself. You should know what a good answer sounds like, so here is what we would ask.
Ask how they measure your AI visibility. If the answer is a single captured answer or a one-time check, that tells you what you need to know. If it involves sampling the same question many times, in the different ways a buyer would ask it, and tracking a rate over time, they are measuring the number that moves.
AI search is new enough that a lot of people are selling confidence they have not earned. The businesses that come out ahead will be the ones who insist on real measurement instead of a good-looking screen grab.
Common Questions
Why do AI tools give different answers to the same question?
These tools generate a fresh response each time rather than pulling a fixed list. So the same question can return different businesses in a different order on each ask. It is normal, not a malfunction.
Does that mean AI search cannot be measured?
It can, but not from a single result. You measure it by sampling the same question many times and tracking how often your business appears. That rate is stable enough to watch and improve, even though any one answer is not.
How many samples does it take to see a real pattern?
The SparkToro study ran each question 60 to 100 times per AI tool to get a sample worth drawing conclusions from. Fewer runs than that, and you are looking at noise rather than a pattern.
Is a screen grab of my business in ChatGPT proof that AI search is working?
No. A single captured answer is one result out of many possible results. It shows your business can appear, not that it appears consistently, and consistency is the part that counts.
Find Out How Often You Show Up
Want to know how often your business shows up in AI answers? We will measure it as a rate across many versions of the question, the way this post describes.
Request a consultation below.