Why Your Business Shows Up in ChatGPT One Day and Vanishes the Next

Here is something nobody selling AI search services likes to admit. Ask ChatGPT the same question twice and you will often get two different answers, with two different sets of businesses named. That is not a glitch. That is how these tools work. And it changes what it means to say your business is winning in AI search.

I want to walk through why that happens, because once you see it, you will never look at an AI ranking screenshot the same way again.

The Proof That Is Not Proof

Picture this. Whoever handles your marketing sends you a screen grab of ChatGPT. There is your company, named right in the answer when someone asked the tool for the best supplier in your category. See, they say, you are showing up in AI search. Proof it is working.

Then you open ChatGPT yourself an hour later, ask the same question, and your company is nowhere in the answer.

Neither result is fake. Both are real answers the tool gave. The catch is that a picture of one answer captures a single roll of the dice, and these tools roll the dice every time someone asks. Treating that one captured answer as proof is like calling a coin weighted because it came up heads once. You did not measure the coin. You just watched one flip.

That captured answer is the thing a lot of people are selling as evidence right now. Once you understand why it proves almost nothing, the whole conversation about AI search changes.

What the Research Actually Shows

This is not a hunch. Over November and December of 2025, Rand Fishkin of SparkToro and Patrick O'Donnell of Gumshoe.ai ran the largest public test of this question so far. Six hundred volunteers ran twelve recommendation prompts through ChatGPT, Claude, and Google's AI a combined 2,961 times, and every response was turned into a ranked list of the businesses named so the lists could be compared.

Here is what stood out to me. The odds of any two responses returning the same list of businesses were under one in a hundred. The odds of getting that same list in the same order were closer to one in a thousand. Ask the same question, get a different answer almost every single time. Fishkin's own conclusion was that any tool selling you a ranking position in AI is selling you nothing.

Ahrefs found a result pointing the same direction from a slightly different angle. Comparing Google's two AI answer formats on the same search, the sources cited differed 87 percent of the time.

So if your whole picture of how you are doing in AI search comes from someone asking a question once and capturing the result, you are not measuring anything. You are looking at one flip of the coin.

The Part That Should Actually Give You Hope

Now here is the other side, and this is the part that matters for your business.

The individual answers bounce around. The overall pattern does not.

In that same study, the businesses that came up most often kept coming up, no matter how the question was worded. In a tight category like headphones, where there are maybe two dozen names in play, the leaders appeared in 55 to 77 percent of runs. Any single answer was unpredictable. Across enough of them, a clear pattern held.

Now the honest caveat, because it applies directly to most businesses reading this. That range comes from a category with a short list of contenders. In a crowded category, say marketing agencies or design firms, where there are tens of thousands of options nationally, the same study found the percentages run dramatically lower. Nobody owns 70 percent of those answers. The bar for showing up consistently is different, and anyone quoting you headphone-sized numbers for a crowded service category is quoting the wrong benchmark.

What holds in both cases is the shape of it. Being mentioned once means very little. Being mentioned at a rate you can measure, across many versions of the question, means quite a lot. That rate is the real thing worth watching. And it is the thing a single captured answer can never show you.

So How Do You Actually Measure It

You measure it the way you would measure anything that varies. You take a lot of samples, not one.

Instead of asking a tool your key question once, you ask it many times, across the different ways a real buyer would phrase it, and you track how often your business comes up. That gives you a rate. A real number you can watch over time and actually improve.

For scale, the SparkToro study ran each of its questions 60 to 100 times per tool to get a sample worth drawing conclusions from. That is the order of magnitude real measurement takes. It is also why a screenshot and a baseline are not the same species of thing. A screenshot is one moment. A baseline is a measurement you can stand on.

One more wrinkle worth knowing about. The same researchers found that when ordinary people were asked to write their own prompt for the same underlying need, they almost never phrased it alike. So sampling one perfectly worded question many times still misses most of reality. Real measurement varies the wording too.

If you want the longer version of how that measurement works in practice, it is laid out on our answer engine optimization services page.

What to Ask Whoever Handles Your Marketing

You do not need to run any of this yourself. But you should know what a real answer sounds like, so here is what I would ask.

Ask them how they measure your AI visibility. If the answer is a single captured answer or a one-time check, you have your answer, and it is not a good one. If the answer involves sampling the same question many times, in the different ways a buyer would ask it, and tracking a rate over time, they are measuring the thing that actually moves.

The honest truth is that AI search is new enough that a lot of people are selling confidence they have not earned. The businesses that come out ahead will be the ones who insist on real measurement instead of a good-looking screen grab.

That is the whole game right now. Measure it honestly, or do not bother claiming you measure it at all.

Start a Conversation

Common Questions

Why do AI tools give different answers to the same question?

These tools generate a fresh response each time rather than pulling a fixed list. So the same question can return different businesses in a different order on each ask. It is normal, not a malfunction.

Does that mean AI search cannot be measured?

It can, but not from a single result. You measure it by sampling the same question many times and tracking how often your business appears. That rate is stable enough to watch and improve, even though any one answer is not.

How many samples does it take to see a real pattern?

The largest public study on this ran each question 60 to 100 times per AI tool to get a sample worth drawing conclusions from. Fewer runs than that, and you are looking at noise rather than a pattern.

Is a screen grab of my business in ChatGPT proof that AI search is working?

No. A single captured answer is one result out of many possible results. It shows your business can appear, not that it appears consistently, which is the thing that actually matters.