Yes. It is possible to track brand mentions in AI search, and you can start today at no cost.
That is the short answer, and it is where every other article on this question stops. The useful answer is longer because "possible" is not a yes-or-no property. Tracking is possible to a precision, and almost nobody measuring AI visibility right now knows what their precision is. Most of them are reporting weekly changes that their setup is mathematically incapable of detecting.
So this piece answers a sharper version of the question: how much of a change in your brand's AI visibility can you actually detect, and how would you know? There is a calculator at the end that answers it for your specific setup, and a simulation that shows what happens when you skip the question.
This is a companion to our earlier guide on how to track brand mentions in AI search, which covers the crawler-verification side — proving that the AI bot traffic in your logs is genuinely from OpenAI or Anthropic rather than a forged user agent. That article deals with what you can verify. This one deals with what you can estimate, and how far to trust the estimate.
What is measurable, what is estimable, what is neither
The honest feasibility answer splits into three tiers. Confusing them is the root of most bad AI visibility reporting.
| Question | Status | Method |
|---|---|---|
| Did the engine's crawler fetch my page? | Measurable | Server logs, verified against vendor-published IP ranges. This is a fact, not an estimate. |
| Did anyone arrive from an AI answer? | Partly measurable | Referral data. ChatGPT tags citation links with utm_source=chatgpt.com. Others often send no referrer at all, so this is a floor, never a total. |
| How often does the engine name my brand? | Estimable only | Repeated prompt sampling. It is a statistic with an error bar, not a number. |
| How many people saw that mention? | Not measurable | No engine publishes answer impressions by brand. Google's Search Console generative AI reports show your own impressions, not your share of anyone's answers. |
Tier one is a fact. Tier three is a survey. When a dashboard prints "34%" in the same font, it loses the distinction that matters most.
Why is your mention rate a survey, not a reading
Large language models are non-deterministic — the same prompt can return different brands on consecutive runs. Ask ChatGPT "best CRM for small business" ten times, and you may get your brand named six times, or three, with nothing having changed in the world.
This makes a mention rate a sample proportion, exactly like a political poll. And like a poll, it is meaningless without an interval. A poll saying "42%" is useless; "42% ± 3" is information.
The right interval for this job is the Wilson score interval, which remains accurate at small sample sizes and near 0% or 100%, where the textbook formula breaks down. For a typical setup — 50 prompts run 5 times each, a 30% mention rate — it looks like this:
Reported rate with interval : 30.0% (95% CI 24.7% to 35.9%) Interval width : +/- 5.6 points
So a tool reporting "you appear in 30% of answers" is really reporting "somewhere between 25% and 36%." If last week said 33% and this week says 30%, nothing has been detected. Those two intervals overlap almost completely.
The mistake hiding in "250 data points"
Here is where the standard approach quietly breaks, and it is a subtle enough error that it survives in tools sold for real money.
Fifty prompts run five times each are routinely described as 250 data points. It is not. Runs of the same prompt are not independent observations. If "best CRM for small business" happens to be a prompt where your brand is strongly associated, all five runs will tend to mention you. They are five looks at the same underlying question, not five independent questions.
Statisticians call this clustering, and ignoring it understates your standard error. The practical consequence is that your effective sample size is closer to the number of prompts than the number of runs. Which leads to the single most useful rule in this article:
Adding runs barely improves your precision. Adding prompts improves it a lot. If your tracking budget doubles, spend it on a wider prompt set, not on re-running the prompts you already have.
The calculator below makes that concrete. Same assumed 30% mention rate:
prompts runs obs floor
50 5 250 27.5p
50 10 500 27.5p <- doubled the runs, gained nothing
100 5 500 19.3p <- doubled the prompts instead
200 5 1000 13.5p
Doubling runs from 5 to 10 moves the detection floor by nothing at all. Doubling prompts from 50 to 100 improves it by 8 points. Most tracking setups are configured backward.
I simulated what happens when you ignore this
The claim that clustering matters is easy to assert and easy to test, so I tested it.
I simulated a brand whose true standing did not change at all between two tracking cycles, then ran the comparison two ways: the pooled z-test that treats every run as independent (what a naive dashboard does), and a paired test that treats each prompt as the unit of analysis (the correct one). Under a true null, an honest test should declare a significant change 5% of the time. Anything higher is a false alarm rate.
I varied one parameter: engine-side churn, meaning how much the models themselves drift between cycles for reasons unrelated to your brand — index refreshes, model updates, retrieval changes.
False-alarm rate under the null, brand did NOT move (nominal 5%)
design churn naive paired-t
50 prompts x 5 0.0 3.9% 4.9%
50 prompts x 5 0.5 6.2% 4.8%
50 prompts x 5 0.8 8.8% 4.7%
50 prompts x 5 1.2 13.3% 4.6%
100 prompts x 3 0.0 3.6% 4.4%
100 prompts x 3 0.5 5.1% 4.9%
100 prompts x 3 0.8 7.1% 5.6%
100 prompts x 3 1.2 9.8% 5.6%
Read the last row of each block. When the engines are churning, the naive comparison declares a significant change 13.3% of the time on a brand that did not move, nearly three times the 5% it advertises. The paired test stays calibrated at 4.6% throughout.
That is the quantified answer to "is it possible to track brand mentions in AI search." It is possible. It is also possible to do it in a way that generates a false alarm on more than one report in eight, and the failure mode is invisible, because a false alarm looks exactly like a result.
One caveat I want to be straight about: the churn parameter is not something I can look up. Nobody publishes how much ChatGPT or Gemini drifts week to week. Which is precisely the argument for measuring your own instead of assuming it.
Measure your own noise floor: run an A/A test
This is the most useful part of this article, and it requires one extra tracking cycle.
Before you trust any AI visibility report, run your prompt set twice with nothing changed in between. No new content, no digital PR, no fixes. Whatever movement you observe is your noise floor — the amount your number wanders on its own.
Then apply the rule: only report changes larger than your A/A movement. If your A/A test moves 6 points, a 5-point "improvement" next month is not an improvement, whatever the dashboard says.
Here is the calculator run on two cycles I generated with identical true brand standing — by construction, nothing changed:
Prompts compared: 50 cycle A: 33.2% (95% CI 27.7%-39.3%) 83/250 cycle B: 25.2% (95% CI 20.2%-30.9%) 63/250 observed shift: -8.0 points test statistic verdict --------------------------------------------------------------- pooled z (treats runs as indep.) -1.97 SIGNIFICANT paired t on prompt rates -1.59 not significant Your measured noise floor: +/- 10.1 points READ THIS. The naive test says you moved; the correct one says you did not. The apparent change sits inside prompt-level variation. Do not report this as a win.
An 8-point drop, flagged as statistically significant by the standard method, on data where nothing changed. If that had been your real Monday report, someone would have spent the week investigating a decline that never happened.
Run it on your own numbers
The calculator is a single Python file with no dependencies. Two modes.
Plan a setup before you commit to it:
python3 noise_floor.py --plan --prompts 50 --runs 5 --rate 0.30
Compare two cycles, exported as CSV with one row per prompt:
python3 noise_floor.py --compare week1.csv week2.csv
The CSV format is deliberately trivial, so you can export it from any tool:
prompt_id,mentions,runs best-crm-for-smb,3,5 crm-alternatives,0,5
It reports the Wilson interval, both tests, and your measured noise floor, based on the spread across prompts rather than on an assumption.
Seven rules that follow from the maths
- Freeze the prompt set. Changing prompts between cycles means you measured your prompt list rather than your brand. The comparison is void.
- Buy prompts, not runs. Three to five runs per prompt is enough. Everything else should go into breadth.
- Run an A/A test first. One wasted cycle buys you a defensible threshold for every cycle after.
- Never report a bare percentage. "31% (95% CI 26–37%, 50 prompts × 5 runs)" is a finding. "31%" is a vibe.
- Analyze at the prompt level. Paired test on prompt rates, not a pooled test on runs.
- Report each engine separately. Blending ChatGPT, Gemini, Claude, and Perplexity into one score averages away the only actionable information you have.
- Match the cadence to the floor. If your noise floor is 27 points, weekly reporting is theatre. Report quarterly, or widen the prompt set until weekly becomes meaningful.
So — is it possible?
Yes, with the precision stated. Crawl eligibility is genuinely measurable, and you can verify it against vendor-published IP ranges rather than trusting a user-agent string, which is what the companion guide is about. Referral traffic is partly measurable and gives you a reliable floor. Mention rate is estimable to roughly ±5 points on a decent design, and detectable changes start at about 15 to 20 points on the setups most teams actually run.
What is not possible is a single confident number with no interval, and anyone selling you one is selling certainty rather than measurement. The difference between a brand that tracks AI search well and one that tracks it badly is not the tool. It is whether they know their own noise floor.
Quick answers
Is it possible to track brand mentions in AI search?
Yes. Crawler activity is directly measurable in server logs, referral traffic is partially measurable, and the mention rate is estimable through repeated prompt sampling. Only the last one needs a confidence interval, and it is the one most tools report without one.
Is it possible to track brand mentions in AI answers accurately?
To a stated precision, yes. On a 50-prompt, 5-run design at a 30% mention rate, the 95% interval is roughly ±5.6 points, and the smallest reliably detectable change is around 20 points. Anything smaller is inside the noise.
How many prompts do I need to track AI brand mentions?
More than you think, and more prompts rather than more runs. Doubling runs from 5 to 10 barely move the detection floor because runs of the same prompt are correlated. Doubling prompts from 50 to 100 improves it substantially.
Why does my AI visibility score change when nothing has changed?
Because language models are non-deterministic, and the engines themselves drift between cycles. In simulation, a naive week-over-week comparison flagged a significant change on an unchanged brand up to 13.3% of the time. Run an A/A test to measure your own noise floor.
What is an A/A test for AI visibility?
Running your frozen prompt set twice with no changes in between. The movement you observe is your noise floor. Only report changes larger than it.
Method note
Simulations used a beta-binomial model with prompt-level heterogeneity (Beta(3,7), mean 30%) and a logit-normal week-to-week perturbation representing engine churn, at 3,000 trials per cell. The analytic minimum-detectable-effect calculation was validated against a Monte Carlo simulation and reproduced the target 80% power to within 0.008. The calculator and its test suite are published, so the numbers here can be reproduced or refuted.