Why run a pilot at all if you cannot publish it?

A pilot tells you whether an effect is worth spending money to measure properly, and it tells you where the measurement will break. It cannot tell you the size of the effect, because one question with three runs has no way to separate a real difference from ordinary run to run noise. Ours was worth running and was not worth publishing.

What is a pre-registered threshold?

It is the number that decides what counts as a finding, written down before any data is collected. Ours said the two measurement routes count as different only if the median overlap ratio came in at or below 0.75 on both metrics. Fixing it in advance is what stops a result being reverse engineered from the data once you have seen it.

Why does an overlap score depend on how many sources each engine returns?

The Jaccard score divides shared domains by the total across both sets, so if one route returns 12 domains and the other returns 10, a perfect result still cannot reach 1.0. Part of any gap it reports is arithmetic rather than content. The overlap coefficient divides by the smaller set instead, which removes that ceiling, which is why both are reported.

← Signal

One Question Told Us Something Dramatic. Thirty One Told Us It Was Wrong.

By Ben Tabas, Founder · Published August 19, 2026 · 5 min read

The short version

We ran a pilot on one question. It produced a dramatic result about our own measurement instrument.
We then ran the same test on 35 questions and analysed 31. The effect shrank by roughly a third and landed on the wrong side of a threshold we had written down before collecting anything.
So this piece carries no finding about the thing we set out to test. It carries a finding about pilots, which is the more useful one.
Two metrics disagreed with each other. The brief said in advance that when they disagree we report the disagreement and draw no conclusion. We did.
Total cost of finding out: $4.61.

What we were testing

We reach Gemini two ways. One is the DataForSEO proxy, which we pay for and which every Gemini number Zion Labs has published has been measured through. The other is Google’s own grounded API, which is free inside its tier and would remove the single largest cost line in a full scan.

The question was whether the two routes hand back the same set of cited domains for the same question and the same model. If they do, we swap and the scan gets cheaper. If they do not, every historic Gemini baseline is locked to the route it was measured on, and swapping would silently break every quarter over quarter comparison we have sold.

These engines are non-deterministic, so two runs of one route already disagree with each other. The comparison is therefore never the raw cross route number. It is the cross route overlap divided by the same route’s own run to run baseline. Call that ratio r. At r near 1.0 the routes are interchangeable. At low r they are different instruments.

The pilot said one thing

One question, three runs down each route. Within the proxy, mean overlap 0.56. Within the direct route, 0.35. Across the two routes, 0.23 against a within route baseline of 0.46.

That is r of 0.50, and it looks enormous. Two routes to the same model, agreeing with each other half as well as either one agrees with itself. It is the kind of number that writes its own headline.

Thirty one questions said another

The brief specified 20 questions. We ran 35. That is a deviation and it belongs in the open: we widened the sample before collection began, the thresholds were never touched, and more data is the conservative direction for a test designed to find a difference. An article about pre-registration that hid its own change of plan would be worth nothing.

So: 35 questions, five runs per route, both routes interleaved in one session so a shift in Google’s index could not masquerade as a route difference. Four questions were dropped for returning fewer than four valid runs on a route, leaving 31 analysed. Both routes served gemini-3.5-flash throughout. On the direct route that is Google’s own reported version; on the proxy it is what the vendor tells us, which we cannot independently verify.

Median r came in at 0.671, with a 95% bootstrap interval of 0.618 to 0.716.

The pre-registered bar for calling the routes different was r at or below 0.75 on both metrics. Jaccard clears it. The second metric does not: on the overlap coefficient, median r is 0.768, interval 0.715 to 0.824. The independent depth trigger, set at 1.5 times, did not fire either. Direct returned a median of 12 domains per run against the proxy’s 10, a factor of 1.20.

The brief, written before collection, said that if the two metrics point opposite ways the study reports the disagreement and draws no conclusion. That is the verdict. We did not prove the routes are different instruments, and this article is not evidence that they are.

Why the second metric mattered

Most readers will not have met the mechanic that split the two numbers, and it is worth two sentences.

Jaccard counts shared domains over the total across both sets. When one route returns 12 domains and the other returns 10, a flawless result is capped below 1.0 before content is considered at all, because the two sets cannot be the same size. The overlap coefficient divides by the smaller set instead, which removes the ceiling.

So part of the gap the pilot found was arithmetic. Direct simply returns more sources per run than the proxy does, and Jaccard charges the routes for that. One metric could not tell the difference between two routes reading the world differently and two routes returning different sized lists. Two metrics could.

There is still something under it

A signed rank test across the 31 questions gives z of -4.86, p of 1.17e-06. Cross route overlap is reliably lower than within route overlap. The effect is real and it is not noise.

It is simply smaller than one question made it look, and it does not clear the bar we set for ourselves before we looked. A real effect that misses a pre-registered threshold is not a finding. It is a reason to run a bigger study later.

What we did as a result

Nothing. We did not switch routes, every published Gemini baseline stays on the route it was measured on, and the cost saving stays uncollected. Inconclusive means no swap, not a re-run with the threshold moved.

That is the whole product. One test told us something dramatic and false. Thirty one told us the truth. The only reason the first one never went out under our name is that the numbers deciding what counted as a finding were fixed in the brief before a single call was made. This is the Fixed Set Method applied to ourselves rather than to a client: fix the questions, run each one several times, and write down what would change your mind before you can see which answer you would prefer.

Limits

One engine family, so this says nothing about whether the proxy reshapes ChatGPT, Perplexity or Claude, all of which we also measure through it. One category, crypto and fintech. One collection day, so a cross section and not a trend. 31 questions of 35, with 4 dropped for too few valid runs. And a comparison of domain sets can only ever say which route differs, never which route is right. Neither route is ground truth, and there is no correct citation set to check either of them against.

Revision history

August 19, 2026 Retitled from 'One Question Is Not a Study' to the specific version of the same point, matching the line the research index already used for this piece. No figure or body claim changed.
August 19, 2026 Published.