How to tell if your documentation search actually got better

How we researched this

We examined the original TREC judgment guidance, ranked-retrieval methods, Elastic’s evaluation API, a controlled click study, Algolia’s discussion of online metrics and the BRIGHT research project. We then constructed a four-query example to show how two scores can disagree. All example queries, ranks and proposed release rules are illustrative. No customer queries or live search systems were used. BRIGHT’s leaderboard and paper describe the same project and count as one line of evidence.

What counts as finding the right guide?

Imagine someone typing rotate signing secret into the docs. The first result is titled “Signing secrets.” It explains how signatures work but gives no rotation procedure. The next result describes a retired dashboard. Both pages match the words. Neither tells a current administrator how to rotate a credential without interrupting incoming requests.

Write the acceptance condition before looking at a ranking: “A workspace administrator on the current version can reach instructions that explain when each secret is accepted and how to verify the switch.” Now a configuration reference and a troubleshooting guide can both qualify if each contains those instructions. A page titled exactly like the query can fail. The unit of judgment is a reader trying to accomplish something with a particular result.

Keep three decisions separate. Does a correct answer exist in the published documentation? Does search return it for this reader? Can someone recognize it and act on it after opening the result? The first calls for a content inventory, the second for a ranked result list, and the third for an observed task. Changing the search engine only addresses part of that chain. Elastic’s ranking evaluation API makes this separation concrete: it consumes example searches and human ratings for documents, then scores the returned order. A human still has to decide what a useful answer contains. Elastic ranking evaluation

For the rotation example, record the product version and the reader's permissions alongside the query. If the result needs an owner account, say so. An owner-only guide might be exactly right for one reader and lead another into a dead end. Give those two tasks different case IDs. Otherwise the apparent disagreement between reviewers will live on as a noisy score.

Which searches belong in the test?

Start with decisions that could change what you ship. For a new title boost, include a lookup whose title really is the best answer and a troubleshooting question whose answer is buried under a different title. For a synonym rule, include both the meaning you intend and a page where the same word means something else. The most revealing case is often the one that could make you reverse the change.

These are proposed cases for an imaginary product. A real test pack needs the language readers actually use. Take a sample of recurring queries, then deliberately add rare tasks whose failure has serious consequences. Keep those groups labeled. A deliberately oversampled credential-recovery case deserves attention on its own, but it should not quietly become a proxy for a large share of all search traffic.

Record the unit you sample. Ten different spellings of the same error code can test spelling tolerance. Counting them as ten independent reader needs makes that one family unusually influential. Keep the raw variations for diagnosis, group them under one task, and decide up front whether the headline score gives each task or each query equal weight.

Research on hard retrieval questions illustrates why this choice matters. Hongjin Su and colleagues designed BRIGHT around questions that require reasoning to connect a query to relevant material. The project reports average nDCG at ten results across twelve datasets. That is a defined collection and scoring rule. It tells a docs team why to include harder questions alongside title lookups, but supplies no expected score for its own help center. ICLR paper · BRIGHT evaluation

How do you judge both rankings fairly?

Freeze the published content, filters, locale, permissions, and index settings that the test should hold constant. Save the old and proposed result lists with stable document IDs and destination anchors. A changed anchor matters if the result still names the correct long reference page but now drops the reader dozens of sections away from the answer.

Pool the results from both versions down to the chosen depth. Deduplicate identical destinations for judging, hide which version retrieved each one, and have a reviewer open the pages. Keep the original ordered lists for scoring. Pooling stops a candidate from being marked wrong merely because it found a useful page the old system missed. NIST uses pooled judgments in TREC and cautions that the judgments belong to the document collection on which they were made. A local pool is a practical review boundary, not a census of every useful page. NIST relevance judgments

Use an explicit rubric. Our proposed labels are 2 for a destination that answers the task under the stated conditions, 1 for useful background that leaves a necessary step unresolved, and 0 for an irrelevant or inapplicable answer. Record unjudged separately. If a result is hazardous, such as a procedure for deleting a live credential when the task is to remove a local copy, attach a separate risk flag. A zero alone cannot distinguish a harmless glossary page from actively misleading instructions.

Write a short reason next to every disputed rating. “Wrong version: parameter removed in v2” gives an engineer something to fix. “Not great” does not. If two reviewers disagree, check whether the task was underspecified before averaging their ratings. Sometimes both are right about the document and disagree about who was searching. Split the case or document the competing interpretations.

When a new candidate introduces an unjudged result near the top, reopen the pool and review it before deciding. Elastic explicitly exposes unrated documents in its evaluation response. A sheet with an empty rating should be just as visible. In our release procedure, an unjudged result in the scored window calls for review. The API’s list of unrated documents tells you where ratings are missing. How missing ratings affect a score depends on the metric and its settings. Record both the missing count and your treatment. Elastic response details

A judgment sheet can be as simple as this row. Give the same case ID to both result lists and keep the reviewer’s reasons even after resolving a disagreement.

The rating describes what the destination contains. Record a misleading snippet separately, and reserve task completion for an observation of a reader doing the work. Our risk flag and labels are editorial choices. TREC supplies the collection and judgment concepts.

How can a better average hide a worse release?

Consider four invented queries with exactly one fully acceptable result each. For this example the visible window is the first three results. Give each query a score of one divided by the position of its first acceptable answer, or zero if none appears within three. Average the four scores with equal weight. This is MRR@3 under a binary definition of an acceptable answer. A zero says only that no acceptable result appears in the scored window. The cutoff and relevance threshold matter as much as the name of the metric. Reciprocal rank in Elastic

Before the change, the average is (1/3 + 1/3 + 1/2 + 1) / 4, or about 0.542. Afterward it is (1 + 1 + 1/2 + 0) / 4, or 0.625. Yet the share of tasks with any acceptable result in the first three drops from 4 of 4 to 3 of 4. Both calculations are correct. The average rewards the two jumps to first place. The coverage count exposes the lost answer. Neither tells you how often a real administrator faces a personal access key leak.

For this fixture we would hold the release: the must-find credential task no longer passes. That decision follows a policy about the consequences of a miss. Choosing a different metric after seeing the outcome would make the next release harder to interpret. Write the critical-case rule before you tune the search and keep the per-query rows with the average.

A different task can demand a different score. Reciprocal rank ignores everything after the first acceptable result. That makes sense when someone needs one exact command. When a reader is comparing authentication approaches, several useful results in a good order matter. A graded discounted-gain measure can give credit for highly useful answers near the top. Its interpretation depends on what your grades mean and which answers enter the ideal ranking used for normalization. Avoid comparing values across packs with different labels, depths or pools as though they described one population. Discounted cumulative gain in Elastic

What should change when a query fails?

Open the result list and trace the first place where the answer disappears. Start by finding the intended page through navigation. If a reviewer cannot locate any current procedure, ask the feature owner to confirm whether the product supports the task. A search result should not promise a capability the product lacks. The right editorial change may be one sentence explaining that a request is unsupported, beside the closest supported alternative.

When testing section-level search, decide what counts as one result. Three anchors on one troubleshooting page may be excellent if they cover three distinct causes. They may also be three copies of the same introduction, crowding a useful independent guide off the screen. Score the actual unit displayed, then look at destination diversity separately. Changing the grouping logic changes what the reader sees even when the underlying page set does not change.

Keep a small holdout set that was never used to choose the fix. If you hand-write a synonym for every failed example and then retest only those examples, the result shows that those edits worked on familiar inputs. It leaves open whether the same change damages a neighboring use of the term. Ask a second reviewer for a paraphrase and a counterexample before shipping.

What do clicks and zero-result searches add?

A search that returns a generic overview has technically produced a result. That makes zero-result rate a useful alarm for empty lists and a poor detector of plausible-looking wrong answers. Algolia’s engineering discussion gives a concrete example: returning popular articles for unmatched queries can hide empty result pages without meeting the search intent. Check a sample of nonempty lists as well as the empty ones. Algolia on online search metrics

Clicks have a different limitation. In their 2005 SIGIR study, Thorsten Joachims and colleagues compared clicks with explicit judgments and eye tracking. Their ten search tasks included five navigational tasks and five informational ones. In the second phase, the researchers recruited 22 participants and recorded usable eye tracking for 16. Five of those 16 saw the top two Google results swapped. The authors found clicks informative but biased by how results were presented. Relative preferences inferred from some click patterns were useful on average. This was a web-search experiment, not a measurement of the percentage of docs users who trust the first link. Study design and findings

Suppose clicks fall after your change. Did a better excerpt answer the question without a click? Did the correct answer disappear? Did the new component stop reporting keyboard selections? All three can produce the same dashboard direction. Check the event path and observe a person trying the task before attributing the movement to quality. A useful follow-up is to ask what the person would do next and verify it against the answer conditions, without pointing to the intended result.

For an online comparison, record the exposure unit, event definition, eligible traffic, test period and any missing events. For a controlled test, decide how to assign readers at random and which outcome will decide the result before the test starts. Keep the event definition stable across variants. A before-and-after comparison alone cannot isolate the change from a new release or a different mix of visitors. If a site has too little traffic for a credible estimate, report that uncertainty. A handful of observed tasks can reveal a concrete failure worth fixing without estimating how common it is. Keep any claim about reduced tickets or faster onboarding for a study that actually measures those outcomes.

What belongs in the release decision?

Save a short decision record with the tested content revision, query-pack version, old and proposed settings, visible depth, rating threshold and reviewer. Attach the actual before-and-after order for every critical task. Include the newly judged pages, disagreements and held-out cases. This lets a later editor distinguish a ranking regression from a content change that invalidated a previously good answer.

For the four-query example, the decision record could read: “Hold the title boost. The install and invoice lookups both moved to first. The v1 guide stayed second. The revocation procedure left the first three. Inspect which generic key-management result displaced it, narrow the boost, and rerun both the failure and an unrelated key lookup.” That tells the next person which work can be preserved, what blocks release and what evidence would change the decision.

After shipping a passing candidate, compare affected query families with their own earlier behavior. Record major documentation launches and instrumentation changes alongside the numbers. Read a small set of failures from outside the test pack, and add a new case when it represents a distinct reader need. Keep some cases untouched as a check against tuning to the test. When an API version retires, revisit the judgment and expected destination instead of letting an obsolete answer remain the standard for success.

A good stopping point is a decision another editor can reproduce. They can open the same pages, see why the urgent query failed, and tell whether the next proposed repair fixed that problem without sacrificing the two gains. That is a more specific claim than saying the site's search is better, and a much easier one to defend.

Limitations

This synthesis offers a release procedure, not an estimate of reader success or a comparison of search providers. A small purpose-built query pack can detect particular regressions but cannot estimate their prevalence without a representative sample. A pooled shortlist may omit useful pages, and an editor’s judgment cannot establish that a reader completed a task. The click study examined general web search in 2005. BRIGHT examines reasoning-intensive retrieval across its own datasets.