Why API docs go stale, even when the build passes

How we researched this

This research synthesis was prepared on October 27, 2026 using 11 primary sources: two historical documentation studies, a 2026 industrial assessment, the OpenAPI specification, official validation and testing documentation, GitHub versioning guidance and a public GitLab maintenance proposal. Research queries were used for discovery; consequential claims were checked against the original material.

We distinguish participants, inspected artifacts, retained artifacts and model-detected issues. We did not pool these populations, run a vendor benchmark, test customer systems or measure a support-cost reduction. The defect-to-check mapping and release workflow are our recommendations based on the evidence, not experimentally measured detection rates. Tool behavior is described as documented on the access date and can change with versions and configuration.

What does a successful documentation build actually prove?

Imagine a customer following a guide to create a project, upload a file and retrieve the result. The reference renders correctly. Every page exists. The request example is valid JSON. The customer still gets a permission error because the guide never explains which account role is required.

That is a hypothetical example, but it captures a useful distinction: the documentation made several promises, and the build tested only some of them. A page can be available while an instruction is wrong. A request can match a schema while the account cannot perform it. A successful response can arrive while the customer still does not know what to do next.

We recommend separating documentation checks into five questions:

1. Can the material be delivered? Pages, assets, navigation and links work.
2. Is the description internally consistent? Referenced operations and declared examples agree with the contract.
3. Does the example run? The shown command or program works in the stated environment.
4. Does the service behave as described? Actual responses and state changes match the documented expectations.
5. Can the intended reader finish the task? Prerequisites, explanations and recovery instructions are sufficient.

These are our working categories, not an industry certification. Their purpose is to stop a narrow check from acquiring a broader meaning in a release report. Instead of saying that the docs passed, say which task, version and conditions were checked. That makes a green result useful to the next maintainer.

What documentation research has actually measured

The evidence is stronger when we preserve the unit being counted. Three studies illustrate different ways to investigate the problem.

The 2015 paper, How API Documentation Fails, is by Gias Uddin and Martin P. Robillard. Its survey findings help explain why polishing layout alone is insufficient: readers also need correct explanations and enough information to act. The first survey received 69 responses from 698 invitees; nonresponse and company-specific sampling limit generalization. These findings do not establish a modern industry-wide failure rate. Study

In Software Documentation Issues Unveiled, Emad Aghajani and coauthors analyzed issues, pull requests, mailing-list threads and Stack Overflow discussions. The distinction between 878 inspected and 824 retained artifacts appears in the study's dataset and results. The retained artifacts are the final analysis set, not independent API failures. A single artifact can discuss multiple issues, so dividing issue types by artifacts would not produce a useful defect percentage. Study

The 2026 assessment by Rayfran Rocha Lima, Davi G. Assunção Pinheiro and Thiago Medeiros de Menezes adds a current industrial example. Its model evaluation used expert annotations on 60 endpoints. The authors describe a feasibility assessment, with results dependent on models, prompts and internal context. This is one case study, not a universal failure rate. Study

Taken together, these sources justify asking several kinds of quality question. They do not justify a headline claiming that a fixed percentage of API documentation is broken. That would require a different sampling design and a shared definition of failure.

A valid OpenAPI description is a useful starting point

OpenAPI provides a structured description of HTTP interfaces, including operations, inputs, responses and security information. It gives tools a common artifact to inspect. The specification expects documentation of successful responses and known errors, while acknowledging that every possible response code may not be known in advance. Specification

The practical benefit is consistency. If the reference page, request example and SDK instructions all depend on a declared field, a shared description gives the team a place to define that field. But a description is still a statement about the service. Validating the statement's structure does not compare it with every deployed implementation.

Consider a hypothetical response described as an object with an identifier. An example containing that identifier may be perfectly valid while leaving out how the identifier is used in the next step. Alternatively, both the schema and the example might agree on a field name that the deployed service no longer returns. Agreement between two artifacts is not agreement with reality.

For that reason, separate two jobs in the release process:

• Check the description for structural errors and broken references.
• Check important behavior against the version of the service the reader will use.

Keep a named owner for the boundary between them. When a live response disagrees with the description, someone must decide whether the implementation, the contract or the guide is wrong. Automatically changing the documentation to match any observed response can preserve an implementation bug as if it were intended behavior.

A generated reference can still be valuable. The mistake is expanding its promise. Use it to keep repeated interface facts aligned, then add the decisions a customer must make: which operation to choose, which permissions to request, how results become available and what failure means.

Validate examples, then verify what they demonstrate

Redocly's official example-validation rule checks media-type examples against their corresponding schemas. Its behavior is configurable, including severity and treatment of additional properties. That is a concrete way to detect an example whose values or structure disagree with the declared contract. It is not a live API execution test. Rule documentation

A public GitLab maintenance proposal from May 2026 provides a useful example. The author described examples with a flat object where a nested user object was required. The merge request proposed updating source examples and regenerating the OpenAPI descriptions. At our check, the merge request was closed without being merged; we use it as a documented proposed correction, not evidence that a particular fix shipped. Its description gives inconsistent warning totals, so we do not repeat a count. Public record

The lesson for a documentation workflow is straightforward: examples are maintained artifacts, even when the reference around them is generated. They deserve a check of their own.

Our recommended review asks three questions:

1. Does the value fit? Types, required fields and nesting agree with the intended contract.
2. Does the request work? The example succeeds with the permissions and environment described to the reader.
3. Does the explanation fit the result? The surrounding text says what happened and how to use the result.

Those questions target different kinds of mistake. A valid object can describe an unrealistic state. A working request can require an administrator when the guide promises ordinary-member access. A correct response can be accompanied by an outdated instruction about where to find it.

Choose examples that expose a real decision rather than filling every field with a placeholder. For instance, distinguish a required value from an optional preference and explain how the reader obtains identifiers. Make examples realistic enough to teach the task, while keeping test credentials and personal information out of the published material.

An executable example needs a meaningful assertion

Python's doctest runs interactive examples and compares observed output with expected output. Its documentation explains strict matching and options that alter comparison or skip examples. A skipped example is not evidence that the instruction worked. Python documentation

Rust's documentation tests illustrate a related distinction. For ordinary Rust documentation tests, compilation and execution without panic do not by themselves check a particular returned value. Assertions can check the expected result; special attributes can change which checks run. Rust also supports hiding setup lines from the rendered example while using them during testing. Rust documentation

Our recommendation is to review the relationship between the tested program and the displayed snippet. Hidden setup can improve readability, but essential customer prerequisites still need to be explained somewhere the reader can find them. A test account that already has every permission can conceal a missing setup step.

For a hypothetical upload example, a useful check would establish more than process completion. It would verify that the response identifies the uploaded item, that the item is retrievable with the documented account and that the visible instructions explain any wait before retrieval. The exact assertions depend on the product's intended behavior.

Avoid maintaining one polished snippet for the page and a separately repaired script for the tests. Prefer a shared source, or a check that confirms the published excerpt still matches the executable example. Otherwise the test can pass while customers copy an older command.

Record what makes the test repeatable: runtime and dependency versions, API version, authentication role, starting data and relevant configuration. If a sample intentionally leaves out cleanup or production error handling, explain that scope. A short example can be useful without presenting itself as a complete application.

Which check targets which failure?

The table below is our proposed mapping. It compares the purpose of checks, not measured vendor performance or a guarantee of detection. A check only catches a failure when the relevant rule, input and assertion are included.

Schemathesis documents checks for status codes, media types, response headers and response-body conformance, among other behaviors. Those checks establish useful properties of the responses actually exercised under the selected configuration. Check reference

Its stateful testing documentation describes connecting operations so values from one response can feed another request. Authentication, seed data, cleanup and discoverable links between operations affect what is exercised. A sequence that never reaches its producer step cannot demonstrate the later steps. Stateful testing guide

For your own suite, make that distinction visible in the result. Report selected operations, executed operations and failed prerequisites separately. An empty selection should not appear equivalent to a completed task. Likewise, a disabled check should be visible as disabled rather than blended into a single reassuring total.

These recommendations apply whether the tool is commercial, open source or an internal script. The important question is the evidence produced by the configured run, not the name in the workflow file.

Treat documentation freshness as versioned evidence

A last-updated date tells readers that a page changed. It does not explain which service version was checked. A wording correction and a successful integration test are different events, and combining them into one freshness label can be misleading.

GitHub's REST guidance provides a concrete example of explicit version selection through the X-GitHub-Api-Version request header. It also tells developers to review breaking changes and test an integration when upgrading. That makes the target of a check explicit instead of leaving it to an unspecified default. GitHub guidance

For documentation you own, we recommend keeping a small verification record behind each critical guide:

This record does not need to clutter the public page. Readers usually need the supported version, prerequisites and a trustworthy example. Maintainers need enough detail to determine whether an old result still applies.

A change in authentication policy should trigger review of setup instructions even when the response schema is unchanged. A renamed dashboard control should trigger review of the screenshots and navigation steps even when API tests still pass. The trigger should follow the claim being made, rather than only the file extension being edited.

When several supported versions exist, test the versions you explicitly claim to support and identify any exceptions. Do not silently redirect every older guide to the newest one if the instructions differ. Readers need to know which contract the page describes.

A practical release workflow for a small team

Start with one important customer journey and make the evidence useful before expanding coverage. The following sequence is our recommendation, not a measured guarantee of fewer incidents.

Choose the journey. Select a task that creates real value, such as obtaining a credential, making a first request and using its result. Include the setup the reader must perform. Pick a failure case that would otherwise leave them stuck.

Inventory the promises. Read the guide sentence by sentence. Mark claims about permissions, required inputs, returned data, timing and cleanup. Separate those from explanations of why a reader would choose the operation. This produces a reviewable scope instead of an undefined request to test the docs.

Attach the smallest useful check. A broken link needs a link check. A malformed example needs validation. A claim about an actual response needs execution. A decision about which operation to choose needs a clear explanation and, ideally, observation of someone attempting the task.

Use a controlled environment. Run write operations against disposable test data in an isolated environment. Make setup and cleanup explicit. A documentation test that accidentally changes customer data is not an acceptable way to increase confidence.

Review from the reader's starting point. Ask a person who did not write the guide to follow it with the stated role and prerequisites. Record where they needed information outside the page. Do not quietly supply missing values and count that as an independent success.

Resolve failures at the right source. If an example is generated, repair the source of that example. If a schema is wrong, agree on the intended service behavior before adjusting it. If the page is missing a prerequisite, add the prerequisite rather than broadening the test account's permissions.

Publish with a bounded statement. Say which journeys and versions were checked. Keep unresolved exceptions visible to the responsible team. Expand the suite when a new failure reveals a missing class of test.

This approach is intended to give a small team a manageable unit of work and leave a record that a future maintainer can inspect: what was expected, what ran and why the guide was considered ready.

Measure blocked tasks as well as passing tests

A useful documentation dashboard should distinguish delivery, correctness and usability. Combining them into one quality score makes it harder to decide what to repair.

These are proposed measures, not data collected by Docsio Research. Define tasks, participant selection and privacy requirements before collecting observations. Our suggested measures are deliberately local to the product:

• Critical-journey coverage: selected customer tasks with a current verification record, divided by the tasks the team has explicitly prioritized.
• Example execution: executable examples attempted, passed, failed and skipped, with reasons for exclusions.
• Missing prerequisites: steps where an observed reader needed an undocumented permission, identifier, setting or dependency.
• Repair time: elapsed time between a confirmed documentation problem and a verified correction.
• Repeat confusion: support questions that continue after a particular guide was revised, examined for the underlying task rather than only matching keywords.

Define the denominator before comparing releases. Adding harder tasks can lower a pass rate even when the documentation improves. Removing failing tasks can raise it without fixing anything. Keep the task inventory and exclusions alongside the headline number.

Support volume alone is also ambiguous. It may rise because more people are using the product or because a newly released feature is attracting questions. If you evaluate a documentation change, compare relevant user cohorts and task attempts where you have appropriate data. Avoid attributing every movement to the article you just updated.

For teams using a documentation platform, the editorial and engineering responsibilities still need to meet. Use the platform to organize, explain and publish the material; connect claims about product behavior to evidence from the product's own tests and reviews. This report does not claim that Docsio automatically performs the verification methods described here.

The aim is a documentation site where a reader can finish a task and a maintainer can explain why the instructions should work. More pages and a successful build are useful inputs. The outcome worth maintaining is a clear, repeatable path through the product.

Limitations

This is a selective synthesis, not a systematic review or an audit of current public APIs. Historical studies describe their original populations; the 2026 assessment concerns one internal ecosystem and includes model-generated findings. We inspected a public maintenance proposal without reproducing its patch or claiming it was deployed. Tool documentation establishes documented capabilities, not independently measured effectiveness. Our workflow, check mapping and suggested metrics are recommendations that teams should adapt and evaluate. No customer systems were tested, and no claim of traffic, revenue, support savings or universal documentation correctness follows from this report.