The task-completion gap in AI-written docs

By Manny Silva, Head of AI Docs Practice

AI agents can write polished documentation, but they often miss what readers need to complete a task. For example, an Azure Service Operator code change added a helper function. The reader’s task, however, was a four-step workflow for having the operator authenticate as an identity it created, and that workflow lived in a linked issue. Six of the seven agents in our paper didn’t document that workflow end-to-end.

That example comes from DoGBench (Documentation Generation Benchmark). DoGBench asks whether agents can produce user-facing documentation that an experienced technical writer would accept in review. If you decide whether an agent’s doc updates ship, task-completion is one of the kinds of gaps to review for.

Polished writing can still leave readers unable to finish a task

The most common failure mode across 1,267 agent sessions was a task-completion gap. In 45.5% of sessions, the agent’s doc update omitted a decision point, procedure, verification step, or recovery path the reader needed. Fabricated content, such as invented classes, flags, or endpoints, showed up in only 6.1% of sessions.

Technical inaccuracies appeared in 36.6% of sessions and misstate a real product’s scope, default, lifecycle, or compatibility, so they read like correct documentation. They’re also not strictly hallucinations in the general sense because the content is grounded in real functionality, but the behavior is misstated.

Our review of sessions also labeled other failure modes that leave readers without critical parts of their tasks:

Missing prerequisites, such as permissions, credentials, or versions, in 21.0% of sessions.
Missing audience and purpose framing, which says who the content is for and when to use it, in 19.9%. For example, a doc update that GPT-5.5 with Codex wrote for Strawberry GraphQL correctly documented how to select an older Apollo Federation version. It didn’t say why a reader would want to, which is to upgrade Strawberry while staying compatible with an older Apollo Router or Gateway. It documented the setting and omitted the decision the setting exists to support.
Information architecture or findability failures, where content sits outside the reader’s likely path, in 20.7%.
Cross-page content inconsistency, where one page changed and another page on the same behavior went stale, in 13.3%.

One doc update can carry several failure modes, so these rates overlap.

All seven agents left a critical defect on one Azure item

Every one of the seven agents in the paper left at least one critical defect on the Azure Service Operator (ASO) item. ASO lets teams deploy and manage Azure resources with Kubernetes tooling. An item is one real docs task. You can inspect this item and every agent’s doc update on dogbench.ai, where doc updates are called patches. A rubric scores each doc update, and a critical defect is a failure the rubric marks as most severe.

The trigger was a code pull request that added a helper function for parsing Azure resource IDs. Its linked GitHub issue described the workflow readers wanted:

Create a managed identity (UserAssignedIdentity) with ASO.
Export the identity’s details to a Kubernetes Secret.
Point a resource’s serviceoperator.azure.com/credential-from annotation at that Secret.
Let ASO authenticate as the identity it created.

Six of the seven agents didn’t document that workflow end-to-end on the pages readers follow. Most documented only the new helper function or a snippet for exporting values to the Secret. GPT-5.5 with Codex said the Secret “can be used” with the annotation and stopped there.

All seven agents failed to explain when the workflow applies to the reader. All seven also failed to correctly document the prerequisites, which included the management credential’s permissions and a workload-identity trust relationship.

The most serious gap was a limitation no agent mentioned. ASO writes the exported Secret only in the managed resource’s namespace. A reader whose workload runs in another namespace has to copy or mirror the Secret, and without that step the workflow fails.

Kimi K2.7 Code with OpenCode came closest. It documented the workflow end-to-end with correct examples, but its example put both resources in one namespace and never mentioned the limitation. A reader with a different layout would follow every step and still fail. Fluent prose makes gaps like this easy to miss, because the doc update reads well and most of it is true.

Agents stop investigating before they know enough

Behind these doc updates, agents often ended their investigations too early. A trajectory is the record of an agent’s work on an item, and we inspected the trajectory behind each of the 1,267 sessions. Three root causes were most common:

The agent explained an interface without checking how readers use it, in 36.0%, or 456 sessions.
The agent missed decisive evidence and filled the gap with a plausible assumption, in 33.1%, or 420 sessions.
The agent stopped at the first plausible page and left other affected pages stale, in 30.1%, or 382 sessions.

Agents treat a change as one piece of information to convey, rather than one step in a journey that needs prerequisites, decisions, verification and recovery. The Azure trigger shows this pattern. The code change added a helper function, and the reader’s task lived in the linked issue.

The common thread is premature closure. Agents switch from investigating to drafting too soon, because they have no reliable rule for when they know enough.

It isn’t about effort. We found no clear relationship between root cause and trajectory length, whether measured in turns or tokens. Effort also didn’t rise with item difficulty. An agent can stop investigating early and then spend many tokens elaborating an incomplete account.

Some of these failures may instead reflect missing context about how readers use the software. Documentation owners bring context that a code change alone may not reveal.

Walk through the reader’s whole task

Agents often don’t understand that a change is part of a larger workflow. Before you approve a doc update, follow the reader from the reason they arrived to the point where they recover from a failure. Ask these questions in order:

Does this change need user-facing documentation at all?
Who is this for, and when would they use it?
What has to be true first, such as permissions, credentials, or versions?
Can the reader get from the first step to the last?
How does the reader know it worked?
What does the reader do when it doesn’t work?
Does each claim hold true everywhere the text implies, including defaults, versions, and namespaces?
Can readers reach this page, and does another page about the same behavior need to change?

If the procedure is runnable, run it yourself or test it with Doc Detective or Playwright. A doc update that passes every question can still have a wrong command, and running it catches that.

Apply the questions to the Azure item, and the gaps show up early. Question 3 catches the missing permissions, and question 7 catches the namespace limitation that every doc update from the seven agents left out.

The readout gives documentation owners these findings in 11 slides

Open the 11-slide readout for documentation owners to share these failure modes and review habits with your team or leadership. The full failure-mode table and trajectory analysis are in the paper’s failure analysis. The paper, data, and example items are at dogbench.ai.

About the author

Manny Silva, Head of AI Docs Practice

Manny Silva is Head of AI Docs Practice at Promptless, where he helps teams optimize their documentation workflows. He has built documentation for Apple, Google, and startups of various sizes. He codified the Docs as Tests strategy and created Doc Detective, an open-source tool for testing documentation. He wrote Docs as Tests: A Strategy for Resilient Technical Documentation and its follow-up, Docs as Tests & AI: A Strategy for Self-Healing Technical Documentation. He likes diving into the deep end as the zeroth user.

← Back to Blog