Skip to content

For the complete documentation index, see llms.txt.

Test whether your docs answer readers' questions

Readers and AI assistants come to your docs with questions. Docs evals test whether your docs answer them. For each doc collection, you write the questions readers ask and the facts a correct answer has to state. A model you pick answers each question from your docs, and a judge model grades the answer against those facts.

The results show which questions your docs support, which facts each answer missed and why, and which pages the model read. When a question fails, Promptless can open a docs task to fix the gap. A trend table shows how each question scores across models, modes, and runs, so you can see whether a docs change fixed an answer.

Docs evals live on the Docs evals page in the Content Strategy area of the dashboard. Select the collection to work on from Doc collection at the top of the page.

  • An admin role in your Promptless organization to add questions and start runs. Other members can view questions, runs, and results, and can export questions.
  • A verified model provider key. Runs use your organization’s own keys, and only models whose provider has passed a test call appear in the run options. See Add your own model provider keys.

Write each question the way a reader would ask it. Then list what a correct answer has to contain.

  1. On the Docs evals page, select Add question.

  2. Enter a Display name and a Slug of lowercase letters and digits joined by single hyphens, up to 64 characters. Trends and the YAML export name the question by its slug.

  3. Enter the Question.

  4. Under Expected facts, add each fact a correct answer states. The judge grades the answer against every fact. A question needs at least one.

  5. Optional. Under Expected pages, add part of the URL or path of each page the model should read, such as configuration-reference.

  6. Select Save question.

Runs ask only questions that are Enabled. Clear Enabled on a question to keep it out of runs without deleting it.

A run asks every enabled question once with one model in one mode, then grades each answer.

  1. Under Start a run, select the Model that answers the questions. Each option shows its price per million input and output tokens.

  2. Under Mode, select how the model reads your docs:

    • HTTP reads your published docs site, the way a reader or an AI assistant does. The collection needs a docs_root_url in its configuration.
    • Raw docs reads the docs repository with a read-only shell, without the published site.
  3. Optional. Under Judge model, select a different model to grade the answers. By default, the run’s own model grades them.

  4. Optional. Clear Open a docs task for each failed question if you only want the results.

  5. Read the cost summary, then select Start run.

Each answer is capped at $2.00 and each grade at $0.50, and your provider keys pay for both. The run appears under Runs as Queued, then Running. The page refreshes the run while it works. To stop an unfinished run, select Cancel on its row.

A collection has one unfinished run at a time. To start another, wait for the current run to end or cancel it. A run answers five questions at a time and must finish within six hours, so it can ask up to 45 enabled questions. With more enabled, the run doesn’t start. Clear Enabled on some questions and run them separately.

Select Results on a run’s row to see one row per question. The results header names the model that judged the answers.

An answer passes when it states every expected fact and the model read every expected page. A page counts only when the model opened it during the run. A URL the answer cites without reading it doesn’t count.

VerdictMeaning
PassThe answer states every expected fact, and the model read every expected page.
FailThe answer missed at least one expected fact or page.
ErrorThe question couldn’t be answered or graded.
StoppedThe run ended before this question finished.

Each result also shows:

ColumnWhat it shows
Unmet expectationsEach missed fact with the judge’s reason, then each expected page the model didn’t read.
Pages visitedThe pages or files the model read.
CostWhat answering and grading the question cost.
Docs taskA link to the task Promptless opened to fix the gap.

When a failed question’s task from an earlier run is still open, the run reuses that task. The result then shows Still open from an earlier run.

The Trends table has one row per question and one column per run, newest first, grouped by model and mode. Each mark shows a run’s verdict for the question. Hover a mark to see what the question cost in that run. Trends appear once a run finishes.

Compare columns to see whether a docs change fixed a failing question, or whether a cheaper model or the other mode answers as well.

Select Export YAML to download the collection’s questions. Each question becomes one case with these keys:

KeyValue
idThe question’s slug
display_nameThe question’s name
enabledWhether runs ask it
questionThe question text
expectThe expected facts
expect_urlsThe expected pages