Doc Detective integration
Overview
Section titled “Overview”Docs describe how your product works. Every setup guide, UI walkthrough, CLI example, and API tutorial contains product assertions: this button exists, this command succeeds, this page appears, this link works.
Promptless uses Doc Detective to make each of those assertions executable. Doc Detective is an open-source implementation of the Docs as Tests framework. That framework tests documentation against the real product experience, beyond proofreading it for style and checking it for broken links.
Use Doc Detective in your docs repository when you want Promptless to keep tests current along with the prose. When a new feature changes the product, Promptless can document the change and add the tests that prove the documented workflow still works. When your repository uses Doc Detective, Promptless:
- Creates Doc Detective tests when new documented workflows, UI paths, commands, or setup procedures are added
- Updates existing Doc Detective tests when product changes make the old workflow stale
- Reuses your existing Doc Detective config and test layout
- Responds to CI failures on its documentation PRs, and updates the suggestion branch when the docs change caused the failure
Executable tests are one of the tools behind Audit and keep your docs healthy. They keep remediated procedures accurate as the product changes, so a workflow you just fixed doesn’t quietly go stale again.
Set up Doc Detective
Section titled “Set up Doc Detective”If your repository already uses Doc Detective, Promptless writes and updates tests without any configuration. Any of these counts: a config file at the repository root, test specs, inline test statements, or a CI job that runs Doc Detective. If your repository doesn’t use Doc Detective yet, add a Doc Detective config file, such as .doc-detective.json, .doc-detective.yaml, or .doc-detective.yml, at the repository root.
Set Doc Detective config path (optional) if your config file isn’t at the repository root or uses another name. Otherwise, leave it blank and Promptless looks for it. You need an admin role in your Promptless organization to edit the Configuration page.
-
On the Doc collections tab of the Configuration page, expand the doc collection.
-
Select More filters (optional).
-
In Doc Detective config path (optional), enter the repo-relative path to your Doc Detective config file, such as
config/doc-detective.json. -
Select Save changes.
In YAML mode, the same setting is doc_detective.config_path on the doc collection. See the Configuration reference for the full doc collection schema.
Default behavior
Section titled “Default behavior”When your repository uses Doc Detective, Promptless handles tests this way by default:
- Test coverage: Promptless writes or updates tests for the workflows a docs change touches and commits them in the same suggestion.
- Validation: Promptless runs your configured syntax check, when you have one, before it opens the suggestion. A syntax check confirms that the test files are valid, not that the documented workflow works.
- Test runs: Your tests run wherever you run Doc Detective, such as in CI on the docs PR. See CI failure handling.
- Results: Promptless records the checks it ran and their results in the suggestion description. Citations on the docs PR link changed passages to the source evidence behind them.
Promptless writes tests against the test accounts and data your tests already use. It doesn’t invent credentials, URLs, or test data to make a test pass.
To change these defaults, add an instruction to PROMPTLESS.md in your Agent Knowledge Base. For example, tell Promptless which kind of tests you prefer. Choose detected test statements, inline test statements (best for most people), a mix of the two, or standalone test files. Standalone files are ideal when editors clobber inline test statements. Promptless loads PROMPTLESS.md into every session.
CI failure handling
Section titled “CI failure handling”Doc Detective is most useful when it runs in CI with the rest of your documentation checks.
When a GitHub check suite fails on an open Promptless docs PR, Promptless:
- Resolves the failing PR back to the Promptless suggestion that created it.
- Fetches the failing check runs and a preview of the logs.
- Starts a follow-on investigation with the PR, failing check names, conclusions, and log preview.
- Determines whether the suggestion caused the failure.
If the suggestion caused the failure, Promptless updates the suggestion branch with the fix. If the failure is pre-existing or unrelated, Promptless keeps the suggestion scoped to its original work. If it finds a separate docs issue outside the suggestion’s scope, it can create separate work for that issue without expanding the original PR.
See also
Section titled “See also”- GitHub repos (docs as code), for every doc collection setting
- Configuration reference, for the
doc_collectionsschema - Vale integration, to enforce prose style in the same suggestions
- Doc Detective documentation, for writing tests and running them in CI