Measure the impact of Promptless
Overview
Section titled “Overview”Start with the Impact page in Settings. It charts the suggestions Promptless created and the suggestions your team merged or closed, week by week. Then add numbers from the Published page, your Git history, and your support and analytics tools, and combine them in a report.
If you maintain an open-source project, use the same report for a foundation, a sponsor, or your community.
Read the weekly backlog chart
Section titled “Read the weekly backlog chart”-
Open Settings, then select Impact under Account.
-
Set Range to 4w, 8w, or 12w for the last 4, 8, or 12 weeks.
-
To report on one doc collection, select it in the Collection filter. The filter appears when your organization has more than one doc collection.
-
Read the chart. Each week shows the suggestions Created, Merged, and Closed, and the Backlog of open suggestions that remains.
Select a priority below the chart to highlight its suggestions. Set View to Stacked to compare the suggestions created each week by priority.
Find more numbers
Section titled “Find more numbers”- Published page. The suggestions merged in the last 30 days, with who merged each one and its doc collection. Select Published in the sidebar.
- Suggestions list. Every suggestion Promptless created, sorted into Ready, PR Open, and Closed, each with an estimated review time. Filter by Collection or Trigger source.
- Triggers page. Every event Promptless processed. The Suggestion Activity filter separates events that produced a suggestion from events that didn’t. The From and To fields set the date range, and the summary line shows the number of matching triggers.
- Git history. Merged docs PRs, with author, reviewers, and timestamps. On GitHub, Promptless opens PRs as
promptless[bot]. On GitLab, find the account name on any merge request Promptless opened.
Choose the metrics to track
Section titled “Choose the metrics to track”Track only the metrics your leadership asks about.
- Backlog trend. On the Impact page, compare the backlog in the first and last week of the period. A flat or falling backlog means your team reviews suggestions as fast as Promptless creates them.
- Doc updates shipped. Count the merged PRs that Promptless authored in the period. For the last 30 days, count the suggestions on the Published page.
- Share of doc changes automated. Divide the merged Promptless PRs by all merged docs PRs in the same repo and period.
- Coverage of product changes. On the Triggers page, set From and To. Set Suggestion Activity to Suggestion Activity and record the count in the summary line. Repeat with No Suggestion Activity. The first count is the events that led to a docs change. The second is the events Promptless checked and skipped.
- Review load. Count the suggestions in Ready, PR Open, and Closed, and add up their estimated review times.
- Time saved. Multiply doc updates shipped by your estimate of the hours a manual update takes. State the estimate in the report.
Count merged PRs from the command line
Section titled “Count merged PRs from the command line”Run these commands in your docs repo. Replace the dates with your period.
Count the merged PRs that Promptless opened, with the GitHub CLI:
gh pr list --author "app/promptless" --state merged \ --search "merged:2026-10-01..2026-10-31" \ --limit 1000 --json number --jq lengthCount all merged PRs in the same period:
gh pr list --state merged \ --search "merged:2026-10-01..2026-10-31" \ --limit 1000 --json number --jq lengthDivide the first count by the second to get the share of doc changes automated.
Without the GitHub CLI, count the commits on your main branch with Git. With squash merges, each commit is one PR:
git log origin/main --author='promptless\[bot\]' \ --since=2026-10-01 --until=2026-11-01 --oneline | wc -lList the pages Promptless changed most often in the period:
git log origin/main --author='promptless\[bot\]' \ --since=2026-10-01 --until=2026-11-01 --name-only --format= \ | sort | uniq -c | sort -rn | head -20Add outcome metrics
Section titled “Add outcome metrics”The metrics above count output. To show the effect on readers and on cost, add metrics from your support, analytics, and docs-platform tools.
| Metric | Where to read it | How to use it |
|---|---|---|
| Support tickets per topic | Your helpdesk, such as Zendesk or Intercom. GitHub Issues or Discussions, if users ask questions there. | Compare ticket volume on a topic before and after a Promptless update to that topic. |
| ”Was this helpful?” ratings and comments | Your docs platform, such as Mintlify, GitBook, or ReadMe, or a feedback widget. | Compare ratings on a page before and after an update. Respondents are self-selected and skew negative, so use the trend, not the score. |
| Site searches with no results | Your site-search analytics, such as Algolia or your docs platform’s search dashboard. | Recurring queries show missing content and the words readers use for it. After an update, check that the no-results rate for the topic falls. |
| Page views and engagement | Your web analytics, such as GA4 or Plausible. | Confirm that an updated page has readers. Read engagement time with views. A high bounce rate is normal on a page that answers a question quickly. |
| Task completion rate | Usability tests or an instrumented funnel. | Measure whether readers can complete the task. This costs the most to collect, so use it for high-stakes topics only. |
| Content freshness, doc coverage, and organic search traffic | Your docs repo and a search console, such as Google Search Console. | Use as supporting context. Report any link to growth or conversion as a correlation. |
Convert outcomes to a dollar value
Section titled “Convert outcomes to a dollar value”If you have the cost figures, calculate:
- Deflection value = tickets deflected × fully loaded cost per ticket. Support operations usually has the cost per ticket.
- Contributor time saved = doc updates shipped × hours saved per update × fully loaded hourly rate.
- ROI % = (attributed value − program and tool cost) ÷ cost × 100.
Without cost figures, report the before-and-after trend.
State each assumption, such as cost per ticket, hours per update, and deflection percentage. Report before-and-after comparisons and ranges, not single figures.
Build the report
Section titled “Build the report”-
Set the period and scope. Choose a month or a quarter, and the doc collections to cover. Use the same period for every number.
-
Collect the output numbers. Read the weekly created, merged, and closed counts from the Impact page. Read merged Promptless PRs from your Git history, suggestion counts and review times from the Suggestions list, and coverage from the Triggers page.
-
Lead with output. Put doc updates shipped and share of doc changes automated first. Add time saved, with your hours-per-update estimate.
-
Add outcomes. Include one or two outcome metrics for the same collections and period. Convert them to a dollar value if you have the cost figures.
-
Cite the source of each number. Name the Impact page, the Suggestions list, the Triggers page, the Published page, or the merged PRs.
See also
Section titled “See also”- Triage the review queue, to find and filter suggestions in the Suggestions list
- Follow what a trigger did, to read trigger outcomes on the Triggers page
- Agent Knowledge Base, for where lasting context about your docs lives
- Run a pilot, for the volume and quality targets to report during an evaluation