Understand findings
A finding describes an instruction problem supported by evidence from real agent sessions. It helps your team decide what to change, why the change matters, and who should own it.
For example, Acme’s documentation skill might tell an agent to verify every example but omit how to choose the correct product version. A finding can connect repeated incorrect examples to that missing guidance. A failed API request caused by an expired credential may instead belong to the team that manages the credential.
Findings appear after an operator has verified completed analysis. To work with one, you need access to your connected Instruction Hub. You also need the finding’s GitHub issue or its page in the Promptless Dashboard.
Use cases
Section titled “Use cases”- A finding is high severity but low confidence, and you must decide whether to act now. Read the evidence before choosing a fix explains what each field measures.
- A finding waits for a repository before Promptless opens its GitHub issue. Follow the finding into your repository explains how Promptless picks a destination and when an administrator chooses one.
- A finding points at a failure the hub doesn’t own, such as an expired credential. Decide the next action helps you route it to the right owner.
- An analysis run completes with zero findings. That is a valid result, and Troubleshooting lists what to confirm.
Read the evidence before choosing a fix
Section titled “Read the evidence before choosing a fix”Start with the summary, impact, and cited occurrences. Check that the described behavior happened and that the instruction failure explains it. A finding carries these fields:
| Field | What it tells you | How to use it |
|---|---|---|
Severity: low, medium, high, critical | How serious the observed problem is. | Prioritize the effect on users and your systems. |
Confidence: low or high | How strongly the evidence supports the finding. | Separate a suspected problem from one supported well enough for remediation. |
| Instruction failure | The gap, conflict, or misleading guidance behind the behavior. | Locate the instruction that needs attention, or question whether the hub owns the cause. |
| Evidence | Cited occurrences and a summary of what happened. | Compare the claim with the session details. |
| Notes | Additional context, uncertainty, or an external owner. | Check whether action belongs outside the hub. |
Severity and confidence answer different questions. A high-severity finding can still have low confidence and need more evidence. Later analysis can add occurrences to an existing finding and promote its confidence when the evidence supports that change.
The analyzer cites evidence from a readable session digest. A digest_range identifies a location in that digest. It is unrelated to the cryptographic hashes that verify stored traces.
Follow the finding into your repository
Section titled “Follow the finding into your repository”PIG analyzes every instruction repository your organization selects in PIG Settings. Each finding records the instruction source it came from: the repository and the exact commit the analyzer read. The finding lists these sources under Instruction sources, each linked to that commit.
The analyzer sends finding updates to Promptless. Promptless creates or updates a GitHub issue for the finding in its destination repository. The issue carries the finding’s title, impact, instruction failure, severity, confidence, and evidence. A destination repository must have Create Repo Issues enabled in PIG Settings.
Promptless creates the issue automatically when the finding’s recorded sources resolve to one selected repository that has Create Repo Issues enabled. A finding needs a destination first in either of these cases:
- Its sources span more than one selected repository.
- It records no source, and your organization has more than one repository selected.
Removing an instruction repository in PIG Settings stops future analysis and issue automation for that source. Findings already recorded against it keep their instruction sources, issues, and pull requests.
As the analyzer processes more sessions, recurring evidence can update an existing finding and its issue. Use the finding identity and the linked issue to follow that history. Creating the issue is a separate step from analysis, so a completed run alone doesn’t prove that GitHub access works.
Finding text and evidence summaries can include session details. Make sure the destination repository’s access matches your intended audience, as described in Trust and data model.
Choose a destination repository
Section titled “Choose a destination repository”An organization administrator sets the destination once per finding, while the finding is open:
-
On the Findings page of the Promptless Dashboard, open the finding.
-
Select Instruction sources and fix destination.
-
Select a repository from the list. The list shows only repositories that have Create Repo Issues enabled.
-
Select Use repository.
Promptless then creates the issue, and any proposed fix, in that repository.
Decide the next action
Section titled “Decide the next action”-
Confirm the observed problem. Read the impact and evidence, including enough surrounding context to avoid correcting intentional behavior.
-
Identify the owner. Shared hub instructions, a repository-local instruction, a tool bug, and a missing permission have different owners.
-
Check for an existing response. Read the finding notes, linked pull request, and current hub instructions before proposing a duplicate fix.
-
Review the remediation outcome. High-confidence findings can enter the remediation workflow. That workflow may propose a hub change, ask for a policy decision, or identify an external action.
The expected outcome is a clear owner and a justified next action. Not every finding needs a new instruction or a pull request.
Troubleshooting
Section titled “Troubleshooting”| Situation | What to check |
|---|---|
| The run succeeded, but no finding appeared | Zero findings is valid. Confirm the run belongs to the session you intended to analyze. |
| A local finding has no GitHub issue | Ask the operator to check hosted synchronization and the repository connection. |
| The finding asks you to select a repository for its GitHub issue | Its recorded sources map to more than one selected repository, or to none. An organization administrator opens the finding and, under Instruction sources, selects a repository that has Create Repo Issues enabled. |
| Evidence seems inconsistent with the current hub | Compare the analyzed repository revision with the current instructions and the session’s installed plugin version. |
| No remediation pull request opened | Check confidence, ownership notes, pending decisions, repository access, and the remediation outcome. |
| Similar issues appear to overlap | Compare their failure mechanism and evidence before treating them as duplicates; different symptoms can have one cause, and similar symptoms can have different causes. |
Related resources
Section titled “Related resources”- Remediate findings, to review and ship a proposed fix
- Trust and data model, for who can see finding text and evidence
- Verify your deployment, to confirm that analysis completed for a session