Observability and troubleshooting
Use this page to find the first PIG stage that stops making progress, and fix that stage without losing trace data. PIG runs as a flow: hosts upload sessions, the analyzer stores traces, analysis completes, and findings and status synchronize to the Promptless Dashboard. A healthy pod covers only part of that flow.
Check the stages in order. The first stage without recent progress tells you which symptom to follow:
| Stage | Evidence of progress | Investigate when | Start with |
|---|---|---|---|
| Host collection | Recent successful collector status and host check-in. | An active pilot host stops checking in or uploading. | A host can’t enroll or upload |
| Durable storage | Recent agent_traces.last_ingested_at, a completed canonical object, and a readable object at the recorded location. | Storage errors appear, objects remain pending, or active hosts stop producing stored traces. | Traces aren’t stored |
| Analysis | Trace analysis run succeeded log entries, correlated by trace_record_id and analysis_run_id. | Failures repeat or eligible quiet sessions don’t complete. | Analysis doesn’t complete |
| Hosted synchronization | Deployment status and newly projected trace metadata or findings reach Promptless. | Local work completes but hosted state remains stale. | The Dashboard is stale |
| Remediation | An outcome explaining a pull request, external owner, or missing evidence. | A finding has no visible progress after prerequisites are met. | Remediation troubleshooting |
Set alert thresholds around your team’s working hours, expected trace volume, and quiet window. An idle host or a successful analysis with zero findings is normal. Establish a baseline with the pilot-session checks before alerting.
The analyzer writes structured JSON logs to container stdout, including the logs of its analysis processes. For retention or centralized search, forward them with your cluster’s log collector. To read recent logs for the deployment name used in the default Helm guide:
kubectl --namespace pig get podskubectl --namespace pig logs deployment/acme-analyzer --since=1hSymptom: A host can’t enroll or upload
Section titled “Symptom: A host can’t enroll or upload”Cause: The host can’t reach or authenticate to the analyzer
Section titled “Cause: The host can’t reach or authenticate to the analyzer”Enrollment and upload need the host credential, a registered hosted deployment, and a hostname that matches the TLS certificate. The host’s network must also reach the analyzer’s enrollment and upload routes. A passing /healthz check shows only that the health route is reachable.
Solution: Check the host’s credential and network path
Section titled “Solution: Check the host’s credential and network path”-
Check the host credential and the hosted deployment registration.
-
Confirm that the hostname matches the analyzer’s TLS certificate.
-
From the host’s network, confirm that enrollment and upload requests reach the analyzer.
-
Follow host enrollment troubleshooting for host-side checks.
Symptom: A host checks in, but its traces aren’t stored
Section titled “Symptom: A host checks in, but its traces aren’t stored”Cause: The analyzer can’t write to PostgreSQL or object storage
Section titled “Cause: The analyzer can’t write to PostgreSQL or object storage”The analyzer records each session in the agent_traces table and writes its canonical object to your object store. A database outage, or missing object-storage authorization, location, prefix, or encryption-key permission, leaves the object pending or failed.
The analyzer retries a pending or failed canonical write in the background, with a persisted backoff capped at five minutes. A write_failed status that clears on its own needs no action.
Solution: Restore storage access and confirm the backlog drains
Section titled “Solution: Restore storage access and confirm the backlog drains”-
Find the session’s row in
agent_tracesand check itstrace_object_statusandtrace_object_last_error. -
Check PostgreSQL availability.
-
Check object-storage authorization, location, prefix, and encryption-key permissions. Keep the same bucket and database while you fix access.
-
Run
pig-trace-analyzer db-statusto see the number of pending objects and when the oldest was ingested. For how to run it, see the worker maintenance commands. -
If the backlog grows, or stops draining after you restore access, contact Promptless support.
Symptom: The analyzer pod isn’t ready or keeps restarting
Section titled “Symptom: The analyzer pod isn’t ready or keeps restarting”Cause: A database or object-storage check fails
Section titled “Cause: A database or object-storage check fails”The analyzer’s readiness endpoint, /readyz, gates upload and analysis traffic on recent database and object-storage checks. When a check fails, the analyzer becomes unready but stays live. The supervisor and the manual worker chart both set the analyzer’s Kubernetes readiness probe to /readyz. The Service and the collector Ingress then stop sending traffic to the pod until storage recovers. The collector Ingress exposes /healthz for load-balancer health checks; /readyz stays inside the cluster.
Solution: Restore storage access
Section titled “Solution: Restore storage access”-
Run
kubectl --namespace pig get podsand confirm that the analyzer pod is running but not ready. -
Read the analyzer logs for the failing storage check.
-
Restore access as described in A host checks in, but its traces aren’t stored.
Cause: The analyzer’s background tasks stopped
Section titled “Cause: The analyzer’s background tasks stopped”The liveness endpoint, /healthz, fails when the analyzer’s background tasks stop. It also fails when a storage check or canonical-object repair makes no progress for 180 seconds. Kubernetes then restarts the pod.
Solution: Read the logs from before the restart
Section titled “Solution: Read the logs from before the restart”-
Inspect the previous container’s logs with
kubectl --namespace pig logs POD_NAME --previous. -
Send the relevant log lines to Promptless support. Leave out Secret contents and full session transcripts.
Symptom: Traces are stored, but analysis doesn’t complete
Section titled “Symptom: Traces are stored, but analysis doesn’t complete”Cause: Analysis isn’t enabled, or the session isn’t eligible yet
Section titled “Cause: Analysis isn’t enabled, or the session isn’t eligible yet”The analyzer picks up a session only while analysis is enabled and after the session becomes eligible, for example once its quiet window passes. The repository and model settings must also be complete.
Solution: Check the analysis configuration
Section titled “Solution: Check the analysis configuration”-
Confirm that analysis is enabled and that the session is eligible, for example that its quiet window has passed.
-
Confirm that the repository and model settings are complete.
-
Run
pig-trace-analyzer db-statusto see the number of queued analyses and when the oldest was queued.
Cause: The analyzer can’t read the repository or call the model
Section titled “Cause: The analyzer can’t read the repository or call the model”Repository read access and model authorization are separate from object-storage access, and a working object store doesn’t establish either one. The worker retries transient failures during an analysis run. A run that reaches failed stays failed: an unchanged session revision doesn’t retry automatically after you fix the dependency.
Solution: Fix the dependency and request re-analysis
Section titled “Solution: Fix the dependency and request re-analysis”-
Find the run’s
Trace analysis run failedlog entry and read itsfailure_category. -
Test repository read access and model authorization separately.
-
Fix the failing dependency.
-
Contact Promptless for supported re-analysis, with the run’s
analysis_run_id. Keep the stored traces, and don’t run ad hoc backfill Jobs or edit analysis rows.
Symptom: Analysis completes, but the Dashboard is stale
Section titled “Symptom: Analysis completes, but the Dashboard is stale”Cause: The analyzer can’t synchronize with Promptless
Section titled “Cause: The analyzer can’t synchronize with Promptless”Synchronization needs outbound access to Promptless, a valid installation credential, and the registered configuration hash. Finding projection also needs a working hosted repository connection.
Solution: Check the connection to Promptless
Section titled “Solution: Check the connection to Promptless”-
Check outbound access from the analyzer to Promptless.
-
Check the installation credential. If you rotated it, confirm that the Secret contains the replacement and that the analyzer restarted after the change.
-
Check that the configuration hash matches the registered value.
-
Check the hosted repository connection.
Cause: Promptless marked the analyzer stale
Section titled “Cause: Promptless marked the analyzer stale”Promptless marks a healthy analyzer stale when 15 minutes pass without a check-in. This threshold is fixed. An analyzer whose storage checks fail reports itself as stale when it checks in. A stale analyzer fails deployment verification and acceptance.
Solution: Restore check-ins
Section titled “Solution: Restore check-ins”-
Confirm that the analyzer pod is ready. If it isn’t, follow The analyzer pod isn’t ready or keeps restarting.
Cause: The run produced no findings
Section titled “Cause: The run produced no findings”A run that completes with zero findings has nothing new to project, so no issue appears in the Dashboard. Check the run and deployment status before treating a missing issue as a failure.
Symptom: An upgrade is blocked or fails
Section titled “Symptom: An upgrade is blocked or fails”Cause: The supervisor blocked or failed the release transition
Section titled “Cause: The supervisor blocked or failed the release transition”For deployments with automatic updates, the supervisor reports release and migration status on the PIGDeployment. A failed dependency check, permission check, or migration blocks the transition.
Solution: Inspect the transition and fix the dependency
Section titled “Solution: Inspect the transition and fix the dependency”-
Inspect the current and target releases, conditions, and supervisor logs:
Terminal window kubectl describe pigdeployment acme --namespace pigkubectl logs deployment/pig-supervisor --namespace pig-system --since=1hkubectl get jobs --namespace pig -
Save the failed migration Job’s logs before retrying.
-
Resolve the reported dependency through its owner: Terraform for cloud resources, your secret manager for credentials, and PIG for application releases.
-
Read updates and recovery before choosing a rollback.
Cause: The manual chart’s migration hook failed
Section titled “Cause: The manual chart’s migration hook failed”The manual worker chart runs schema migrations in a Helm hook Job. The chart removes a successful hook Job and replaces a failed one on the next attempt.
Solution: Inspect the migration hook and release history
Section titled “Solution: Inspect the migration hook and release history”-
Inspect
job/pig-trace-analyzer-migrateand its logs before retrying. -
Review the release history with
helm history pig-trace-analyzer --namespace pig.
These names assume the release replacement is complete. For an existing installation, find its names with helm list --namespace pig and kubectl get deployments,jobs --namespace pig. Follow the manual release replacement before installing under the new name.
For more information
Section titled “For more information”- Manage updates and recovery covers the supervisor’s pause, blocked-upgrade, and recovery behavior. For Helm-managed releases, coordinate restoration and any compatible application rollback with Promptless.
- Keep PostgreSQL backups and object-storage retention and version recovery under your organization’s recovery policy. Rehearse restoring them as a consistent set: restoring only the database or only the objects can leave records and content out of sync.
- The manual Helm reference lists the worker chart’s optional Datadog tracing and Sentry error-reporting values. These chart values aren’t
PIGDeploymentfields. - When you contact Promptless support, include the PIG release, the time window, and the analysis log identifiers
analysis_run_id,trace_record_id,analyzer_version, andrepository_sha. Each run’s final log entry also includessuccessful_finding_write_count, and a failed run’s entry includesfailure_category.