Skip to content

For the complete documentation index, see llms.txt.

Observability and troubleshooting

Use this page to find the first PIG stage that stops making progress, and fix that stage without losing trace data. PIG runs as a flow: hosts upload sessions, the analyzer stores traces, analysis completes, and findings and status synchronize to the Promptless Dashboard. A healthy pod covers only part of that flow.

Check the stages in order. The first stage without recent progress tells you which symptom to follow:

StageEvidence of progressInvestigate whenStart with
Host collectionRecent successful collector status and host check-in.An active pilot host stops checking in or uploading.A host can’t enroll or upload
Durable storageRecent agent_traces.last_ingested_at, a completed canonical object, and a readable object at the recorded location.Storage errors appear, objects remain pending, or active hosts stop producing stored traces.Traces aren’t stored
AnalysisTrace analysis run succeeded log entries, correlated by trace_record_id and analysis_run_id.Failures repeat or eligible quiet sessions don’t complete.Analysis doesn’t complete
Hosted synchronizationDeployment status and newly projected trace metadata or findings reach Promptless.Local work completes but hosted state remains stale.The Dashboard is stale
RemediationAn outcome explaining a pull request, external owner, or missing evidence.A finding has no visible progress after prerequisites are met.Remediation troubleshooting

Set alert thresholds around your team’s working hours, expected trace volume, and quiet window. An idle host or a successful analysis with zero findings is normal. Establish a baseline with the pilot-session checks before alerting.

The analyzer writes structured JSON logs to container stdout, including the logs of its analysis processes. For retention or centralized search, forward them with your cluster’s log collector. To read recent logs for the deployment name used in the default Helm guide:

Terminal window
kubectl --namespace pig get pods
kubectl --namespace pig logs deployment/acme-analyzer --since=1h

Cause: The host can’t reach or authenticate to the analyzer

Section titled “Cause: The host can’t reach or authenticate to the analyzer”

Enrollment and upload need the host credential, a registered hosted deployment, and a hostname that matches the TLS certificate. The host’s network must also reach the analyzer’s enrollment and upload routes. A passing /healthz check shows only that the health route is reachable.

Solution: Check the host’s credential and network path

Section titled “Solution: Check the host’s credential and network path”
  1. Check the host credential and the hosted deployment registration.

  2. Confirm that the hostname matches the analyzer’s TLS certificate.

  3. From the host’s network, confirm that enrollment and upload requests reach the analyzer.

  4. Follow host enrollment troubleshooting for host-side checks.

Symptom: A host checks in, but its traces aren’t stored

Section titled “Symptom: A host checks in, but its traces aren’t stored”

Cause: The analyzer can’t write to PostgreSQL or object storage

Section titled “Cause: The analyzer can’t write to PostgreSQL or object storage”

The analyzer records each session in the agent_traces table and writes its canonical object to your object store. A database outage, or missing object-storage authorization, location, prefix, or encryption-key permission, leaves the object pending or failed.

The analyzer retries a pending or failed canonical write in the background, with a persisted backoff capped at five minutes. A write_failed status that clears on its own needs no action.

Solution: Restore storage access and confirm the backlog drains

Section titled “Solution: Restore storage access and confirm the backlog drains”
  1. Find the session’s row in agent_traces and check its trace_object_status and trace_object_last_error.

  2. Check PostgreSQL availability.

  3. Check object-storage authorization, location, prefix, and encryption-key permissions. Keep the same bucket and database while you fix access.

  4. Run pig-trace-analyzer db-status to see the number of pending objects and when the oldest was ingested. For how to run it, see the worker maintenance commands.

  5. If the backlog grows, or stops draining after you restore access, contact Promptless support.

Symptom: The analyzer pod isn’t ready or keeps restarting

Section titled “Symptom: The analyzer pod isn’t ready or keeps restarting”

Cause: A database or object-storage check fails

Section titled “Cause: A database or object-storage check fails”

The analyzer’s readiness endpoint, /readyz, gates upload and analysis traffic on recent database and object-storage checks. When a check fails, the analyzer becomes unready but stays live. The supervisor and the manual worker chart both set the analyzer’s Kubernetes readiness probe to /readyz. The Service and the collector Ingress then stop sending traffic to the pod until storage recovers. The collector Ingress exposes /healthz for load-balancer health checks; /readyz stays inside the cluster.

  1. Run kubectl --namespace pig get pods and confirm that the analyzer pod is running but not ready.

  2. Read the analyzer logs for the failing storage check.

  3. Restore access as described in A host checks in, but its traces aren’t stored.

Cause: The analyzer’s background tasks stopped

Section titled “Cause: The analyzer’s background tasks stopped”

The liveness endpoint, /healthz, fails when the analyzer’s background tasks stop. It also fails when a storage check or canonical-object repair makes no progress for 180 seconds. Kubernetes then restarts the pod.

Solution: Read the logs from before the restart

Section titled “Solution: Read the logs from before the restart”
  1. Inspect the previous container’s logs with kubectl --namespace pig logs POD_NAME --previous.

  2. Send the relevant log lines to Promptless support. Leave out Secret contents and full session transcripts.

Symptom: Traces are stored, but analysis doesn’t complete

Section titled “Symptom: Traces are stored, but analysis doesn’t complete”

Cause: Analysis isn’t enabled, or the session isn’t eligible yet

Section titled “Cause: Analysis isn’t enabled, or the session isn’t eligible yet”

The analyzer picks up a session only while analysis is enabled and after the session becomes eligible, for example once its quiet window passes. The repository and model settings must also be complete.

Solution: Check the analysis configuration

Section titled “Solution: Check the analysis configuration”
  1. Confirm that analysis is enabled and that the session is eligible, for example that its quiet window has passed.

  2. Confirm that the repository and model settings are complete.

  3. Run pig-trace-analyzer db-status to see the number of queued analyses and when the oldest was queued.

Cause: The analyzer can’t read the repository or call the model

Section titled “Cause: The analyzer can’t read the repository or call the model”

Repository read access and model authorization are separate from object-storage access, and a working object store doesn’t establish either one. The worker retries transient failures during an analysis run. A run that reaches failed stays failed: an unchanged session revision doesn’t retry automatically after you fix the dependency.

Solution: Fix the dependency and request re-analysis

Section titled “Solution: Fix the dependency and request re-analysis”
  1. Find the run’s Trace analysis run failed log entry and read its failure_category.

  2. Test repository read access and model authorization separately.

  3. Fix the failing dependency.

  4. Contact Promptless for supported re-analysis, with the run’s analysis_run_id. Keep the stored traces, and don’t run ad hoc backfill Jobs or edit analysis rows.

Symptom: Analysis completes, but the Dashboard is stale

Section titled “Symptom: Analysis completes, but the Dashboard is stale”

Cause: The analyzer can’t synchronize with Promptless

Section titled “Cause: The analyzer can’t synchronize with Promptless”

Synchronization needs outbound access to Promptless, a valid installation credential, and the registered configuration hash. Finding projection also needs a working hosted repository connection.

Solution: Check the connection to Promptless

Section titled “Solution: Check the connection to Promptless”
  1. Check outbound access from the analyzer to Promptless.

  2. Check the installation credential. If you rotated it, confirm that the Secret contains the replacement and that the analyzer restarted after the change.

  3. Check that the configuration hash matches the registered value.

  4. Check the hosted repository connection.

Cause: Promptless marked the analyzer stale

Section titled “Cause: Promptless marked the analyzer stale”

Promptless marks a healthy analyzer stale when 15 minutes pass without a check-in. This threshold is fixed. An analyzer whose storage checks fail reports itself as stale when it checks in. A stale analyzer fails deployment verification and acceptance.

  1. Confirm that the analyzer pod is ready. If it isn’t, follow The analyzer pod isn’t ready or keeps restarting.

  2. Follow Check the connection to Promptless.

A run that completes with zero findings has nothing new to project, so no issue appears in the Dashboard. Check the run and deployment status before treating a missing issue as a failure.

Cause: The supervisor blocked or failed the release transition

Section titled “Cause: The supervisor blocked or failed the release transition”

For deployments with automatic updates, the supervisor reports release and migration status on the PIGDeployment. A failed dependency check, permission check, or migration blocks the transition.

Solution: Inspect the transition and fix the dependency

Section titled “Solution: Inspect the transition and fix the dependency”
  1. Inspect the current and target releases, conditions, and supervisor logs:

    Terminal window
    kubectl describe pigdeployment acme --namespace pig
    kubectl logs deployment/pig-supervisor --namespace pig-system --since=1h
    kubectl get jobs --namespace pig
  2. Save the failed migration Job’s logs before retrying.

  3. Resolve the reported dependency through its owner: Terraform for cloud resources, your secret manager for credentials, and PIG for application releases.

  4. Read updates and recovery before choosing a rollback.

Cause: The manual chart’s migration hook failed

Section titled “Cause: The manual chart’s migration hook failed”

The manual worker chart runs schema migrations in a Helm hook Job. The chart removes a successful hook Job and replaces a failed one on the next attempt.

Solution: Inspect the migration hook and release history

Section titled “Solution: Inspect the migration hook and release history”
  1. Inspect job/pig-trace-analyzer-migrate and its logs before retrying.

  2. Review the release history with helm history pig-trace-analyzer --namespace pig.

These names assume the release replacement is complete. For an existing installation, find its names with helm list --namespace pig and kubectl get deployments,jobs --namespace pig. Follow the manual release replacement before installing under the new name.

  • Manage updates and recovery covers the supervisor’s pause, blocked-upgrade, and recovery behavior. For Helm-managed releases, coordinate restoration and any compatible application rollback with Promptless.
  • Keep PostgreSQL backups and object-storage retention and version recovery under your organization’s recovery policy. Rehearse restoring them as a consistent set: restoring only the database or only the objects can leave records and content out of sync.
  • The manual Helm reference lists the worker chart’s optional Datadog tracing and Sentry error-reporting values. These chart values aren’t PIGDeployment fields.
  • When you contact Promptless support, include the PIG release, the time window, and the analysis log identifiers analysis_run_id, trace_record_id, analyzer_version, and repository_sha. Each run’s final log entry also includes successful_finding_write_count, and a failed run’s entry includes failure_category.