Troubleshooting
Start with bun run deploy ps, Mainframe’s /api/health, and the Deployment dashboard. Then read the relevant service logs with bun run deploy logs mainframe world-sim or Grafana’s log panel.
| Symptom | What to check |
|---|---|
| Login works on one origin but not the other | MAINFRAME_BASE_URL, trusted origins, HTTPS, and browser cookie behavior |
| Terminal shows no link | Its saved Mainframe URL, DNS/TLS, /ws, and Core health |
| A module starts but never connects | Enabled module identity, stable API key, provider URL, and protocol-error metrics |
| No virtual aircraft appear | World profile health, seed preview, and persisted world identities |
| Auto-operator is connected but does nothing | Serial/provider selection, fresh state, battery thresholds, existing assignments/proposals, and mission coverage |
| Video never becomes live | Publisher connectivity to RTMP, signed assignments, SRS hooks, and SRS stream traffic |
| WebRTC fails after signaling succeeds | Reachability of UDP 8000 and the advertised PUBLIC_IP |
| Live video works but replay is missing | Closed DVR files, indexer failures, S3 access, and recent media manifests |
| Database queue grows | Postgres health, transaction failures, disk latency/capacity, and incoming rate |
| All historical data seems missing after a change | Compose project name and selected environment/volumes |
| Optional module shows as down | Selected profiles and regenerated Prometheus targets |
An S3 outage leaves failed recordings locally for retry. Do not delete the DVR volume to make the error disappear. Resolve the storage problem and watch failed segments drain after the next recovery scan.
An initial connection refusal can be startup ordering. Wait for the health checks and provider reconnect before diagnosing a regression. If a dependency remains unhealthy, use its specific metrics and logs rather than repeatedly rebuilding all services.
For an incident involving hardware, preserve the module’s capture files and the relevant timestamps. Avoid broad resets: they can remove the history needed to understand the failure.
Inspect and retry DVR jobs
Section titled “Inspect and retry DVR jobs”Closed recordings keep a private .flv.index.json sidecar with the processing stage, attempts, next retry, bounded error code and output checksum. Storage failures retry with exponential backoff and jitter (up to one hour). Verified MP4 output survives retries. A lost upload acknowledgment is reconciled by streaming SHA-256 verification before the manifest is published; conflicting bytes preserve both objects for inspection.
Three identical decoder failures on an unchanged, closed file quarantine it by default (MEDIA_INDEXER_QUARANTINE_AFTER, 1–10). Quarantine preserves the raw FLV and stops automatic decoder retries. An old modification time alone does not prove a file is closed: the indexer requires the SRS close callback or evidence of segment rotation.
Inside the indexer container, from /app/apps/mainframe, run:
bun run media:jobs listbun run media:jobs inspect --file live/ENTITY/main/SEGMENT.flvUse paths relative to MEDIA_RECORDINGS_DIR. Stop the indexer before changing job state, then run the same command in a one-off indexer container sharing its recordings volume:
bun run media:jobs acknowledge --file live/ENTITY/main/SEGMENT.flvbun run media:jobs retry --file live/ENTITY/main/SEGMENT.flvAcknowledgment marks a known quarantine for monitoring; it does not erase media or make the recording playable. Retry clears backoff and decoder failure counts; restart the indexer to process it. There is no automatic permanent disposal. Keep the raw file until an operator has explicitly decided it is no longer needed.