Troubleshooting

Start with bun run deploy ps, Mainframe’s /api/health, and the Deployment dashboard. Then read the relevant service logs with bun run deploy logs mainframe world-sim or Grafana’s log panel.

Symptom	What to check
Login works on one origin but not the other	MAINFRAME_BASE_URL, trusted origins, HTTPS, and browser cookie behavior
Terminal shows no link	Its saved Mainframe URL, DNS/TLS, /ws, and Core health
A module starts but never connects	Enabled module identity, stable API key, provider URL, and protocol-error metrics
No virtual aircraft appear	World profile health, seed preview, and persisted world identities
Auto-operator is connected but does nothing	Serial/provider selection, fresh state, battery thresholds, existing assignments, and mission coverage
Video never becomes live	Publisher connectivity to RTMP, signed assignments, SRS hooks, and SRS stream traffic
WebRTC fails after signaling succeeds	Reachability of UDP/TCP 8000 and the advertised PUBLIC_IP
Live video works but replay is missing	Closed DVR files, indexer failures, S3 access, and recent media manifests
Database queue grows	Postgres health, transaction failures, disk latency/capacity, and incoming rate
All historical data seems missing after a change	Compose project name and selected environment/volumes
Optional module shows as down	Selected profiles and regenerated Prometheus targets
Voice panel reads Voice unavailable	The voice profile is selected (the launcher sets LIVEKIT_URL only then) and /api/health lists livekit
Voice joins but nobody hears anybody, or audio is one-way	7882/udp and 7881/tcp open to PUBLIC_IP on the host firewall; PUBLIC_IP is the address the browsers can reach; LIVEKIT_STUN_SERVERS='[]' on a LAN box without internet; chrome://webrtc-internals shows the selected candidate pair
Voice joins listen-only although the role talks	The page is not a secure context (plain http:// by IP has no microphone; serve over HTTPS), or the browser denied the microphone (Microphone denied · Retry)
Audio blocked · Enable	The browser’s autoplay policy: click Enable once per page
A removed user is still in a room	Mainframe reaches http://livekit:7880 (the Twirp API) on the Compose network; voice: cannot remove in its log

An S3 outage leaves failed recordings locally for retry. Do not delete the DVR volume to make the error disappear. Resolve the storage problem and watch failed segments drain after the next recovery scan.

An initial connection refusal can be startup ordering. Wait for the health checks and provider reconnect before diagnosing a regression. If a dependency remains unhealthy, use its specific metrics and logs rather than repeatedly rebuilding all services.

For an incident involving hardware, preserve the module’s capture files and the relevant timestamps. Avoid broad resets: they can remove the history needed to understand the failure.

Inspect and retry DVR jobs

The SRS worker keeps durable local upload and archive job journals beside original FLV chunks. Check /health and /metrics on the private worker port, controller status, and the Recording reliability dashboard. srs_preservation_pending_bytes, srs_preservation_oldest_seconds, srs_archive_pending and failure counts distinguish upload and packaging problems. A stale controller or failed stream inventory is an explicit health failure.

An original closes only after the controller verifies the stock recorder’s reopen/exit and writes its journal. The uploader retries independently with bounded multipart memory, verifies size and SHA-256 in the bucket, then lets archive processing and verified cleanup proceed. An unavailable decoder context is reported without discarding the original. Do not delete local originals, pending capture chunks or journals to clear an alert.

Original FLV archival is always enabled for the SRS recorder, independently of the forensic profile. Forensic mode additionally captures the received RTMP traffic; it does not control whether original FLVs are uploaded. Local originals are only removed after their bucket acknowledgement and archive dependencies verify.

Replay packaging splits codec and resolution changes into separately described MP4s, preserving compressed pictures rather than transcoding them. All parts of one original replace its catalog entries in one transaction after upload. The old MP4 objects and original FLV remain in storage. Startup pictures received before an independently decodable keyframe remain in the original and are explicitly counted as unavailable in replay metadata.

To rebuild older SRS replay with the current packager, run the worker’s bun src/indexer/srs-rebuild.ts --from ISO_TIMESTAMP --to ISO_TIMESTAMP first. Apply with --apply --confirm MEDIA_PRESERVE_HOST after reviewing that preview. The command verifies originals, saves the prior queue results under the private diagnostics/rebuilds directory, and requeues only completed legacy jobs. Existing replay remains indexed until each replacement succeeds. It never deletes bucket objects or rewrites source footage.

Mainframe’s Media setting Start streams automatically defaults to on. Backend demand starts online video sources without a viewer; providers handle device recovery. Start and Stop remain available in each feed’s stream menu. A manual stop lasts until the source goes offline and returns. The Autel provider waits 90 seconds before retrying a silent publisher, and requires healthy recorder inventory before interrupting a connected publisher with no fresh video.

The managed MQTT broker logs all packet types, including connection, subscription and publish metadata. Full inbound message payloads remain in the private Autel capture files, subject to their configured size and retention bounds. Container logs retain the configured rotation limits; they are not a payload archive.

Recent MP4s are disposable derived cache entries: two running jobs, sixteen pending streams and 512 MiB including scratch reservations. Cache pressure returns unavailable rather than blocking recording. A stopped video stream with continued silent Autel AAC is not video coverage.

The old media:jobs command and native-recorder quarantine sidecars are historical audit tools; they do not control the SRS worker. See the checked-in deploy/srs/README.md for current runtime and rollback boundaries.