Monitoring and logs

Prometheus collects numeric measurements over time. Grafana queries those measurements and presents dashboards. Loki stores container logs, and Alloy forwards logs from this Compose project. The Terminal browser is not instrumented by this stack.

Open /grafana/ on the Mainframe domain and sign in with the separate Grafana credentials from the deployment environment. Self-registration and anonymous access are disabled. Metrics endpoints and the SRS API remain private.

Dashboard guide

Dashboard	Use it to answer
Deployment	Which service is down, and is the host under pressure?
Core	Are samples arriving and persisting? Are commands, tasking, documents, and HTTP requests working?
Modules	Is each module connected, healthy upstream, publishing, and handling commands?
Media and replay	Is SRS receiving video? Are DVR segments reaching S3 and Mainframe?
Simulator and automation	Where are virtual aircraft? Are batteries cycling and missions covered?
Host and services	Are CPU, RAM, disk, network, Caddy, and Postgres healthy?

A process being reachable is only the first check. An available radar-net module may still report stale upstream data. An available indexer may still have failed segments. A connected auto-operator should also have recent successful evaluations and mission coverage.

What is collected

Core measures validated sample ingest, committed samples, storage queues and write failures, HTTP latency and errors, command lifecycle updates, tasking phases, entity counts, client sessions, leases, documents, and dependency checks.

The provider SDK measures module connections, reconnects, upstream health, queued/dropped samples, publishing rate, protocol errors, and provider command outcomes. Modules can expose numeric workload statistics such as stale records, active analysis sessions, or broker message counts.

World-sim adds camera rendering time, rendered frame count, active encoders, encoder failures, aircraft states, battery levels, and coordinates. Aircraft serials are labels because the fleet is small and bounded; temporary detection IDs are not labels. The map requires an internet basemap, while position measurements remain in the local monitoring store.

Alerts and retention

Prometheus rules detect unavailable services, disconnected modules, unhealthy dependencies, storage failure/backlog, DVR stalls, repeated encoder failures, stalled automation, prolonged mission gaps, and low disk or memory. The incidents group covers what production once had unseen: a container restarted without a deploy, any kernel OOM kill, replay cursors lost, slow seek snapshots, a full replay gate, late phone positions, video refused while the phone reports it live, Front refusing phones, one query hogging Postgres, long transactions and a disk filling within a day.

Prometheus hands firing alerts to Alertmanager, which groups them by deployment, service and alert name, waits 30 seconds for the first notification, batches updates every five minutes and repeats an unresolved alert every four hours. Every alert carries a deployment label (the Compose project name), so one chat can serve several hosts.

Prometheus retains up to 30 days or 15 GB, whichever limit is reached first. Loki retains 30 days. Application replay has its own retention settings and does not depend on Prometheus.

Alert delivery

Delivery is set in the deployment environment; nothing is sent until a destination is set:

Variable	Use
ALERTMANAGER_WEBHOOK_URL	POSTs Alertmanager’s webhook JSON to this URL. A chat service that needs its own format needs an adapter in between.
ALERTMANAGER_TELEGRAM_BOT_TOKEN, ALERTMANAGER_TELEGRAM_CHAT_ID	Sends one message per group to a Telegram chat: severity, alert, deployment, service and summary. Both are required; the chat id is numeric (-100… for a group).

Either, both or none can be set. With none, alerts still group in Alertmanager and show in Grafana under Alerting (the Alertmanager data source). The Alertmanager UI listens on 127.0.0.1:9093 of the host only (ALERTMANAGER_PORT); open it through an SSH tunnel. Secrets are written to files inside the container, never into the rendered configuration. A setting that does not parse stops the container with the reason in its log.

After changing a destination, restart the alertmanager service and send a test alert from the host:

curl -XPOST 127.0.0.1:9093/api/v2/alerts -H 'content-type: application/json' \
  -d '[{"labels":{"alertname":"DeliveryTest","severity":"warning","service":"alertmanager"},"annotations":{"summary":"Delivery test"}}]'

It arrives after the 30-second group wait. AlertDeliveryFailing fires when a destination refuses notifications; AlertmanagerUnreachable shows in Grafana only, since nothing can deliver it.

Dashboards, data sources, scrape configuration, and alert rules are versioned under deploy/monitoring/. Edit the source files to make durable changes; provisioned dashboards are read-only in Grafana. Running bun run deploy config regenerates the target list for the selected modules.

Recording health is checked per active SRS stream whose entity is online and whose policy permits recording. Freshness uses the deployment’s 60-second segment cadence. The media dashboard distinguishes trailing delay, missing intervals between indexed segments, pending original bytes, upload failures, archive backlog and recent-cache pressure. An intentional publisher stop removes its expected recording series.

Gap checks cover the observed continuous active session, capped at one hour. After a Core restart, observation starts again; these checks do not certify the older archive. Inventory failures retain the previous observations and raise a stale-monitor alert. Each inventory is capped at 128 stream labels; exceeding that cap is also visible and alerts.

The node exporter uses the Linux host network namespace, following the exporter’s container guidance. Its listener binds only to the private Docker bridge gateway. The launcher discovers this address; set NODE_EXPORTER_HOST_GATEWAY only when using a different private bridge. With raw Compose, supply the gateway explicitly when it differs from 172.17.0.1. Prometheus resolves node-exporter to the same private address via Compose extra hosts.

Host network charts exclude loopback, Docker bridges, and veth interfaces to avoid double-counting traffic. On macOS, these are Linux VM metrics. Network history collected before the host namespace correction must not be used for bandwidth estimates; record the rollout time in the deployment evidence.

The PostgreSQL dashboard shows database/table/index and snapshot growth, query calls, execution time, rows, and temporary-block use. The monitoring startup job installs pg_stat_statements idempotently after PostgreSQL becomes healthy; the server preloads the extension. This is required by PostgreSQL. The exporter limits statement results to 100 and disables SQL-text labels. Its pinned custom-query support supplies temporary-block counts; revisit that configuration when upgrading the exporter.

For a bounded, read-only diagnosis from the deployment checkout:

docker compose --env-file deploy/.env.staging -f deploy/compose.yaml exec -T postgres psql -U mainframe -d mainframe < deploy/monitoring/postgres/diagnose.sql

The result includes query IDs, timings and storage sizes. If SQL text is needed to identify a specific query, inspect pg_stat_statements through an authorized database session and keep the result private; normalized query text can still contain identifying data. Do not reset statistics during a before/after measurement.

Respond to operational alerts

• Command outcome failed/unresolved: inspect the command audit and its child launch. An unresolved outcome is not evidence that the aircraft stayed on the ground. Reconcile device telemetry before retrying. Lease and serial-filter refusals are counted separately and do not page as provider failures.
• Container memory: compare working set, process RSS, replay active/queued transfers and post-load idle periods. Capture a before/after sample before restarting; a restart alone does not establish the cause.
• Dropped operational events: inspect event kind/reason counters. Command audit lookup remains available even when the events feed is limited.
• Recording alerts: open the media dashboard for the specific stream, then inspect pending/quarantined jobs using the troubleshooting commands. An acknowledged quarantine remains a gap in the recording.
• Backup freshness: inspect the most recent backup manifest and maintenance result. Success means the off-host upload was verified, not merely that a local dump was started.
• OOM kill or restart: journalctl -k | grep -i oom names the victim; read the service’s log from before the restart. A Mainframe restart loses every replay cursor; clients re-create theirs, so ReplayCursorsLost firing afterwards means one did not.
• Replay snapshots or gate: compare the Postgres dashboard’s query time with the replay transfer panels. A seek waits for the state snapshot; a full gate shows as NO RECORDING or a stalled video.
• Phones late, refused, or video refused while live: Front’s phones_refused, request_failed and store_write_failed lines name the reason and a sample of phones; Mainframe logs media_offer_refused with the entity.

Run the alert regression checks with the pinned Prometheus image:

docker run --rm --network none --entrypoint /bin/promtool -v "$PWD/deploy/monitoring/prometheus:/rules:ro" -w /rules prom/prometheus:v3.14.0 test rules media-rules.test.yml operational-rules.test.yml srs-rules.test.yml incident-rules.test.yml
bun deploy/monitoring/check-postgres.ts

A delivered test (above) is required before calling notifications operational. Check the delivery configuration with the pinned Alertmanager image, and the log pipeline with the pinned Alloy and Loki images:

bun deploy/monitoring/check-alertmanager.ts
bun deploy/monitoring/check-alloy.ts

Access logs retain every status, duration, byte count and original path, but omit request/response headers and query strings. The proxy generates a request ID and forwards it to Mainframe for replay correlation. Runtime proxy logs also strip headers and query parameters. Search the Caddy request_id and Core requestId fields together.

Mainframe, the media indexer, the derivative worker, the provider SDK in every module, and Front write one JSON object per line: ts, level, component, event, message, then fields; errors keep their name, a redacted message, a code and a few stack frames. Alloy makes level a Loki label (debug, info, warn, error) and component, event and code structured metadata, so a search is a label plus filters:

{service="mainframe", level="error"}
{service="mainframe"} | event="cursor_replay_failed"
{service=~".+"} | component="provider" | event="provider_rejected"
{service="front"} | event="phones_refused"

A failure that repeats (a timer, a retry loop, a reconnecting provider) is logged once when it starts, with a count each minute while it lasts, and once when it ends (state is started, ongoing or ended).

SRS logs connection and recorder events. Mainframe reports refused media access without credentials. Periodic operations and indexing jobs report failures and recovery counts; counters include every failure. Raw controller captures (autel, vanagas) remain separate protected files, not operational logs.