Monitoring and logs

Prometheus collects numeric measurements over time. Grafana queries those measurements and presents dashboards. Loki stores container logs, and Alloy forwards logs from this Compose project. The Terminal browser is not instrumented by this stack.

Open /grafana/ on the Mainframe domain and sign in with the separate Grafana credentials from the deployment environment. Self-registration and anonymous access are disabled. Metrics endpoints and the SRS API remain private.

Dashboard guide

Dashboard	Use it to answer
Deployment	Which service is down, and is the host under pressure?
Core	Are samples arriving and persisting? Are commands, tasking, documents, and HTTP requests working?
Modules	Is each module connected, healthy upstream, publishing, and handling commands?
Media and replay	Is SRS receiving video? Are DVR segments reaching S3 and Mainframe?
Simulator and automation	Where are virtual aircraft? Are batteries cycling and missions covered?
Host and services	Are CPU, RAM, disk, network, Caddy, and Postgres healthy?

A process being reachable is only the first check. An available weather module may still report stale upstream data. An available indexer may still have failed segments. A connected auto-operator should also have recent successful evaluations and mission coverage.

What is collected

Core measures validated sample ingest, committed samples, storage queues and write failures, HTTP latency and errors, command lifecycle updates, tasking phases, entity counts, client sessions, leases, documents, and dependency checks.

The provider SDK measures module connections, reconnects, upstream health, queued/dropped samples, publishing rate, protocol errors, and provider command outcomes. Modules can expose numeric workload statistics such as weather sites, stale records, active analysis sessions, or broker message counts.

World-sim adds camera rendering time, rendered frame count, active encoders, encoder failures, aircraft states, battery levels, and coordinates. Aircraft serials are labels because the fleet is small and bounded; temporary detection IDs are not labels. The map requires an internet basemap, while position measurements remain in the local monitoring store.

Alerts and retention

Prometheus rules detect unavailable services, disconnected modules, unhealthy dependencies, storage failure/backlog, DVR stalls, repeated encoder failures, stalled automation, prolonged mission gaps, and low disk or memory. Firing alerts are visible in Grafana and Prometheus. External notification delivery requires a configured receiver; the repository does not invent an email or chat destination.

Prometheus retains up to 30 days or 15 GB, whichever limit is reached first. Loki retains seven days. Application replay has its own retention settings and does not depend on Prometheus.

Dashboards, data sources, scrape configuration, and alert rules are versioned under deploy/monitoring/. Edit the source files to make durable changes; provisioned dashboards are read-only in Grafana. Running bun run deploy config regenerates the target list for the selected modules.

Recording health is checked per active SRS stream whose entity is online and whose policy permits recording. Freshness uses the deployment’s 60-second segment cadence. The media dashboard distinguishes trailing delay, missing intervals between indexed segments, pending raw bytes, and acknowledged/unacknowledged quarantine. An intentional publisher stop removes its expected recording series.

Gap checks cover the observed continuous active session, capped at one hour. After a Core restart, observation starts again; these checks do not certify the older archive. Inventory failures retain the previous observations and raise a stale-monitor alert. Each inventory is capped at 128 stream labels; exceeding that cap is also visible and alerts.

The node exporter uses the Linux host network namespace, following the exporter’s container guidance. Its listener binds only to the private Docker bridge gateway. The launcher discovers this address; set NODE_EXPORTER_HOST_GATEWAY only when using a different private bridge. With raw Compose, supply the gateway explicitly when it differs from 172.17.0.1. Prometheus resolves node-exporter to the same private address via Compose extra hosts.

Host network charts exclude loopback, Docker bridges, and veth interfaces to avoid double-counting traffic. On macOS, these are Linux VM metrics. Network history collected before the host namespace correction must not be used for bandwidth estimates; record the rollout time in the deployment evidence.

The PostgreSQL dashboard shows database/table/index and snapshot growth, query calls, execution time, rows, and temporary-block use. The monitoring startup job installs pg_stat_statements idempotently after PostgreSQL becomes healthy; the server preloads the extension. This is required by PostgreSQL. The exporter limits statement results to 100 and disables SQL-text labels. Its pinned custom-query support supplies temporary-block counts; revisit that configuration when upgrading the exporter.

For a bounded, read-only diagnosis from the deployment checkout:

docker compose --env-file deploy/.env.staging -f deploy/compose.yaml exec -T postgres psql -U mainframe -d mainframe < deploy/monitoring/postgres/diagnose.sql

The result includes query IDs, timings and storage sizes. If SQL text is needed to identify a specific query, inspect pg_stat_statements through an authorized database session and keep the result private; normalized query text can still contain identifying data. Do not reset statistics during a before/after measurement.

Respond to operational alerts

• Command outcome failed/unresolved: inspect the command audit and its child launch. An unresolved outcome is not evidence that the aircraft stayed on the ground. Reconcile device telemetry before retrying. Lease and serial-filter refusals are counted separately and do not page as provider failures.
• Container memory: compare working set, process RSS, replay active/queued transfers and post-load idle periods. Capture a before/after sample before restarting; a restart alone does not establish the cause.
• Dropped operational events: inspect event kind/reason counters. Command audit lookup remains available even when the events feed is limited.
• Recording alerts: open the media dashboard for the specific stream, then inspect pending/quarantined jobs using the troubleshooting commands. An acknowledged quarantine remains a gap in the recording.
• Backup freshness: inspect the most recent backup manifest and maintenance result. Success means the off-host upload was verified, not merely that a local dump was started.

Run the alert regression checks with the pinned Prometheus image:

docker run --rm --network none --entrypoint /bin/promtool -v "$PWD/deploy/monitoring/prometheus:/rules:ro" -w /rules prom/prometheus:v3.14.0 test rules media-rules.test.yml operational-rules.test.yml
bun deploy/monitoring/check-postgres.ts

External notification delivery remains unconfigured until an operator destination and its credentials are supplied. When enabling delivery, group alerts by deployment, service and alert name, wait 30 seconds for the initial group, batch updates for five minutes and repeat unresolved alerts no more often than every four hours. A delivered test is required before calling notifications operational.

Access logs retain every status, duration, byte count and original path, but omit request/response headers and query strings. The proxy generates a request ID and forwards it to Mainframe for replay correlation. Runtime proxy logs also strip headers and query parameters. Search the Caddy request_id and Core requestId fields together.

SRS logs warnings/errors. Mainframe records authenticated publisher authorization and close events without stream tokens. The SRS launcher redacts URL credentials before Docker captures output. Periodic operations and DVR jobs report first failures and recovery counts; counters still include every failure. Raw Autel captures remain separate protected files, not operational logs.