Updates, backups, and retention

A normal deployment rebuild preserves named volumes. Keep the Compose project name and environment file stable: changing the project name creates a different set of volumes and can look like an empty deployment.

Update static sites

Push the reviewed frontend or docs change to main. Cloudflare Pages builds and deploys both sites using the configuration managed in the infra repo. Check both Pages deployment results and verify the custom domains, docs navigation, and Terminal login against Mainframe. No server checkout or Docker command is needed for a static-only change.

Update the backend

bun run deploy backup
# Check out the reviewed release or commit.
bun run deploy config
bun run deploy up
bun run deploy ps

Mainframe’s image applies migrations before starting. A failed migration should leave the service unhealthy rather than serving against an unexpected schema. Review migration compatibility before rolling back an image; restoring the old image does not undo a database migration.

After an update, verify login, module reconnects, telemetry, live video, and a new DVR segment. Check the logs and command failures. Restarting a process can interrupt live streams and pending commands even when all persisted data is intact.

Backups

bun run deploy backup writes a Postgres custom-format dump under the ignored deploy/backups/ directory with restricted permissions. Copy backups off the server; a local dump does not protect against losing the host.

For verified private off-host backups, run on the deployment host:

bun deploy/recovery/backup.ts --env-file deploy/.env.staging
bun deploy/recovery/restore.ts --env-file deploy/.env.staging --prefix backups/scheduled/PROJECT/BACKUP_ID
# After the restore drill passes, install the daily host timer as root:
sh deploy/recovery/install.sh

The backup command prints its prefix. It streams the custom-format database dump into the private bucket, downloads it to verify SHA-256 and size, and only then publishes a verified manifest. The scheduled job runs daily after 03:00 UTC, with up to 15 minutes of jitter. BACKUP_RETENTION_DAYS defaults to seven; cleanup only touches complete scheduled backups for this Compose project. Manual/initial backups and media objects are independent. A failed run preserves the previous success timestamp and its local working copy for diagnosis.

The manifest also inventories independently copied world-sim state, native Autel configuration/accounts, broker dynamic-security state, and deployed TAK server certificates. These artifacts contain credentials and belong only in the private bucket. The environment file, bucket bootstrap credentials, Terraform state and offline TAK CA signing key remain in the operator’s separate secret and infrastructure stores. Losing the host must not lose the only bucket key.

An online database dump is one database snapshot. Module files have separate capture times; this is not a coordinated application snapshot. Media stays in the private bucket, and its retention can invalidate old backup references. For a coordinated recovery point, quiesce application writers and preserve the database, module volumes, pending DVR files and object store together. Host backups do not replace backups of attached volumes or object storage.

The restore drill downloads and checks every backup object, restores into a temporary PostgreSQL container with no network and no exposed ports, counts restored tables, checks media metadata and twenty recent object references, and compares enabled controller accounts with broker clients. It writes a restricted JSON report and removes its temporary container/volume. It does not start modules or send aircraft commands. A module-state mismatch requires reconciliation before enabling automation in a recovered deployment.

Inspect systemctl status terminal-backup.timer terminal-backup.service and journalctl -u terminal-backup.service. The node-exporter textfile collector exposes last-success, latest-run and maintenance-success gauges; backup age alerts after 36 hours. If a killed process leaves deploy/backups/PROJECT/.offhost.lock, first verify that no backup process is running, then remove that empty lock directory before retrying. Retained failed upload prefixes have no successful manifest and require explicit operator cleanup.

Restore into a separate environment first. Restore database and object data before starting Core and modules, then verify a known historical replay and a newly recorded segment. Keep the previous environment available until that check succeeds.

Raw controller captures

Autel captures are separate from operational logs. AUTEL_CAPTURE_RETENTION_HOURS and AUTEL_CAPTURE_MAX_BYTES bound recognized hourly JSONL files; zero disables each bound. The base deployment leaves them disabled; staging uses seven days and 2 GiB when no explicit environment override is present. Maintenance checks every five minutes and removes at most 24 closed files per run. The actual open file, the current hour, changed files, symlinks and unrelated files are protected. An oversized active file can exceed the limit until rotation. An incomplete inventory pauses deletion and reports failure instead of guessing.

Preview without deleting:

bun services/autel/scripts/capture-retention.ts --directory /PATH/TO/CAPTURE --hours 168 --max-bytes 2147483648 --active-file /PATH/FROM/AUTEL/HEALTH

The CLI never deletes; the running module owns deletion because it knows which file it is writing. Health and metrics expose the last successful inspection, retained bytes and failure status. Capture/maintenance failure is contained and does not interrupt MQTT telemetry or aircraft control.

Retention settings

Data	Default	Staging example
State history	90 days	90 days
Telemetry	14 days	30 days
Persisted bulk	48 hours	48 hours
Recorded media	168 hours	720 hours
Events, command audits, document history	No automatic age limit	No automatic age limit

Override them with MAINFRAME_STATE_RETENTION_DAYS, MAINFRAME_TELEMETRY_RETENTION_DAYS, MAINFRAME_BULK_RETENTION_HOURS, and MAINFRAME_MEDIA_RETENTION_HOURS. State partitions are monthly, so their actual horizon can exceed the configured age by up to a partition. Telemetry and bulk use daily partitions.

Retention runs during startup and daily. Media retention removes both indexed segments and their objects. To shorten the recording horizon immediately, use the existing media:retain tool with the deployment environment and review its output. Bucket lifecycle policies should not delete objects sooner than Core’s configured retention, or the database will retain references to missing recordings.

Validate container profile isolation without building images with bun run check:images --no-build (equivalently bun scripts/check-images.ts --no-build). The Compose CI job runs the same command.