Skip to main content
This page describes how the MongoDB data is backed up, the retention, and what recovery looks like. Several of the gaps in the legacy backup approach are addressed here.

​
How Backups Work

A backup script (backup-mongodb.sh) runs on the instance on a schedule set up at provisioning time. For a given environment it runs mongodump against the Fleet Management Platform database (FMP_DEV or FMP_PROD), compresses it, and uploads it to the backup S3 bucket. The bucket name is read from the published SSM configuration rather than hard-coded. Cron schedule on the instance:
  • DEV database: every day at 02:00.
  • PROD database: every day at 02:05.
Today SG-Infra is used only by Fleet Management Platform, so the instance hosts the FMP_DEV and FMP_PROD databases. As more systems adopt SG-Infra it will host many databases, and (as on the legacy instance) new databases are expected to be added by editing the backup script directly on the running instance, with that coverage then living on the instance rather than fully in source control.

​
Retention

The backup bucket is versioned, with a lifecycle rule that keeps non current versions for 7 days. Each nightly upload uses the same object name, but because the bucket is versioned the previous backups are kept as older versions for a week, rather than being lost on overwrite. This is an improvement over the legacy repo. The practical position:
  • The recovery point is up to 24 hours (the gap between nightly dumps).
  • There is roughly a week of backup history, so a bad or corrupt backup does not immediately destroy the last good one.
  • The backups are logical dumps (mongodump), so they capture the data but not a point in time snapshot.

​
Recovery

There are two recovery paths, and together they are stronger than the legacy single path:
  • Instance loss: because the data volume is separate and retained, a replacement instance can attach the existing volume and mount the data as it is. This avoids restoring from backup for the case of losing or replacing the instance.
  • Data loss or corruption: restore from the backup bucket with mongorestore. Versioning gives about a week of history to restore from. Usually done via the EC2 Windows workstation. Connections directly to MongoDB should be extremely limited.
As with the legacy repo, restore is a manual process and there is no scripted, regularly tested restore runbook in the code, so the real recovery time is whatever a manual restore takes on the day. These are suggestions, not current behaviour:
  • Run backups more frequently than daily if a 24 hour recovery point is too high for production.
  • Add scheduled EBS snapshots of the data volume as a second, independent recovery path.
  • Write and regularly test a restore runbook.

​
Key Files

Last modified on June 30, 2026