> ## Documentation Index
> Fetch the complete documentation index at: https://playbook.smedleygroup.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Backups and Disaster Recovery

> How the shared MongoDB data is backed up, the current retention, and what recovery involves.

This page describes how the shared MongoDB data is backed up, what the retention really is, and what recovery looks like.

## How Backups Work

A backup script (`backup_mongo.sh`) runs on the MongoDB instance on a schedule set up during provisioning. For a given environment it runs `mongodump` against the database (`TalentId_DEV` or `TalentId_PROD`), compresses the result, and copies it to the backup S3 bucket (`talentid-mongo-backups`).

Cron schedule on the instance:

* **DEV database:** every day at 01:00.
* **PROD database:** every day at 02:00.
* **On reboot:** a separate job resyncs the database password from Secrets Manager (not a backup, but it runs from the same cron setup).

> [!WARNING]
>
> The instance hosts many more databases than the two named in the repository's script. In practice the backup script is edited directly on the running instance to add each new database, so the live backup coverage is broader than, and can drift from, what is in source control. A rebuild from the repository alone would not back up those extra databases until the script is re-edited on the new instance.

## Retention

> [!WARNING]
>
> The backup bucket is not versioned, and each run writes to a fixed file name (`TalentId_DEV.gz` and `TalentId_PROD.gz`). Every night overwrites the previous night. There is effectively one backup per environment at any time, with no history.

The practical consequences:

* The recovery point is up to 24 hours. Anything written since the last nightly dump is lost if the instance is lost.
* There is no protection against a bad backup. If a night runs against a damaged database, or the dump fails part way, the last good backup has already been overwritten.
* There is no point in time recovery and no long term retention.

## Recovery

Restoring is a manual process and is not automated in this repository. Recovery after losing the instance means redeploying the MongoDB stack to create a fresh instance, then restoring the latest dump from the backup bucket with `mongorestore`. There is no scripted restore and no regularly tested runbook in the repo, so the real recovery time is whatever a manual rebuild and restore takes on the day. Usually done via the EC2 Windows workstation. Direct database connections should be extremely rare.

## Recommended Improvements

These are suggestions, not current behaviour:

* Enable versioning on the backup bucket, plus a lifecycle rule, so several days or weeks of backups are kept.
* Run backups more frequently than daily if a 24 hour recovery point is too high for production.
* Add EBS snapshots of the data volume as a second, independent recovery path.
* Write and regularly test a restore runbook.

> [!NOTE]
>
> [SG-Infra](/software/developer/estate/infra/sg-infra) already addresses the first two gaps: its backup bucket is versioned with retention, and its data lives on a separate retained volume. See [SG-Infra Backups and Disaster Recovery](/software/developer/estate/infra/sg-infra/backups).

## Key Files

| File | Role |
| - | - |
| `sg_infra/mongo_conf/backup_mongo.sh` | The backup script (`mongodump` then copy to S3) |
| `sg_infra/mongo_conf/init_instance.sh` | Sets up the backup cron schedule |
| `sg_infra/talent_id_externaldependencies_stack.py` | References the backup bucket |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.