Skip to main content
Active

​
Overview

These pages were written in September 2026 by David Shortland. They describe the estate as it was at that point.
Infrastructure BAU is the recurring operational work that keeps the AWS estate running between projects. None of it is feature work and none of it is incident response. It is the routine checking that catches problems while they are still cheap, and it exists because almost everything in the estate is self-managed rather than a managed AWS service. The estate has no dedicated operations function. This section is the substitute: a fixed rhythm of weekly, monthly and quarterly checks, currently sized to about a day a week.
This section covers infrastructure only. Application code maintenance is separate, covering dependency and framework updates (for example .NET and Angular package upgrades), CVE triage on those dependencies, and keeping the test suites green. That work is not documented here.

​
Key Facts

​
How You Connect

There are two routes, and which one you use depends on what you are checking. For anything on a MongoDB instance itself, open the EC2 console, select the instance, choose Connect, then the Session Manager tab, and connect from there. That gives you a shell on the instance in the browser, with no keys and nothing installed locally. The Windows workstation (also referred to as the EC2 desktop) is a third EC2 instance in the same VPC as the MongoDB instances, reachable through Session Manager and Remote Desktop. Studio 3T is installed on it, and it is the route to the databases for routine checks such as sizes and growth.
Do not connect to the MongoDB instances directly from your own machine. Database access goes through Studio 3T on the Windows workstation, and instance level checks go through the EC2 console using Connect > Session Manager.

​
Connecting To The Windows Workstation

The workstation is normally kept stopped, so connecting to it is start, use, stop. It sits in a private subnet with no public address, so there is no direct Remote Desktop route to it: Session Manager forwards the Remote Desktop port to a local port on your machine instead. The commands below are PowerShell, and need the AWS CLI and the Session Manager plugin installed, with a named profile configured for the account and permission to read the credentials secret.
1

Start the instance

Start it from the EC2 console, or from PowerShell:
Wait for the instance to reach the running state before opening the tunnel.
2

Retrieve the credentials

The username and password are held in Secrets Manager under /Infra/Workstation/rdp. Read the secret in the console, or from PowerShell:
3

Open the port forwarding session

Leave this running for as long as you need the connection. Closing it closes the tunnel.
4

Connect with Remote Desktop

Open Remote Desktop Connection against the forwarded local port, and log in with the credentials from the previous step.
5

Stop the instance when you are finished

Close the port forwarding session, then stop the instance from the EC2 console or from PowerShell:
Confirm it has actually reached the stopped state rather than assuming the command took effect.
The workstation is expensive to run and must be stopped once you are done. Leaving it running is a standing cost for a machine nobody is using, and it will show up in the weekly cost review as an unexplained increase.
--profile TID refers to a named AWS CLI profile configured locally. Substitute whichever profile name you have configured for the account.

​
Why This Is Needed

The estate is built almost entirely on self-managed infrastructure, which means the operational burden sits with the team rather than with AWS:
  • The databases are self-hosted. Both MongoDB instances run on EC2 that the team owns end to end: operating system, patching, disk, backups and recovery. See Shared MongoDB Instance.
  • Each database is a single point of failure. Single instances with single node replica sets, no automatic failover. Recovery is a manual restore.
  • Backup coverage lives on the instances. The backup scripts are edited directly on the running instances as databases are added, so source control does not reflect what is actually backed up. See Backups and Disaster Recovery.
  • The blast radius is the whole estate. One VPC and one shared database serve most systems, so a single problem can affect everything at once.
  • Some failures are silent. A Vivenu webhook that stops delivering, a backup that writes an empty file, or a queue quietly discarding messages will not raise an alarm. They are found by looking.

​
The Cadence

Weekly Checks

Database health, backups, Vivenu webhooks, alarms, queues, cost and pipelines.

Monthly Deep Dives

Restore rehearsal, capacity and data growth, and cost optimisation.

Quarterly Reviews

Access and security review, documentation refresh, and vendor review.

​
How To Use This Section

Work through Weekly Checks in order once a week. Most weeks nothing will be wrong, and the value is the trend rather than the snapshot: disk that grew faster than usual, a backup that shrank, a cost line that moved. Record what you saw so the following week has something to compare against.
Anything you find that is not covered by these pages should be added to them. The estate has very little operational documentation, and this section is only useful for as long as it reflects what is actually being run.
Last modified on September 4, 2026