> ## Documentation Index > Fetch the complete documentation index at: https://playbook.smedleygroup.com/llms.txt > Use this file to discover all available pages before exploring further. # Weekly Checks > The recurring weekly infrastructure checks: database health, backups, Vivenu webhooks, alarms, queues, cost, and pipelines. This page is the weekly working list. It is ordered so that the checks most likely to be hiding a real problem come first. Instance level checks are done through the EC2 console, using **Connect > Session Manager**. Database level checks are done through Studio 3T on the Windows workstation. See [How You Connect](/software/developer/operations/index#how-you-connect). *** ## MongoDB Instance Health Both MongoDB instances are self-managed EC2, so nothing here is watched by AWS on your behalf. Check the legacy shared instance first, since it backs most of the estate. | Instance | Repo | Notes | | - | - | - | | `Mongo Node V72` | [TalentIdPipeline-Infra-CDK](/software/developer/estate/infra/talentidpipeline-infra-cdk/mongodb) | `t2.medium`, single 30 GB root volume, MongoDB 7.0 | | SG-Infra instance | [SG-Infra](/software/developer/estate/infra/sg-infra/mongodb-and-volume) | `t3.large`, separate data volume, MongoDB 8.2 in Docker | ### Instance Level, Through The EC2 Console Select the instance in the EC2 console, choose **Connect**, then the **Session Manager** tab. See [How You Connect](/software/developer/operations/index#how-you-connect). * **Disk.** On the legacy instance the MongoDB data path (`/var/lib/mongo`) sits on the same 30 GB root volume as the operating system, so filling the disk takes out the database and the host together. Record the percentage used each week so you have a growth trend and a projected exhaustion date, not just a snapshot. * **CPU credits.** The legacy instance is a `t2`, which is burstable. If `CPUCreditBalance` reaches zero the instance is throttled and the database becomes slow for no visible reason. This is the most likely cause of an unexplained slowdown on the legacy instance. * **Memory.** 4 GB on the legacy instance, shared between the operating system and the MongoDB working set. * **Instance state.** Confirm both instances are running, and check for any scheduled AWS maintenance events against them. ### Database Level, Through Studio 3T * **Replica set.** Confirm the node is `PRIMARY` and healthy. Both instances are single node replica sets, so the replica set exists for the oplog and transactions rather than for redundancy. * **Database sizes.** Record the size of each database weekly so you have a growth trend to compare against. * **Collection growth.** Look at the collections that grow with trading: `Tickets`, `Transactions`, `Checkouts` and `Audit` on the GKL databases. Growth that is out of proportion to trading volume is worth understanding. *** ## Backup Verification Backups run as cron jobs on each instance, dumping with `mongodump` and copying the archive to S3. Each system has its own backup bucket: | Bucket | Contents | | - | - | | `talentid-mongo-backups` | TalentID databases | | `gklplatform-mongo-backups` | GKL Platform databases | | The SG-Infra bucket | Fleet Management Platform databases. Its name is read from SSM rather than fixed | Verify the output rather than assuming the job ran. **1. Confirm last night's archives exist** In the S3 console, check that the `LastModified` timestamp on each archive is from last night. **2. Compare the sizes against last week** Record each archive size weekly. A dump that suddenly halves is the classic silent failure: the job reports success while writing an incomplete or empty archive. A size that never changes is equally suspicious. **3. Reconcile the database list** Compare the databases visible in Studio 3T against the archives present in the backup buckets. The backup scripts are maintained on the running instances rather than in source control, so a database added without a matching backup entry is silently unprotected. > [!WARNING] > > The legacy backup bucket is not versioned and each run writes to a fixed object name, so every night overwrites the previous night. There is one backup per database at any time. If a night runs against a damaged database, the last good backup is already gone. The SG-Infra bucket is versioned with seven days of history and does not have this problem. See [Backups and Disaster Recovery](/software/developer/estate/infra/talentidpipeline-infra-cdk/backups). The recovery point is up to 24 hours. Anything written since the nightly dump is lost in a restore, and everyone relying on this data should know that number. *** ## Vivenu Webhook Health Vivenu delivers every sale into the GKL API by webhook. If a webhook stops delivering, sales data silently stops arriving: tickets do not appear, drivers cannot be assigned, and nothing in AWS raises an alarm because no request was ever made. In Vivenu, go to **Developer > Webhooks** for each seller and review the delivery history. > [!WARNING] > > The target is zero failed attempts. Any failure means a sale, ticket or transaction that the platform does not know about, and each one needs investigating rather than noting. There are four webhooks per seller (**Event**, **Ticket**, **Transaction** and **Checkout**), so the number to review grows with every hub added. See [Configuring the Webhooks](/software/third-party/vivenu/hub-deployment/development/configure-webhooks) for the full set and naming convention. When you find a failure, the status code tells you where to look: | Symptom | Likely cause | | - | - | | `401` responses | The HMAC key in Vivenu no longer matches the key held in the GKL API region configuration | | `5xx` responses | A fault on our side. Check the API Gateway metrics and the API Lambda logs for the same timestamp | | Timeouts | The API Lambda is cold starting or running long. Check duration metrics around the failure | | No recent deliveries at all | The webhook has been deactivated, or the seller has had no activity. Confirm which | Cross-check anything you find against API Gateway 4xx and 5xx counts for the same period, so you can tell a Vivenu side problem from ours. ### Recovering Missed Deliveries Vivenu gives up retrying an event after a number of failed attempts. Once you have fixed the underlying problem, you can trigger the event to resend from the Vivenu dashboard, and the data comes through as it would have originally. Work back through the failed attempts and resend each one, then confirm in the platform that the affected records have appeared. This is the difference between a webhook failure being an inconvenience and being permanent data loss, so it is worth doing before the failures age out of the dashboard. *** ## Alarms and Error Triage Review what fired during the week and, more importantly, what should have fired and did not. **CloudWatch alarms.** The GKL API stack defines five alarms per environment, all delivering to an email topic: | Alarm | Threshold | | - | - | | API Lambda invocations | More than 500 per minute | | API Lambda errors | More than 1 per minute | | Notification Lambda errors | More than 1 per minute | | HubSpot Lambda errors | More than 1 per minute | | WAF blocked requests | More than 100 in 5 minutes | The invocation alarm is a cost and abuse tripwire rather than an error signal, since 500 per minute is far above normal traffic. **Lambda errors.** Use CloudWatch Logs Insights across the Lambda log groups for the week and group by error message. Look for new signatures rather than raw volume: a new exception type appearing once matters more than a familiar one appearing often. **SES reputation.** Check the bounce and complaint rates in the SES console. All transactional email leaves from the platform sender address, and sustained bounces put the sending domain at risk. **WAF.** Review sampled blocked requests and separate genuine attacks from real users hitting the rate limit. **API Gateway.** Check 5xx rate and p99 latency on both stages. *** ## Queue and Integration Health The estate uses SQS across several systems, not just GKL Platform. Telemetry and TalentID have their own queues, and some of those have dead letter queues configured. * **Queue depth.** Check `ApproximateNumberOfMessagesVisible` and `ApproximateNumberOfMessagesNotVisible` across the queues in both environments. A depth that is not draining means the consuming Lambda is failing. * **Dead letter queues.** Where a queue has a dead letter queue, check whether anything has landed in it, and work out what failed. Where a queue has no dead letter queue, a repeatedly failing message is retried until it expires and is then discarded with no record, so queue depth is the only signal you get. * **Notification email.** Confirm transactional email is still being sent, for example registration confirmations and password resets. * **BigQuery export.** Confirm the expected export object has landed in the shared export bucket. The export is triggered externally rather than by anything in the platform code. *** ## Cost Review The estate is small enough that a single unexpected resource is visible week to week, and large enough that nobody notices it in a monthly total. **1. Compare week on week by service** In Cost Explorer, group by service and compare against the previous week. Then group anything that moved by usage type to find what actually changed. **2. Review the standing items** NAT gateways, interface VPC endpoints, EC2 and EBS for the database instances and the Windows workstation, S3 storage across the backup and export buckets, CloudFront, and CodeBuild minutes. **3. Check the anomaly alerts** Review anything AWS Cost Anomaly Detection has raised since the last review. > [!NOTE] > > Every system shares one account with no cost allocation tags, so the question "what does GKL Platform cost?" cannot currently be answered from Cost Explorer alone. *** ## Pipeline and Drift Check * **Failed executions.** Check every pipeline across the infrastructure and application repos for failed or stuck runs. * **Drift.** Run `cdk diff` for each CDK app against the deployed stacks. > [!WARNING] > > Infrastructure has been changed outside of source control before, so a non empty `cdk diff` should be treated as something to investigate and resolve rather than as background noise. Left alone, drift means a redeploy silently reverts a change that someone made deliberately. * **Branch positions.** Confirm the deployment branches are where you expect them, since a push to a deployment branch releases with no approval step. *** ## Anomaly Buffer Keep some time back for investigating whatever the checks above turned up. Something usually needs a second look, and without reserved time it either gets skipped or pushes the rest of the week's work aside. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.