> ## Documentation Index
> Fetch the complete documentation index at: https://playbook.smedleygroup.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Ingestion & Ranking

# Overview

JSON files are ingested through the admin website. On upload, the file is parsed, patched by the admin user and stored in S3. Then the API triggers a recalculation of KPIs for the drivers in the file, which triggers enriched KPIs and finally rankings & global scores for *all* drivers (rankings are generated by the API). The process takes around 10 minutes currently.

Once the rankings are generated, they are viewable on the user webapp.

Through the Admin page, you can edit the drivers, and merge 2 drivers into 1, that are erroneously counted as different people.

# Components

## Admin Webapp

| Purpose | Link | Notes |
| - | - | - |
| App | [https://dev-admin.global.talent-id.com/drivers](https://dev-admin.global.talent-id.com/drivers) | |
| Repo | [https://eu-west-2.console.aws.amazon.com/codesuite/codecommit/repositories/TalentID-Global-WebApplication-Admin/browse?region=eu-west-2](https://eu-west-2.console.aws.amazon.com/codesuite/codecommit/repositories/TalentID-Global-WebApplication-Admin/browse?region=eu-west-2) | |
| CDK Repo | [https://eu-west-2.console.aws.amazon.com/codesuite/codecommit/repositories/TalentID-Global-WebApplication-Admin-CDK/browse?region=eu-west-2](https://eu-west-2.console.aws.amazon.com/codesuite/codecommit/repositories/TalentID-Global-WebApplication-Admin-CDK/browse?region=eu-west-2) | |
| Build Pipeline | [https://eu-west-2.console.aws.amazon.com/codesuite/codepipeline/pipelines/dev-ADMIN-TalentId-Global-React-Web-App/view?region=eu-west-2](https://eu-west-2.console.aws.amazon.com/codesuite/codepipeline/pipelines/dev-ADMIN-TalentId-Global-React-Web-App/view?region=eu-west-2) | Builds but does not deploy |
| Deploy Pipeline | [https://eu-west-2.console.aws.amazon.com/codesuite/codepipeline/pipelines/dev-TalentId-Global-Web/view?region=eu-west-2](https://eu-west-2.console.aws.amazon.com/codesuite/codepipeline/pipelines/dev-TalentId-Global-Web/view?region=eu-west-2) | 'Release Change' to update the website (deploys both the user & admin sites together) |

## User Webapp

| Purpose | Link | Notes |
| :- | :- | :- |
| App | [https://dev.global.talent-id.com/](https://dev.global.talent-id.com/) | |
| Repo | [https://eu-west-2.console.aws.amazon.com/codesuite/codecommit/repositories/TalentID-Global-WebApplication-User/browse?region=eu-west-2](https://eu-west-2.console.aws.amazon.com/codesuite/codecommit/repositories/TalentID-Global-WebApplication-User/browse?region=eu-west-2) | |
| CDK Repo | [https://eu-west-2.console.aws.amazon.com/codesuite/codecommit/repositories/TalentID-Global-WebApplication-User-CDK/browse?region=eu-west-2](https://eu-west-2.console.aws.amazon.com/codesuite/codecommit/repositories/TalentID-Global-WebApplication-User-CDK/browse?region=eu-west-2) | It's out of date - need build for Angular not React, and Node version 22+ |
| Build Pipeline | [https://eu-west-2.console.aws.amazon.com/codesuite/codepipeline/pipelines/dev-ADMIN-TalentId-Global-React-Web-App/view?region=eu-west-2](https://eu-west-2.console.aws.amazon.com/codesuite/codepipeline/pipelines/dev-ADMIN-TalentId-Global-React-Web-App/view?region=eu-west-2) | Builds but does not deploy |
| Deploy Pipeline | [https://eu-west-2.console.aws.amazon.com/codesuite/codepipeline/pipelines/dev-TalentId-Global-Web/view?region=eu-west-2](https://eu-west-2.console.aws.amazon.com/codesuite/codepipeline/pipelines/dev-TalentId-Global-Web/view?region=eu-west-2) | 'Release Change' to update the website (deploys both the user & admin sites together) |

## Lambdas

There are 3 lambdas.

The rough flow is: API (file import) → SNS → SQS → Lambda 1 → SNS → SQS → Lambda 2 → SNS → SQS → Lambda 3 → API (rankings).

When an event is FKL, it generates extra enriched KPIs & rankings into a separate collection (note: stage 3 enriched KPIs, and stage 1/2 KPIs are not duplicated for FKL). At stage 4, 5 & 6 the enriched KPIs and rankings are duplicated. Lambda 3 gets 2 separate invocations from lambda 2 in the case of an FKL event.

Lambdas are run on drivers in batches rather than waiting for all calculations of the previous stage to finish.

![Lambdas Drawio](https://mintcdn.com/smedley_group/J4Qb1kGrTQEJOjVR/images/lambdas.drawio.png?fit=max\&auto=format\&n=J4Qb1kGrTQEJOjVR\&q=85\&s=325d41b3989801d8341cb0f188fdb07e)

## API

## JSONs Bucket

All the events imported are kept at the [talentid-global-json](https://eu-west-2.console.aws.amazon.com/s3/buckets/talentid-global-json?region=eu-west-2) bucket. No backups are made. The file stored is the patched version after it's edited in the admin webapp.

# Running Locally

To run locally, set up the API, database, admin webapp and user webapp.

To set up the API, you need Docker Compose (which may require virtualisation enabled, and WSL installed/updated).

Extract [db-setup.zip](https://robsmedley.sharepoint.com/sites/mainshare/_layouts/15/guestaccess.aspx?share=IQCe2v0u1NhSQouDE_zyFiRxAXRCP8JHimqlhCUcfH8CLTE\&e=HCt8a7) (Windows only)\
Run

```text theme={null}
docker-compose up -d
```

You can connect to it with Studio 3T with a new connection with URI

```text theme={null}
mongodb://admin:admin@localhost:30000/admin?retryWrites=true&loadBalanced=false&connectTimeoutMS=10000&authSource=admin&authMechanism=SCRAM-SHA-1&3t.uriVersion=3&3t.connection.name=mongo+test+3&3t.databases=admin&3t.alwaysShowAuthDB=true&3t.alwaysShowDBFromUserRole=true
```

You can use Studio 3T or Mongo Tools to restore the latest backup, found at [https://talentid-mongo-backups.s3.eu-west-2.amazonaws.com/TalentId-Global\_DEV.gz](https://talentid-mongo-backups.s3.eu-west-2.amazonaws.com/TalentId-Global_DEV.gz)

If you want the actual current version, you need to export it using the EC2 workstation.

Check the README on the lambdas repo for information on how to run. Expect it to take several hours to run each stage, since it generates the KPIs on all drivers locally.

To run the user & admin webapps, do

```text theme={null}
npm install
npm start
```

# Known Issues

Updating a driver in the admin webapp does not edit the JSONs the driver is in (whereas a merge does)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.