Overview
JSON files are ingested through the admin website. On upload, the file is parsed, patched by the admin user and stored in S3. Then the API triggers a recalculation of KPIs for the drivers in the file, which triggers enriched KPIs and finally rankings & global scores for all drivers (rankings are generated by the API). The process takes around 10 minutes currently. Once the rankings are generated, they are viewable on the user webapp. Through the Admin page, you can edit the drivers, and merge 2 drivers into 1, that are erroneously counted as different people.Components
Admin Webapp
User Webapp
Lambdas
There are 3 lambdas. The rough flow is: API (file import) → SNS → SQS → Lambda 1 → SNS → SQS → Lambda 2 → SNS → SQS → Lambda 3 → API (rankings). When an event is FKL, it generates extra enriched KPIs & rankings into a separate collection (note: stage 3 enriched KPIs, and stage 1/2 KPIs are not duplicated for FKL). At stage 4, 5 & 6 the enriched KPIs and rankings are duplicated. Lambda 3 gets 2 separate invocations from lambda 2 in the case of an FKL event. Lambdas are run on drivers in batches rather than waiting for all calculations of the previous stage to finish.
API
JSONs Bucket
All the events imported are kept at the talentid-global-json bucket. No backups are made. The file stored is the patched version after it’s edited in the admin webapp.Running Locally
To run locally, set up the API, database, admin webapp and user webapp. To set up the API, you need Docker Compose (which may require virtualisation enabled, and WSL installed/updated). Extract db-setup.zip (Windows only)Run