> ## Documentation Index
> Fetch the complete documentation index at: https://playbook.smedleygroup.com/llms.txt
> Use this file to discover all available pages before exploring further.

# DAQ Upload and Conversion

> How a DAQ run travels from the kart to STL: the local upload, the zip contents, run numbering, and what the converter does.

Telemetry data recorded by DAQ units is uploaded to S3, where it is processed into STL format, which our software can read. Each file should represent a single run of multiple laps of a kart.

Conversion and reconversion are automatic. The conversion uses the TalentID API and GKL Platform API to cross reference data such as the passings, the driver and the event type. This page covers the journey up to the point of conversion; [Timing and Lap Splitting](/software/developer/estate/systems/telemetry/timing-and-lap-splitting) covers what those cross references are and how laps are cut.

## Uploading From The Track

The DAQ uploads to the local telemetry API on the track machine, at its `/upload` endpoint, and that API uploads to S3.

The API assembles a zip containing:

| Contents | Purpose |
| - | - |
| The binary recording | The run itself |
| A combined logging table | Describes the expected signals, including for different battery types |
| Metadata | See below |

The metadata carries more than the run itself:

| Field | Purpose |
| - | - |
| `Id` | The document ID this run is recorded under in TalentID. The handle for it everywhere else |
| `StartTime`, `EndTime` | The run window, used with the transponder to match timing |
| `TransponderId` | The match key against a timing competitor |
| `HubRegion` | Which region the run belongs to |
| `DeviceId` | The kart, which becomes `EK{id}` in the output path |
| `RunNumber` | Determined locally, from the run zips already stored on the machine for that event |
| `ZipFilesChecksum` | Drives the decision to reconvert |
| `Force` | Convert regardless of either checksum |
| `HeaderVersion` | Which binary format parser to use |
| `GpsPresent`, `Category` | Recorded alongside the run |

> [!WARNING]
>
> Because the run number comes from local state, re-uploading a run from anywhere other than the track machine will give it the wrong number, unless all the previous run zips for that day are present locally first.

If the file is broken in some way, or does not pass validation, it is stored on the track machine and not uploaded. On upload failure, if the full file was correctly received, the binary is saved locally on the machine.

## Reaching The Cloud

**1. The zip lands in S3**

The upload goes to the `daq-data/` prefix of the telemetry bucket.

**2. The upload is fanned out**

S3 object creation notifies an SNS topic, which feeds an SQS queue with a dead letter queue behind it.

**3. The converter runs**

The queue triggers the converter Lambda, which is a container image built by the pipeline and pushed to ECR. See [Deployment And The Pipeline](/software/developer/estate/systems/telemetry/deployment) for how it is built and pinned.

## What Conversion Does

Once on S3, the Lambda processes the file and tracks conversion progress in the TalentID database, which is what makes it visible on the DAQ page in the Operator Dashboard.

Processing runs in this order:

**1. Resample and decode**

The raw stream is resampled to a fixed 10 ms interval, which is 100 samples per second, and decoded against a DBC.

**2. Normalise**

Channels are renamed and scaled against the combined logging table, and any missing channels are added.

**3. Forward fill**

A zero order hold carries each signal's last value forward, so every channel has a value at every sample.

**4. Lap split**

The run is cut into laps using timing from TalentID. See [Timing and Lap Splitting](/software/developer/estate/systems/telemetry/timing-and-lap-splitting).

**5. Export**

The result is written out as STL files.

> [!NOTE]
>
> The forward fill is bounded per channel, at twice that channel's expected sample interval. A signal that simply stops reporting therefore goes to NaN rather than being held at a stale value indefinitely.

The output is a pair of MATLAB `.mat` files per run, `Run{NN}_Info.mat` and `Run{NN}_Tel.mat`, saved to S3. Their keys are recorded against the run in TalentID, which is how the Operator Dashboard knows a run has converted successfully.

> [!NOTE]
>
> The run is registered in TalentID before conversion is attempted, not after. That is why a run appears in the Operator Dashboard while conversion is still pending, and why a failed conversion is visible rather than silently absent.

## Where The Output Goes

Converted runs are filed into the STL database in the telemetry bucket, which engineers sync down and open in SimTelLab. The `Race` or `Test` segment of that path comes from the `isRaceDay` answer GKL API gives during conversion. See [The STL Database and SimTelLab](/software/developer/estate/systems/telemetry/stl-database).

## Where To Go Next

| Page | For |
| - | - |
| [Timing and Lap Splitting](/software/developer/estate/systems/telemetry/timing-and-lap-splitting) | How timing is matched, how laps are cut, and how reconversion is triggered |
| [Troubleshooting](/software/developer/estate/systems/telemetry/troubleshooting) | A run that did not arrive, did not convert, or does not show on the dashboard |
| [Track Operations](/software/developer/estate/systems/telemetry/track-operations) | The machine the upload comes from |

## Key Files

| File | Role |
| - | - |
| `telemetry/telemetry-api/src/api.py` | The local API, including the `/upload` endpoint |
| `telemetry/telemetry-api/src/lambda_function.py` | Converter Lambda entry point |
| `telemetry/telemetry-api/src/offloaded/pipeline/` | The conversion stages |
| `CdkBuild/lib/telemetry_cdk_stack.ts` | Bucket notification, topic, queues and the Lambda |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.