Uploading From The Track
The DAQ uploads to the local telemetry API on the track machine, at its/upload endpoint, and that API uploads to S3.
The API assembles a zip containing:
The metadata carries more than the run itself:
If the file is broken in some way, or does not pass validation, it is stored on the track machine and not uploaded. On upload failure, if the full file was correctly received, the binary is saved locally on the machine.
Reaching The Cloud
1
The zip lands in S3
The upload goes to the
daq-data/ prefix of the telemetry bucket.2
The upload is fanned out
S3 object creation notifies an SNS topic, which feeds an SQS queue with a dead letter queue behind it.
3
The converter runs
The queue triggers the converter Lambda, which is a container image built by the pipeline and pushed to ECR. See Deployment And The Pipeline for how it is built and pinned.
What Conversion Does
Once on S3, the Lambda processes the file and tracks conversion progress in the TalentID database, which is what makes it visible on the DAQ page in the Operator Dashboard. Processing runs in this order:1
Resample and decode
The raw stream is resampled to a fixed 10 ms interval, which is 100 samples per second, and decoded against a DBC.
2
Normalise
Channels are renamed and scaled against the combined logging table, and any missing channels are added.
3
Forward fill
A zero order hold carries each signal’s last value forward, so every channel has a value at every sample.
4
Lap split
The run is cut into laps using timing from TalentID. See Timing and Lap Splitting.
5
Export
The result is written out as STL files.
The forward fill is bounded per channel, at twice that channel’s expected sample interval. A signal that simply stops reporting therefore goes to NaN rather than being held at a stale value indefinitely.
.mat files per run, Run{NN}_Info.mat and Run{NN}_Tel.mat, saved to S3. Their keys are recorded against the run in TalentID, which is how the Operator Dashboard knows a run has converted successfully.
The run is registered in TalentID before conversion is attempted, not after. That is why a run appears in the Operator Dashboard while conversion is still pending, and why a failed conversion is visible rather than silently absent.
Where The Output Goes
Converted runs are filed into the STL database in the telemetry bucket, which engineers sync down and open in SimTelLab. TheRace or Test segment of that path comes from the isRaceDay answer GKL API gives during conversion. See The STL Database and SimTelLab.