> ## Documentation Index
> Fetch the complete documentation index at: https://playbook.smedleygroup.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting

> Where to look when a DAQ run does not arrive, does not convert, or does not appear on the dashboard, when a kart streams rubbish signals, plus known issues and how to run a conversion locally.

Symptoms first, then the fix that is made at the track, then how to reproduce a conversion on your own machine, then the issues that are known and unfixed.

## Where To Look First

| Resource | Use it for |
| - | - |
| [TelemetryRawOffloadedDataConverterLambda-prod logs](https://eu-west-2.console.aws.amazon.com/cloudwatch/home#logStream:group=/aws/lambda/TelemetryRawOffloadedDataConverterLambda-prod) | Conversion and reconversion |
| [TalentIdToTelemetryReconversionChecker-prod logs](https://eu-west-2.console.aws.amazon.com/cloudwatch/home?region=eu-west-2#logStream:group=/aws/lambda/TalentIdToTelemetryReconversionChecker-prod) | Checking whether any files needed reconversion after a timing import |
| [prod-sqsTimingIngest logs](https://eu-west-2.console.aws.amazon.com/cloudwatch/home?region=eu-west-2#logsV2:log-groups/log-group/\$252Faws\$252Flambda\$252Fprod-sqsTimingIngest) | Importing a new timing file |
| [telemetry-data-sg-prod](https://eu-west-2.console.aws.amazon.com/s3/buckets/telemetry-data-sg-prod?region=eu-west-2\&tab=objects) | DAQ runs, converted racing and charging runs, live logs, backups |
| The Telemetry Dashboard on the track machine, on port 4200 | Whether a kart is showing charging data at all |
| InfluxDB's own UI on the track machine, on port 8086 | Browsing a kart's raw measurements directly, including what they are named |
| [Internal Signals](/software/developer/estate/systems/telemetry/internal-signals) | Working out what a kart is doing from the `x` prefixed signals the pipeline emits about itself |

## Common Symptoms

> **A file did not appear in daq-data**
>
> Check the `live-logs` folder in `telemetry-data-sg-prod` for a failure, and the dump folder on the track machine for the missing file if the logs name it.
>
> That file can be uploaded, but to get the right run number it should be re-uploaded using the track machine. Failing that, make sure all the previous run zips for that day are present locally in the `LocalData` folder first, because the API derives the run number from how many runs already exist there for that event.
>
> Also check that it did not go to dev, or to a different region, and check the converter logs or the database to see whether the file was received.

> **The converted file did not appear**
>
> Check the conversion logs.
>
> It may have run out of memory, or be delayed by other conversions in the same Lambda invocation.

> **A conversion failed**
>
> Inspect the conversion logs in CloudWatch, or try to convert the zip locally with force set. See [Running A Conversion Locally](/software/developer/estate/systems/telemetry/local-development#running-a-conversion-locally).
>
> There is no automatic retry on conversion failure unless one of the two checksums changes. See [Triggering A Reconversion Deliberately](/software/developer/estate/systems/telemetry/timing-and-lap-splitting#triggering-a-reconversion-deliberately).

> **The file is present but does not show on the Operator Dashboard**
>
> Most likely there is no timing associated with it. Check the transponder ID, hub region and time range in the metadata, and verify that it crosses a passing in the timing JSON.
>
> Files without timing can be found in the `TelemetryData` collection in the `TalentId_PROD` database, with `CompetitorInfo` as `null`.

> **A kart is producing rubbish signals, named xFrameIdUnknown**
>
> The byte order set in `ip-config.json` does not match the ECAN currently fitted to that kart.
>
> The kart appears blank in the Telemetry Dashboard on port 4200, because it has no named signals to show. It is not silent though. Go into InfluxDB's own UI on port 8086 and its measurements are almost all called `xFrameIdUnknown_{id}` with an ID that means nothing. The receiver is reading the frames with the opposite endianness to the one the device sends, so the frame IDs come out wrong, nothing in the DBC matches them, and the signals cannot be named.
>
> Byte order belongs to the ECAN rather than to the kart, so there are two ways to arrive here. A kart new to the fleet has never had its ECAN confirmed either way round. A kart that was collecting data perfectly well and is now blank has most likely had its ECAN swapped, after a failure or otherwise, and the replacement sends the other way round. The fix is the same either way. See [Fixing The Byte Order For An ECAN](#fixing-the-byte-order-for-an-ecan).

## Fixing The Byte Order For An ECAN

This one is fixed at the track, in `ip-config.json`, and it needs no deployment and no restart of the stack. Whoever is on the machine can do it: the operator at the track, remote support, or a developer over [TeamViewer](/software/developer/estate/systems/telemetry/track-operations#remote-access).

> [!NOTE]
>
> On a kart with a DAQ fitted there is no ECAN, because the DAQ replaces it. The setting still governs how the receiver reads the frames, so it has to match the DAQ instead. Everything below applies either way.

**1. Confirm it is that connection and not the fleet**

A kart blank in the Telemetry Dashboard has either stopped talking or is talking nonsense, so go to InfluxDB's own UI on port 8086 and look at what that kart is actually writing. If it is producing `xFrameIdUnknown_` measurements while the rest of the fleet is normal, it is that connection's configuration. Byte order is set per connection and belongs to the device on the other end of it, so what the rest of the fleet is set to is not evidence of what this device needs.

**2. Flip isLittleEndian**

Open `ip-config.json` in the receiver's `config` folder, find the connection for that kart, and invert `isLittleEndian`. Change nothing else. Where a kart has more than one connection, each has its own ECAN and its own copy of the field, so flip only the one producing the rubbish.

**3. Let the receiver restart itself**

The receiver polls the file every five seconds. It sees the change, validates it, stops itself, and Docker brings it straight back up on the new configuration. There is nothing to run. See [Changing It While Running](/software/developer/estate/systems/telemetry/track-operations#changing-it-while-running).

**4. Check that data is now being collected**

Within a few seconds of the receiver coming back, that kart should be writing named measurements in InfluxDB and no new `xFrameIdUnknown_` ones, and it should start populating in the Telemetry Dashboard. If it has gone quiet instead of correcting itself, read the receiver's logs. An edit that fails validation is logged and ignored, and the receiver carries on with the previous configuration.

To confirm the receiver actually read your edit, check the dated `ip-config` log alongside the receiver's own, which records the full contents of the file every time it is read. See [Configuration Reload](/software/developer/estate/systems/telemetry/receiver#configuration-reload).

> [!NOTE]
>
> The rubbish already written stays in InfluxDB until the day is stopped and the database is cleared. Judge the fix on whether correct data is arriving now, not on the old measurements disappearing.

> [!WARNING]
>
> The correct setting is a fact about the device, but `ip-config.json` records it against the kart, and the file is excluded from source control. So the correction lives only on that machine, and it is silently wrong again the next time that kart's device is swapped. Record it against the device rather than the kart.

## Known Issues

* The zip checksum depends on the combined logging table and the binary file, but not on any metadata such as hub region or run number.
* Run number logic is awkward and hard to insert manually.
* Large files, such as those generated on the Dyno, can cause a conversion Lambda invocation to run out of memory. This does not show as an error, because the process is simply killed.
* STL still shows UTC time. Which fields should be in local time has not been decided.
* Aentron karts are not officially supported in production. The `aentron` battery spec is implemented and the karts stream, but the full factory test has been outstanding since September 2025, so any `aentron` data should be treated as unverified. Scheduling and running that test is the thing that closes this.

## Related Documentation

| Page | Relevance |
| - | - |
| [DAQ Upload and Conversion](/software/developer/estate/systems/telemetry/daq-pipeline) | What is supposed to happen |
| [Timing and Lap Splitting](/software/developer/estate/systems/telemetry/timing-and-lap-splitting) | Reconversion, and why a run may have no laps |
| [The Charging Data Receiver](/software/developer/estate/systems/telemetry/receiver) | Frame layout, decoding and configuration reload |
| [Internal Signals](/software/developer/estate/systems/telemetry/internal-signals) | What the `x` prefixed signals mean |
| [Local Development](/software/developer/estate/systems/telemetry/local-development) | Reproducing a conversion, and simulating a kart |
| [Track Operations](/software/developer/estate/systems/telemetry/track-operations) | The machine, its logs, its dump folder and `ip-config.json` |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.