> ## Documentation Index
> Fetch the complete documentation index at: https://playbook.smedleygroup.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Track Operations

> What runs on the machine at the track, how it is started and stopped, what uploads happen in the background, and where a person has to step in.

Telemetry is the only part of the estate that is not cloud hosted. It runs as a Docker stack on a local machine at the track, on the same network as the karts, so the machine itself has to be there and on that network. Driving it is a different matter: someone at the track normally runs the scripts, but the machine can also be reached remotely.

There is **one machine per hub region**, configured for that region once at setup. Its `HUBREGION` is persisted as a machine level environment variable and tags the logs and backups it produces, so region is baked into the machine rather than chosen per event.

This page covers the machine at the track. For what happens to the data once it reaches S3, see [DAQ Upload and Conversion](/software/developer/estate/systems/telemetry/daq-pipeline) and [Timing and Lap Splitting](/software/developer/estate/systems/telemetry/timing-and-lap-splitting).

## Remote Access

The track machines are reachable over **TeamViewer**, which is how you run a script or read a log without being at the track.

> [!NOTE]
>
> Credentials are held by Sammy Bryan (`sammy.bryan@smedleygroup.com`) and Holly Single (`holly.single@smedleygroup.com`). Ask either of them when you need access.

## The Event Cycle

The stack is started and stopped every day. Once per event the setup script is run instead, which is how the machine picks up the latest scripts and container images.

| When | Run | What it does |
| - | - | - |
| Once per event, before it starts | `setup_application.bat` | Pulls the latest distribution and container images, then starts the stack itself |
| Every other morning | `start_application.bat` | Brings the stack up on what is already on the machine |
| End of every day | `stop_application.bat` | Backs up to S3, then clears InfluxDB and the logs for the next day |

> [!NOTE]
>
> Setup does not have to happen on the morning of the first day. Running it the night before, or anywhere with a decent connection, is preferable. Once it has run, `start_application.bat` is all that is needed for the rest of the weekend.

During the day, charging data streams in continuously, DAQ runs upload per run, and logs sync to S3 every 30 minutes.

### Network Limitations

Tracks frequently have poor internet, and several routine operations move a lot of data over it:

* **Pulling container images** during setup. The images are large, and over a weak connection this can take a long time. This is the main reason to run setup ahead of the event rather than on the morning.
* **The end of day backup upload**, which sends the InfluxDB, NATS and log archive to S3.
* **Charging exports**, which sync the exported STL files up to S3.

> [!NOTE]
>
> These are known limitations and they are bound by the connection at the track, not by the application. If one of them is slow, it is usually the network rather than a fault to investigate.

> [!WARNING]
>
> Because InfluxDB is cleared at the end of each day, the machine only ever holds the current day's charging data. Anything from an earlier day exists solely in that day's backup archive in S3, and getting it back means restoring the archive.

## What Runs At The Track

Everything is brought up by a single Docker Compose stack.

| Container | Role |
| - | - |
| `receiver` | Accepts TCP connections from the karts and publishes onto the raw NATS subject |
| `raw_to_telemetry` | Converts raw messages into telemetry messages. Runs five replicas |
| `telemetry_to_influx` | Writes telemetry messages into InfluxDB. Runs five replicas |
| `raw_to_file` | A debugging capture. Always consumes the raw stream, but writes nothing unless a connection has `saveRawData` set, which is off everywhere by default |
| `nats1`, `nats2`, `nats3` | A three node NATS JetStream cluster carrying all messages between the services |
| `influxdb` | Time series store for charging data, exposed on port 8086, which also serves InfluxDB's own UI |
| `exporter_api` | Flask API on port 5432. Receives DAQ uploads, queries InfluxDB, syncs to S3. See [Exporter API](/software/developer/estate/systems/telemetry/exporter-api) |
| `telemetry_ui` | The dashboard, on port 4200. See [Telemetry Dashboard](/software/developer/estate/systems/telemetry/dashboard) |
| `telemetry_cron` | Scheduled background jobs, currently the log sync |

> [!NOTE]
>
> The two highest throughput services, `raw_to_telemetry` and `telemetry_to_influx`, are explicitly scaled to five replicas at startup. Starting the stack with a plain `docker compose up` gives one of each and will not keep up.

For what those containers actually do, frame by frame, see [The Charging Data Receiver](/software/developer/estate/systems/telemetry/receiver).

## The Kart Fleet Configuration

The `receiver` container does not discover karts. It reads them from `ip-config.json`, in the receiver's `config` folder, which is mounted into the container. That file defines every kart the machine will connect to, and it is the single most important piece of local configuration on the machine.

> [!WARNING]
>
> `ip-config.json` is deliberately excluded from source control, so it is not in the repository and not in the distribution published to S3. A newly set up machine will not have one. It has to be written by hand or copied from an existing machine.

> [!NOTE]
>
> A complete working file is kept in the source of these docs, as the JSON at `images/software/developer/estate/systems/telemetry/ip-config.json`. It is a simulator configuration rather than a track one, so its addresses and kart IDs are wrong for any real fleet, but it is a far better starting point than an empty file: it covers both battery specs, both `txChunkSize` values, the single and multi connection cases, and the full request frame set for each spec. For an annotated extract of it, see [The Configuration To Point It At](/software/developer/estate/systems/telemetry/local-development#the-configuration-to-point-it-at).

### What It Contains

A list of karts, each with its connections, plus the CAN request frames used to poll them:

| Field | Meaning |
| - | - |
| `kartId` | The kart's identifier, used throughout the pipeline |
| `type` | The kart type |
| `batterySpec` | The battery specification, which also selects the DBC |
| `category` | The kart's category, `BB`, `CAD` or `JNR`, which is also what determines how many battery nodes it has |
| `connections` | One entry per battery node the kart exposes |

Each connection carries `ipAddress` and `port`, `isLittleEndian`, `isDataLoggerAttached`, `txChunkSize`, `saveRawData`, `isActive`, and `fullRequestDatasetsPerSecond`, which sets the polling rate and is 2 everywhere.

`isActive` is optional and defaults to true, so a connection with no `isActive` at all is an active one. Set it to false to park a connection the receiver should skip, which is how a kart is taken out of the fleet without deleting its entry.

> [!NOTE]
>
> `saveRawData` is a debugging switch, not a production setting. It is `false` on every connection in the shipped configuration, and turning it on makes `raw_to_file` write that connection's frames to local disk. Nothing downstream reads them. Turn it on for one kart while you are investigating and turn it back off, and be aware the capture is deleted with the logs when the day is stopped with a backup. See [The Raw File](/software/developer/estate/systems/telemetry/receiver#the-raw-file).

Two of those change what the receiver does rather than only what it records. `isDataLoggerAttached` decides whether the receiver polls the kart's battery nodes itself or only listens, and `isLittleEndian` has to match the device. See [Why The Receiver Sends Requests](/software/developer/estate/systems/telemetry/receiver#why-the-receiver-sends-requests).

> [!WARNING]
>
> `isLittleEndian` has to match the byte order the device actually sends, and nothing works it out for you. It is a property of the **device attached to the kart**, the ECAN or the DAQ where one is fitted, rather than of the kart, so a value that is right today can be wrong tomorrow: swap a failed ECAN for a replacement and the new one may send the other way round even though nothing about the kart has changed. Set it the wrong way round and the kart still connects and still streams, but the frames decode to nonsense and InfluxDB fills with `xFrameIdUnknown_{id}` measurements for that kart instead of named signals, so the kart shows up blank in the dashboard. See [Fixing The Byte Order For An ECAN](/software/developer/estate/systems/telemetry/troubleshooting#fixing-the-byte-order-for-an-ecan).

> [!WARNING]
>
> Setting `batterySpec` to `aentron` puts that kart on a spec that has never had its factory test, so it is not officially supported in production. The configuration validates and the kart streams, so nothing here will stop you. See [Aentron karts are not officially supported](/software/developer/estate/systems/telemetry#how-it-works).

> [!NOTE]
>
> The DBC is not configured here. It is derived from the battery spec and the connection index. Where a battery spec exposes a DBC per node this gives `{batterySpec}_node{index}.dbc`, and otherwise `{batterySpec}.dbc`. The DBC files do ship in the distribution.

### Validation

The receiver validates the file at startup and refuses to start if it fails. The rules are:

* The kart list cannot be empty
* Kart IDs must be unique
* Every IP and port combination must be unique **across all karts**, not just within one
* Each connection needs a valid IP address, a port between 1 and 65535, and a valid combination of type, battery spec and category

### Changing It While Running

The receiver polls the file every five seconds. If it has been modified, the new content differs, and it validates, the receiver **stops itself**, and Docker's restart policy brings it straight back up on the new configuration. See [Configuration Reload](/software/developer/estate/systems/telemetry/receiver#configuration-reload) for every outcome, including where the file it read is logged.

> [!NOTE]
>
> If an edit fails validation the receiver does not restart. It logs the validation errors and carries on with the previous configuration. So if you change a kart and nothing happens within a few seconds, read the receiver's logs rather than assuming the change took.

## Setting Up A Machine For A New Region

A new region needs its own machine, set up once. After that the machine follows the [event cycle](#the-event-cycle) like any other.

**Prerequisites on the machine:**

* **Windows.** The scripts are batch files, and the region is persisted with `setx`.
* **Docker**, including Compose.
* **AWS CLI**, installed and configured. The scripts default to the `default` profile, and it needs to read the telemetry bucket and pull from ECR.
* **Credentials in the user's `.aws` directory.** The exporter API and cron containers mount it read only, so a profile that only exists as environment variables is not enough.
* **Network placement where the karts can reach it** over TCP.
* **The region already present in `build_application.bat`.** It offers a fixed numbered menu of regions. A genuinely new region means adding it to that script and letting the pipeline republish the distribution first.
* **An `ip-config.json` for this region's fleet.** It is not in source control, so it does not arrive with the distribution.

**1. Copy the setup script across by hand**

`setup_application.bat` has to arrive manually, because it is the thing that fetches everything else. Copy it from the repository or from an existing machine. It and `ip-config.json` are the only two files that do not come down from S3.

**2. Run it**

It checks the AWS CLI is present, then syncs the distribution folder down from the telemetry bucket into the local directory. The bucket is hard coded to the production one, and the sync uses the `default` profile.

**3. Choose the region**

`setup_application.bat` then calls `build_application.bat`, which asks which region this machine is. The answer is written to the machine with `setx HUBREGION`, so it persists across reboots. The question is asked again every time the setup script runs, which is once per event, so answer it with the same region each time.

**4. Images are pulled and the stack starts**

The script logs in to ECR, pulls the images for every service, and then calls `start_application.bat` itself. The machine comes up running.

**5. Put the kart fleet configuration in place**

The distribution does not include `ip-config.json`, so the receiver has nothing to connect to until one is added. Write it for this region's fleet, or copy one from another machine and edit the addresses. See [The Kart Fleet Configuration](#the-kart-fleet-configuration).

> [!NOTE]
>
> Despite its name, `build_application.bat` does not build anything. Images are built by the pipeline and pushed to ECR; the track machine only pulls them.

## Getting An Update Onto The Machine

The track machine does not build from source control, and it does not update itself. The pipeline publishes the `distribution/` folder to the telemetry bucket with `--delete`, and the machine picks it up whenever `setup_application.bat` is next run, which in normal operation is once per event. See [Deployment And The Pipeline](/software/developer/estate/systems/telemetry/deployment) for what the pipeline is and what else it does.

> [!WARNING]
>
> Deploying a change to a track is therefore two separate acts: merging so the pipeline publishes, and then the setup script running at that track. Each region picks a change up separately, whenever that region next runs setup, so the same change can be live at one track and not another for weeks.

### Getting An Urgent Change Out Mid Event

If something has to land before the next event's setup, it can be done. Connect over [TeamViewer](#remote-access) and run `setup_application.bat` yourself, or ask the people at the track to run it.

Pick the moment: it restarts the stack and pulls images, so do it during a break or in the evening rather than mid session, and expect the pull to take a while on a track connection.

## Starting

`start_application.bat` brings the stack up with the replica counts set and forces recreation of the containers. It is run directly on the mornings after the first, and is also called for you at the end of `setup_application.bat`:

```
docker compose -f docker-compose.yml -f docker-compose.ecr.yml up -d --scale telemetry_to_influx=5 --scale raw_to_telemetry=5 --force-recreate
```

## What Happens While It Runs

Three things happen in the background without anyone driving them.

**Charging data streams in continuously.** Karts connect to `receiver` over TCP, which publishes onto the raw NATS subject. Two consumers read that subject in parallel. `raw_to_telemetry` converts the messages and republishes them onto the telemetry subject, where `telemetry_to_influx` picks them up and writes them to InfluxDB for the dashboard to display, and that is the whole of the production path. `raw_to_file` also reads the subject, but it is a debugging capture that writes nothing unless someone has turned `saveRawData` on for a connection, so in normal running it consumes and discards. See [The Raw File](/software/developer/estate/systems/telemetry/receiver#the-raw-file).

**DAQ runs are uploaded per run.** A DAQ unit posts a completed run to the exporter API's `/upload` endpoint, which zips it with its logging table and metadata, works out the run number from local state, and uploads it to S3 to be converted. A file that fails validation, or an upload that is interrupted, stays on the machine rather than going to S3, and if the full file was received before the failure the binary is kept locally. See [DAQ Upload and Conversion](/software/developer/estate/systems/telemetry/daq-pipeline) for the detail.

**Logs sync every 30 minutes.** The `telemetry_cron` container runs one job:

```
*/30 * * * * /bin/bash /app/scripts/sync_logs_s3.sh
```

It uploads today's log files, plus the exporter API log, to `s3://telemetry-data-sg-{stage}/live-logs/{hub region}/{date}/`. Together with the uploads themselves and the Teams notifications the exporter API sends on a sync, this is how the track is visible from outside it, and the logs are what you read first when a run does not appear.

## Stopping

`stop_application.bat` is run at the end of each day of an event. It is interactive and asks two questions before doing anything. In normal operation both are answered yes, so each day ends with a backup in S3 and the next day starts with an empty database.

**1. Complete the backup procedure?**

If yes, it backs up three things while the services are still running: InfluxDB via a live `influx backup`, the whole NATS cluster directory, and the logs, including the exporter API container logs. All three go into one archive named `{region}_telemetry_backup_{timestamp}.tar.gz`, written to the local backup directory and then uploaded to `s3://telemetry-data-sg-{stage}/backups/`.

**2. Start with a fresh InfluxDB?**

Only offered if the backup was selected. If yes, the InfluxDB data directory, configuration database and write ahead log are removed after the backup completes, so the next start begins empty.

**3. Services stop**

The script runs `docker compose stop` followed by `docker compose down`.

Logs are cleared automatically whenever the backup ran, on the same condition as InfluxDB.

> [!WARNING]
>
> Neither InfluxDB nor the logs are cleared unless the archive reached S3. A backup that was written locally but failed to upload leaves the data in place deliberately. If you are expecting a fresh database on the next start and do not get one, the S3 upload is the first thing to check.

## Where Data Ends Up

| Data | Destination |
| - | - |
| Charging telemetry | InfluxDB on the track machine for the current day only, captured in that day's stop archive |
| Raw message stream | Nowhere by default. Local disk on the track machine only where `saveRawData` is set for a connection, and cleared with the logs when the day is stopped with a backup |
| DAQ run zips | `daq-data/` in the telemetry bucket, which triggers conversion |
| Logs | `live-logs/{hub region}/{date}/`, every 30 minutes |
| InfluxDB, NATS and logs archive | `backups/`, on stopping with backup selected |
| Charging runs exported from the dashboard | `STL-database/` in the telemetry bucket, for engineers to review in SimTelLab |
| Files that failed to upload | The dump folder on the track machine only |

## Where A Person Has To Step In

Most of the pipeline is automatic, but several failure modes have no automatic recovery and are only found by someone looking. These are the points where a human in the loop step belongs.

* **A kart streaming rubbish signals.** A byte order that does not match the ECAN fitted to the kart is not a failure, so nothing logs it and nothing alerts on it. It is found only by someone noticing a kart sitting blank in the dashboard and then seeing in InfluxDB that its measurements are all called `xFrameIdUnknown_{id}`. Flipping `isLittleEndian` for that connection and letting the receiver restart itself corrects it. Suspect it on any kart new to the fleet, and on any kart that was fine until its ECAN was changed. See [Fixing The Byte Order For An ECAN](/software/developer/estate/systems/telemetry/troubleshooting#fixing-the-byte-order-for-an-ecan).
* **A charging run that failed to convert at the track.** The data is no longer on the machine once the day has ended, so pull that day's archive from `backups/`, restore the InfluxDB data locally, investigate what went wrong, and reconvert from the restored copy.
* **A run with no matching timing.** It converts as a single unsplit lap and never appears against a competitor. Check the transponder ID, hub region and time window in the metadata against the timing.
* **A conversion that failed outright.** Nothing retries unless one of the two checksums changes, so a failure stays a failure until someone forces a reconversion. See [Triggering A Reconversion Deliberately](/software/developer/estate/systems/telemetry/timing-and-lap-splitting#triggering-a-reconversion-deliberately) for the three ways to do that, and note that only a re-upload can force one.
* **A run that never arrived.** Read the sync'd logs for the upload failure, find the file in the dump folder on the track machine, and re-upload it. Run numbering depends on local state, so re-uploading from the track machine is preferable to doing it by hand.
* **Laps that were padded or NaN filled.** Only visible as warnings in the converter logs. A lap can look complete in the output and be substantially interpolated.

## Key Files

| File | Role |
| - | - |
| `distribution/setup_application.bat` | Syncs the distribution down from S3, then hands off to the build script |
| `distribution/build_application.bat` | Region menu, ECR login, image pull, then starts the stack |
| `distribution/start_application.bat` | Brings the stack up with the replica counts |
| `distribution/stop_application.bat` | Interactive backup, clear and stop |
| `distribution/config.bat` | Loads `.env` and applies defaults |
| `distribution/docker-compose.yml` | The full container stack |
| `telemetry/telemetry-cron/` | The scheduled job container |
| `distribution/telemetry/telemetry-cron/scripts/sync_logs_s3.sh` | The 30 minute log sync |
| `telemetry/telemetry-api/src/api.py` | The exporter API, including the DAQ upload endpoint |

## Related Documentation

| Page | Relevance |
| - | - |
| [DAQ Upload and Conversion](/software/developer/estate/systems/telemetry/daq-pipeline) | What happens to a run once it reaches S3 |
| [The Charging Data Receiver](/software/developer/estate/systems/telemetry/receiver) | How the receiver polls, decodes and writes, and its internal signals |
| [Telemetry Dashboard](/software/developer/estate/systems/telemetry/dashboard) | The application an operator actually uses on this machine |
| [Exporter API](/software/developer/estate/systems/telemetry/exporter-api) | The local API, its endpoints and its logs |
| [Deployment And The Pipeline](/software/developer/estate/systems/telemetry/deployment) | How the images and the distribution are built and published |
| [Troubleshooting](/software/developer/estate/systems/telemetry/troubleshooting) | Runs that did not arrive, convert, or appear |
| [Timing and Lap Splitting](/software/developer/estate/systems/telemetry/timing-and-lap-splitting) | How timing turns a run into laps |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.