Skip to main content
Telemetry is the only part of the estate that is not cloud hosted. It runs as a Docker stack on a local machine at the track, on the same network as the karts, so the machine itself has to be there and on that network. Driving it is a different matter: someone at the track normally runs the scripts, but the machine can also be reached remotely. There is one machine per hub region, configured for that region once at setup. Its HUBREGION is persisted as a machine level environment variable and tags the logs and backups it produces, so region is baked into the machine rather than chosen per event. This page covers the machine at the track. For what happens to the data once it reaches S3, see DAQ Upload and Conversion and Timing and Lap Splitting.

​
Remote Access

The track machines are reachable over TeamViewer, which is how you run a script or read a log without being at the track.
Credentials are held by Sammy Bryan (sammy.bryan@smedleygroup.com) and Holly Single (holly.single@smedleygroup.com). Ask either of them when you need access.

​
The Event Cycle

The stack is started and stopped every day. Once per event the setup script is run instead, which is how the machine picks up the latest scripts and container images.
Setup does not have to happen on the morning of the first day. Running it the night before, or anywhere with a decent connection, is preferable. Once it has run, start_application.bat is all that is needed for the rest of the weekend.
During the day, charging data streams in continuously, DAQ runs upload per run, and logs sync to S3 every 30 minutes.

​
Network Limitations

Tracks frequently have poor internet, and several routine operations move a lot of data over it:
  • Pulling container images during setup. The images are large, and over a weak connection this can take a long time. This is the main reason to run setup ahead of the event rather than on the morning.
  • The end of day backup upload, which sends the InfluxDB, NATS and log archive to S3.
  • Charging exports, which sync the exported STL files up to S3.
These are known limitations and they are bound by the connection at the track, not by the application. If one of them is slow, it is usually the network rather than a fault to investigate.
Because InfluxDB is cleared at the end of each day, the machine only ever holds the current day’s charging data. Anything from an earlier day exists solely in that day’s backup archive in S3, and getting it back means restoring the archive.

​
What Runs At The Track

Everything is brought up by a single Docker Compose stack.
The two highest throughput services, raw_to_telemetry and telemetry_to_influx, are explicitly scaled to five replicas at startup. Starting the stack with a plain docker compose up gives one of each and will not keep up.
For what those containers actually do, frame by frame, see The Charging Data Receiver.

​
The Kart Fleet Configuration

The receiver container does not discover karts. It reads them from ip-config.json, in the receiver’s config folder, which is mounted into the container. That file defines every kart the machine will connect to, and it is the single most important piece of local configuration on the machine.
ip-config.json is deliberately excluded from source control, so it is not in the repository and not in the distribution published to S3. A newly set up machine will not have one. It has to be written by hand or copied from an existing machine.
A complete working file is kept in the source of these docs, as the JSON at images/software/developer/estate/systems/telemetry/ip-config.json. It is a simulator configuration rather than a track one, so its addresses and kart IDs are wrong for any real fleet, but it is a far better starting point than an empty file: it covers both battery specs, both txChunkSize values, the single and multi connection cases, and the full request frame set for each spec. For an annotated extract of it, see The Configuration To Point It At.

​
What It Contains

A list of karts, each with its connections, plus the CAN request frames used to poll them: Each connection carries ipAddress and port, isLittleEndian, isDataLoggerAttached, txChunkSize, saveRawData, isActive, and fullRequestDatasetsPerSecond, which sets the polling rate and is 2 everywhere. isActive is optional and defaults to true, so a connection with no isActive at all is an active one. Set it to false to park a connection the receiver should skip, which is how a kart is taken out of the fleet without deleting its entry.
saveRawData is a debugging switch, not a production setting. It is false on every connection in the shipped configuration, and turning it on makes raw_to_file write that connection’s frames to local disk. Nothing downstream reads them. Turn it on for one kart while you are investigating and turn it back off, and be aware the capture is deleted with the logs when the day is stopped with a backup. See The Raw File.
Two of those change what the receiver does rather than only what it records. isDataLoggerAttached decides whether the receiver polls the kart’s battery nodes itself or only listens, and isLittleEndian has to match the device. See Why The Receiver Sends Requests.
isLittleEndian has to match the byte order the device actually sends, and nothing works it out for you. It is a property of the device attached to the kart, the ECAN or the DAQ where one is fitted, rather than of the kart, so a value that is right today can be wrong tomorrow: swap a failed ECAN for a replacement and the new one may send the other way round even though nothing about the kart has changed. Set it the wrong way round and the kart still connects and still streams, but the frames decode to nonsense and InfluxDB fills with xFrameIdUnknown_{id} measurements for that kart instead of named signals, so the kart shows up blank in the dashboard. See Fixing The Byte Order For An ECAN.
Setting batterySpec to aentron puts that kart on a spec that has never had its factory test, so it is not officially supported in production. The configuration validates and the kart streams, so nothing here will stop you. See Aentron karts are not officially supported.
The DBC is not configured here. It is derived from the battery spec and the connection index. Where a battery spec exposes a DBC per node this gives {batterySpec}_node{index}.dbc, and otherwise {batterySpec}.dbc. The DBC files do ship in the distribution.

​
Validation

The receiver validates the file at startup and refuses to start if it fails. The rules are:
  • The kart list cannot be empty
  • Kart IDs must be unique
  • Every IP and port combination must be unique across all karts, not just within one
  • Each connection needs a valid IP address, a port between 1 and 65535, and a valid combination of type, battery spec and category

​
Changing It While Running

The receiver polls the file every five seconds. If it has been modified, the new content differs, and it validates, the receiver stops itself, and Docker’s restart policy brings it straight back up on the new configuration. See Configuration Reload for every outcome, including where the file it read is logged.
If an edit fails validation the receiver does not restart. It logs the validation errors and carries on with the previous configuration. So if you change a kart and nothing happens within a few seconds, read the receiver’s logs rather than assuming the change took.

​
Setting Up A Machine For A New Region

A new region needs its own machine, set up once. After that the machine follows the event cycle like any other. Prerequisites on the machine:
  • Windows. The scripts are batch files, and the region is persisted with setx.
  • Docker, including Compose.
  • AWS CLI, installed and configured. The scripts default to the default profile, and it needs to read the telemetry bucket and pull from ECR.
  • Credentials in the user’s .aws directory. The exporter API and cron containers mount it read only, so a profile that only exists as environment variables is not enough.
  • Network placement where the karts can reach it over TCP.
  • The region already present in build_application.bat. It offers a fixed numbered menu of regions. A genuinely new region means adding it to that script and letting the pipeline republish the distribution first.
  • An ip-config.json for this region’s fleet. It is not in source control, so it does not arrive with the distribution.
1

Copy the setup script across by hand

setup_application.bat has to arrive manually, because it is the thing that fetches everything else. Copy it from the repository or from an existing machine. It and ip-config.json are the only two files that do not come down from S3.
2

Run it

It checks the AWS CLI is present, then syncs the distribution folder down from the telemetry bucket into the local directory. The bucket is hard coded to the production one, and the sync uses the default profile.
3

Choose the region

setup_application.bat then calls build_application.bat, which asks which region this machine is. The answer is written to the machine with setx HUBREGION, so it persists across reboots. The question is asked again every time the setup script runs, which is once per event, so answer it with the same region each time.
4

Images are pulled and the stack starts

The script logs in to ECR, pulls the images for every service, and then calls start_application.bat itself. The machine comes up running.
5

Put the kart fleet configuration in place

The distribution does not include ip-config.json, so the receiver has nothing to connect to until one is added. Write it for this region’s fleet, or copy one from another machine and edit the addresses. See The Kart Fleet Configuration.
Despite its name, build_application.bat does not build anything. Images are built by the pipeline and pushed to ECR; the track machine only pulls them.

​
Getting An Update Onto The Machine

The track machine does not build from source control, and it does not update itself. The pipeline publishes the distribution/ folder to the telemetry bucket with --delete, and the machine picks it up whenever setup_application.bat is next run, which in normal operation is once per event. See Deployment And The Pipeline for what the pipeline is and what else it does.
Deploying a change to a track is therefore two separate acts: merging so the pipeline publishes, and then the setup script running at that track. Each region picks a change up separately, whenever that region next runs setup, so the same change can be live at one track and not another for weeks.

​
Getting An Urgent Change Out Mid Event

If something has to land before the next event’s setup, it can be done. Connect over TeamViewer and run setup_application.bat yourself, or ask the people at the track to run it. Pick the moment: it restarts the stack and pulls images, so do it during a break or in the evening rather than mid session, and expect the pull to take a while on a track connection.

​
Starting

start_application.bat brings the stack up with the replica counts set and forces recreation of the containers. It is run directly on the mornings after the first, and is also called for you at the end of setup_application.bat:

​
What Happens While It Runs

Three things happen in the background without anyone driving them. Charging data streams in continuously. Karts connect to receiver over TCP, which publishes onto the raw NATS subject. Two consumers read that subject in parallel. raw_to_telemetry converts the messages and republishes them onto the telemetry subject, where telemetry_to_influx picks them up and writes them to InfluxDB for the dashboard to display, and that is the whole of the production path. raw_to_file also reads the subject, but it is a debugging capture that writes nothing unless someone has turned saveRawData on for a connection, so in normal running it consumes and discards. See The Raw File. DAQ runs are uploaded per run. A DAQ unit posts a completed run to the exporter API’s /upload endpoint, which zips it with its logging table and metadata, works out the run number from local state, and uploads it to S3 to be converted. A file that fails validation, or an upload that is interrupted, stays on the machine rather than going to S3, and if the full file was received before the failure the binary is kept locally. See DAQ Upload and Conversion for the detail. Logs sync every 30 minutes. The telemetry_cron container runs one job:
It uploads today’s log files, plus the exporter API log, to s3://telemetry-data-sg-{stage}/live-logs/{hub region}/{date}/. Together with the uploads themselves and the Teams notifications the exporter API sends on a sync, this is how the track is visible from outside it, and the logs are what you read first when a run does not appear.

​
Stopping

stop_application.bat is run at the end of each day of an event. It is interactive and asks two questions before doing anything. In normal operation both are answered yes, so each day ends with a backup in S3 and the next day starts with an empty database.
1

Complete the backup procedure?

If yes, it backs up three things while the services are still running: InfluxDB via a live influx backup, the whole NATS cluster directory, and the logs, including the exporter API container logs. All three go into one archive named {region}_telemetry_backup_{timestamp}.tar.gz, written to the local backup directory and then uploaded to s3://telemetry-data-sg-{stage}/backups/.
2

Start with a fresh InfluxDB?

Only offered if the backup was selected. If yes, the InfluxDB data directory, configuration database and write ahead log are removed after the backup completes, so the next start begins empty.
3

Services stop

The script runs docker compose stop followed by docker compose down.
Logs are cleared automatically whenever the backup ran, on the same condition as InfluxDB.
Neither InfluxDB nor the logs are cleared unless the archive reached S3. A backup that was written locally but failed to upload leaves the data in place deliberately. If you are expecting a fresh database on the next start and do not get one, the S3 upload is the first thing to check.

​
Where Data Ends Up

​
Where A Person Has To Step In

Most of the pipeline is automatic, but several failure modes have no automatic recovery and are only found by someone looking. These are the points where a human in the loop step belongs.
  • A kart streaming rubbish signals. A byte order that does not match the ECAN fitted to the kart is not a failure, so nothing logs it and nothing alerts on it. It is found only by someone noticing a kart sitting blank in the dashboard and then seeing in InfluxDB that its measurements are all called xFrameIdUnknown_{id}. Flipping isLittleEndian for that connection and letting the receiver restart itself corrects it. Suspect it on any kart new to the fleet, and on any kart that was fine until its ECAN was changed. See Fixing The Byte Order For An ECAN.
  • A charging run that failed to convert at the track. The data is no longer on the machine once the day has ended, so pull that day’s archive from backups/, restore the InfluxDB data locally, investigate what went wrong, and reconvert from the restored copy.
  • A run with no matching timing. It converts as a single unsplit lap and never appears against a competitor. Check the transponder ID, hub region and time window in the metadata against the timing.
  • A conversion that failed outright. Nothing retries unless one of the two checksums changes, so a failure stays a failure until someone forces a reconversion. See Triggering A Reconversion Deliberately for the three ways to do that, and note that only a re-upload can force one.
  • A run that never arrived. Read the sync’d logs for the upload failure, find the file in the dump folder on the track machine, and re-upload it. Run numbering depends on local state, so re-uploading from the track machine is preferable to doing it by hand.
  • Laps that were padded or NaN filled. Only visible as warnings in the converter logs. A lap can look complete in the output and be substantially interpolated.

​
Key Files

Last modified on September 9, 2026