> ## Documentation Index
> Fetch the complete documentation index at: https://playbook.smedleygroup.com/llms.txt
> Use this file to discover all available pages before exploring further.

# The Charging Data Receiver

> How the receiver polls the karts, what a frame looks like on the wire, how frames are decoded against the DBC and written to InfluxDB, and the internal signals the pipeline emits about itself.

The receiver is the only thing in the estate that talks to a kart directly. Everything on this page happens on the track machine, inside four of its containers, and all of it is driven by `ip-config.json`. See [The Kart Fleet Configuration](/software/developer/estate/systems/telemetry/track-operations#the-kart-fleet-configuration) for the file itself.

This page is the mechanism. Read it when you are working on the pipeline, or when you are staring at InfluxDB trying to work out what a kart is actually doing.

## The Shape Of The Pipeline

Four containers and two NATS subjects.

| Subject | Published by | Consumed by |
| - | - | - |
| `raw_data` | `receiver` | `raw_to_telemetry` and `raw_to_file`, independently of each other |
| `telemetry_data` | `raw_to_telemetry`, and `receiver` directly for its internal signals | `telemetry_to_influx` |

Both subjects live on a single JetStream stream called `telemetry`, file backed, holding **one hour** of history and capped at **1 GB**. Whichever limit is reached first starts discarding.

> [!NOTE]
>
> Each consuming service has its own durable name, and every replica of that service shares it. That shared name is what makes the five replicas of `raw_to_telemetry` split the work between them rather than each processing every message. It is also why `raw_to_file` sees the whole of `raw_data` independently, despite reading the same subject.

The receiver publishes its own internal signals straight onto `telemetry_data`, skipping the decode stage entirely. Internal signals are already named values, so there is nothing for the DBC to do.

> [!WARNING]
>
> A publish is retried five times, half a second apart, and then throws. Since the throw happens inside the connection's send or receive loop, a NATS outage lasting more than about two and a half seconds does not just lose messages: it tears the kart connection down and rebuilds it.

## Why The Receiver Sends Requests

Not everything on the kart volunteers its data, and that single fact is why the receiver transmits at all.

| Source | Behaviour |
| - | - |
| The controller | Emits its signals continuously and unprompted. For example, throttle and vehicle speed arrive whether anyone asks for them or not |
| The battery nodes | Emit nothing on their own. A node reports a value only when it is asked for it |

> [!NOTE]
>
> Neither is reached directly. A connection is to the **one device fitted on the kart**, an ECAN or a DAQ, which bridges the kart's CAN bus to TCP. Requests and responses both pass through it. See [What Is On A Kart](/software/developer/estate/systems/telemetry/architecture#what-is-on-a-kart).

The request frames held in `littleEndianTxData` are that asking. Each one is a request to a battery node for the values of the DBC signals we want from it, and the responses that come back are what the DBC then decodes into channels. Stop sending them and the controller signals keep arriving while the battery data simply stops.

### When Something Else Does The Asking

`isDataLoggerAttached` means "a device on this kart is already polling the battery nodes". It covers two situations:

* **A data logger fitted alongside the ECAN.** Both devices are present, and the logger does the polling.
* **A DAQ fitted.** There is no ECAN on the kart at all. The DAQ replaces the whole of the ECAN's functionality and adds the running telemetry on top, and it does the polling itself.

Either way the receiver must not add its own requests on top. It runs the consumer loop only, and reads what the attached device's polling has already put on the wire.

> [!NOTE]
>
> This is also why the receiver discards frames whose payload bytes are all zero when the flag is set. Those are the attached device's own requests going past rather than a node's answer, so they are counted as `xDataLoggerReqsPerConsumeCnt` and never sent for decoding.

### More Requests Than There Are Nodes

The receiver does not know how many battery nodes a kart has, or which node IDs they are using. Nothing in `ip-config.json` says, and nothing asks the kart first. So it requests every node it could have and lets the kart sort it out.

The DBC defines messages for **five** battery nodes, node 0 through node 4, and the request set covers all five. How many are actually fitted depends on the category:

| Category | Battery nodes |
| - | - |
| Bambino (`BB`) | 1 |
| Cadet (`CAD`) | 2 |
| Junior (`JNR`) | 3 |

> [!WARNING]
>
> Those are the values the configuration files actually use. The `KartCategory` enum declares its serialised value for cadet as `CC`, and `CAD` binds only because the deserialiser also matches the enum member name. Both parse today, but renaming that member would stop every real configuration from loading, and since an invalid file at startup prevents the container starting, that would take a track offline on its next start.

A node that is present answers. A node that is not present says nothing whatsoever. So on a junior the receiver asks all five nodes on every pass and gets replies from three of them, and the three are not necessarily nodes 0 to 2: the IDs present are whatever that kart's nodes are set to, so in practice it might be 1, 3 and 4.

> [!NOTE]
>
> Most of a bambino's request traffic is therefore unanswered by design, and that is not a fault. An unanswered request produces nothing at all, so it costs time on every pass but it does not appear as an `xFrameIdUnknown_` measurement. Absent nodes look like fewer response counters, never like rubbish.

## One Task Per Connection

The receiver does not discover karts and does not multiplex them. At startup it reads `ip-config.json`, keeps the karts with at least one active connection, and within each kart keeps the connections whose `isActive` is true. `isActive` is optional and defaults to true, so a connection that omits it is kept. Every surviving connection gets its own long running task, its own TCP socket and its own counters. Nothing is shared between connections.

Each connection then runs up to two loops against that one socket:

| Loop | Runs when | What it does |
| - | - | - |
| Producer | `isDataLoggerAttached` is `false` | Writes CAN request frames to the socket at the configured rate |
| Consumer | Always | Reads frames off the socket, filters them, and publishes them |

> [!NOTE]
>
> A connection with a data logger or a DAQ attached is **listen only**, for the reasons above. So `isDataLoggerAttached` changes what the receiver *does*, not merely what it records.

## Transmitting Requests

The request frames are not in the code. They live in `ip-config.json` under `littleEndianTxData`, keyed by battery spec, and they are held **little endian**. One pass through the whole set is a dataset, which is the unit `fullRequestDatasetsPerSecond` counts.

For a big endian connection the receiver converts them at startup: byte 0 is copied through untouched, bytes 1 to 4 are reversed, and bytes 5 to 12 are reversed. A wrong `isLittleEndian` therefore does not only garble what comes back, it also byte swaps the questions being asked.

Frames are then grouped into chunks of `txChunkSize` and each chunk is written to the socket as a single write, with a fixed delay between chunks:

```
delay between chunks (ms) = 1000 / (fullRequestDatasetsPerSecond * chunkCount) / 2
```

After each complete pass through the chunks, rate control compares the number of datasets sent against the time elapsed since that connection opened, and sleeps for the shortfall.

> [!NOTE]
>
> Because the comparison is against elapsed time since the connection opened, `fullRequestDatasetsPerSecond` is held as a long run average rather than enforced second by second. A connection that falls behind will run faster than the configured rate until it has caught up, and one that is rebuilt starts its averaging again from zero.

Every frame written is also recorded on the raw message, so the raw file on disk contains both directions of the conversation.

### The Rate, And Why It Is Low

`fullRequestDatasetsPerSecond` is **2** on every connection in every hub region, so battery signals are requested, and therefore collected, at **2 Hz**. That is the rate the rate controller holds, and requests are the only thing on the kart whose rate the receiver governs at all.

The controller is neither polled nor rate limited by anything here. It broadcasts on its own schedule, at **50 Hz**, and the receiver simply records what arrives. The two streams in InfluxDB therefore have very different densities: battery channels sparse, controller channels far denser.

> [!WARNING]
>
> Do not raise `fullRequestDatasetsPerSecond` casually. An earlier factory test at a higher request rate produced dropped frames, and the component that could not keep up was never identified. 2 Hz is a tested figure rather than an arbitrary default, and the failure it avoids is lost data rather than an error.

> [!NOTE]
>
> The charging export aggregates into one second windows and forward fills, so a 2 Hz battery channel is comfortably denser than the exported grid. See [Charging Export](/software/developer/estate/systems/telemetry/charging-export#what-the-export-does-to-the-data).

## Receiving Frames

Every frame on the wire is exactly **13 bytes**. The consumer loop reads exactly 13 bytes, adds the frame to the current message, and keeps going while the socket still has data available. One burst of available data becomes one published message.

| Bytes | Contents |
| - | - |
| 0 | A single byte header, left untouched by the big endian conversion |
| 1 to 4 | The frame ID, being the node ID and command. Read as a 32 bit integer, little or big endian according to `isLittleEndian` |
| 5 to 12 | The eight byte data payload |

Two frames are counted but deliberately not forwarded:

* **The attached device's own requests.** Where `isDataLoggerAttached` is true, a frame whose bytes from 5 onward are all zero is treated as the logger or DAQ polling rather than a node responding.
* **Frame ID 1793.** Dropped unconditionally.

Each burst produces two publishes: the surviving frames onto `raw_data`, and a KPI message of internal signals onto `telemetry_data`.

## Decoding

`raw_to_telemetry` consumes `raw_data`. For each received frame it extracts the frame ID from bytes 1 to 4, using the byte order carried on the message rather than re-reading configuration, and looks that ID up in the DBC for that connection. DBCs are parsed once and cached in memory per container.

* **Found.** The payload is read as a single 64 bit value from offset 5, and every signal the DBC message defines is unpacked from it. One frame becomes one sample per signal.
* **Not found.** One sample named `xFrameIdUnknown_{frameId}` with a value of 1, and the payload is discarded.

> [!NOTE]
>
> Byte order *within* the payload is a property of each signal in the DBC, so the DBC handles it. `isLittleEndian` governs only the frame ID and the request frames. That is the whole of its influence, and it is enough to make a device look silent.

The DBC is never named in configuration. It is derived from the battery spec and the connection's index. Where a battery spec exposes a DBC per node the file is `{batterySpec}_node{index}.dbc`, and otherwise it is `{batterySpec}.dbc`. The DBC files ship in the distribution.

> [!WARNING]
>
> If decoding throws, the consumer logs the error and **acknowledges the message anyway**. The frames in it are gone and will not be redelivered. A decode bug therefore shows up as quietly missing data rather than as a backlog.

## Configuration Reload

A background service polls `ip-config.json` every five seconds. It compares the file's last write time, and if it has moved, re-reads and re-validates the file.

| Outcome | What happens |
| - | - |
| File unchanged in content | Nothing. The new timestamp is accepted and polling continues |
| Changed and valid | The receiver logs it and **stops itself**. Docker's restart policy brings it back up on the new configuration |
| Changed and invalid | The validation errors are logged and the previous configuration keeps running |
| Unreadable or malformed JSON | The error is logged and the previous configuration keeps running |

Validation is all or nothing across the whole file: a mistake on one kart prevents the change to another kart in the same edit from taking effect.

> [!NOTE]
>
> Every time the file is read, its **entire contents** are written to a separate dated log, at `Logs/Container/FromTcp/Consumer/ToRaw/Producer/ip-config/{dd.MM.yyyy}.ip-config.log`. That gives a timestamped history of every version of the configuration the receiver has ever run, and those logs sync to S3 with the rest. It is the fastest way to answer what a kart was set to at the time, and whether an edit was actually picked up.

> [!WARNING]
>
> Validation at startup behaves differently from validation on reload. An invalid file at startup throws and the container will not start. An invalid edit while running is merely ignored. So a broken edit does not take a track offline, but a broken file does stop a cold start.

## Node Labels

Where a battery spec exposes a DBC per node and the kart has more than one connection, each connection is labelled `Node0`, `Node1` and so on by index. A kart with a single connection gets no label, and nor does one whose battery spec has a single DBC.

The label matters because it prefixes every internal signal name for that connection, giving `Node0_xTotalResCnt`. On a multi node kart the label is the only thing that tells you which node an internal signal came from.

> [!NOTE]
>
> The label and the DBC are decided separately. A single connection kart on a per node battery spec still reads the node 0 DBC, but carries no label, so its signals arrive unprefixed.

> [!NOTE]
>
> `xEndpoint` is the exception to the prefixing. Its label is appended rather than prefixed, giving `xEndpoint_Node0`. That matters for the InfluxDB write below.

## Writing To InfluxDB

`telemetry_to_influx` consumes `telemetry_data` into a bounded in memory channel, then batches. A batch is flushed when it reaches **5000 points** or **1000 ms** have passed, whichever comes first.

One sample becomes one point:

| Point element | Value |
| - | - |
| Measurement | The sample name, with the node label and its underscore stripped off when the name starts with `Node` |
| Tag `kart` | The kart ID |
| Tag `battery` | The node label, and only present when the name started with `Node` |
| Field `value` | The sample value, always a double |
| Timestamp | Nanosecond precision, from when the frame was received |

> [!NOTE]
>
> This is why the same signal from two nodes of one kart shares a measurement name and is separated only by the `battery` tag, and it is why the STL export prefixes the battery name onto its columns. It is also why `xEndpoint_Node0`, whose label is a suffix, stays a measurement of its own with no `battery` tag.

> [!WARNING]
>
> A failed InfluxDB write is logged and the batch is dropped. Nothing retries it, and nothing surfaces it beyond the container log.

## The Raw File

> [!WARNING]
>
> `raw_to_file` is a debugging aid and is not part of the production data path. Nothing downstream reads what it writes, and the dashboard, the charging export and the DAQ conversion all work without it. It is off for every kart in the shipped configuration and should be left that way unless you are actively investigating something.

When it is on, both sent and received frames are written, which makes it the only record of what was actually asked as well as answered.

| Detail | Value |
| - | - |
| Path in the container | `rawData/{date}/{guid}/rawData_{timestamp}.bin` |
| Path on the track machine | `Logs/Container/FromRaw/Consumer/ToFile/Leaf/rawData/` |
| Format | Length prefixed protobuf, one `RawDataMessages` record at a time |
| Rolls every | `RAW_LINES_PER_FILE` messages, which defaults to 1000 and is not set in Compose |

> [!NOTE]
>
> A rolled file is zipped and its `.bin` deleted, so a finished file on disk is `rawData_{timestamp}.zip`. Only the file currently open is a `.bin`, and it stays a `.bin` if the container is killed rather than stopped cleanly.

> [!WARNING]
>
> A dated directory and an empty `.bin` are created every time the container starts, whether or not any connection has `saveRawData` set. Empty `rawData` folders are normal and are not evidence that raw capture is on.

Reading one back is an offline job. `ConvertRawDataLogToCsv` in the solution takes a folder of those zips and writes every record into a single CSV. Nothing ships raw files off the machine on its own, because the log sync only matches names carrying today's date in `dd.mm.yyyy` form and the raw file names do not use it.

> [!WARNING]
>
> Raw files sit under `Logs/`, so they are copied into the day's stop archive and then deleted along with the rest of `Logs/` when the backup succeeds. Anything you captured is gone from the machine at the end of the day. Copy it off before stopping, or pull it back out of that day's backup in S3.

## Connecting And Reconnecting

Connection handling is a loop that never gives up, and it narrates itself entirely through internal signals.

**1. Attempt**

`xConnectionAttempt` is emitted for every attempt, successful or not.

**2. Success or failure**

Success adds `xNewConnection`. Failure adds `xTcpConnectionFailed` and `xPing`, the latter being an ICMP result for that connection's IP address, which is how you tell "unreachable" from "reachable but refusing".

**3. Run until something breaks**

The producer and consumer loops run against the socket. If either throws, it emits `xTcpConnectionKilled` and cancels its sibling, so both stop together.

**4. Rebuild**

The socket is disposed and the outer loop connects again from scratch. The container does not restart, and neither do the other karts.

> [!NOTE]
>
> A rebuilt connection gets a new set of counters, so `xTotalReqsCnt` and `xTotalResCnt` reset to zero. A cumulative counter that has gone backwards is a reconnection, not a fault in the counter.

## Key Files

All paths are under `telemetry/TelemetryOnGround/` in [can-wifi](https://github.com/Smedley-Group-Advanced-Technology/can-wifi).

| File | Role |
| - | - |
| `Container.FromTcp.Consumer.ToRaw.Producer/Services/WifiReceiver.cs` | The connection loops, transmit, receive and reconnection |
| `Container.FromTcp.Consumer.ToRaw.Producer/Services/ConfigurationService.cs` | Reading and validating `ip-config.json`, and the big endian conversion of the request frames |
| `Container.FromTcp.Consumer.ToRaw.Producer/Services/RateControlService.cs` | Holding the configured dataset rate |
| `Container.FromTcp.Consumer.ToRaw.Producer/Services/TelemetryCollectionService.cs` | Building the KPI messages of internal signals |
| `Container.FromTcp.Consumer.ToRaw.Producer/Utils/ReceivedFrameProcessingExtensionMethods.cs` | The frame filters and the response counter keys |
| `Container.FromTcp.Consumer.ToRaw.Producer/Models/KartListJsonConverter.cs` | Node labels and DBC derivation |
| `Container.FromRaw.Consumer.ToTelemetry.Producer/Services/DbcService.cs` | DBC decoding, and `xFrameIdUnknown_` |
| `Container.FromTelemetry.Consumer.ToInflux.Leaf/Services/InfluxDbTelemetrySaverService.cs` | Batching and writing to InfluxDB |
| `Container.FromTelemetry.Consumer.ToInflux.Leaf/Utils/TelemetryMessageExtensionMethods.cs` | Sample to point mapping, including the `battery` tag |
| `Container.FromRaw.Consumer.ToFile.Leaf/Services/RawDataFileWriterService.cs` | The debugging raw capture, its `saveRawData` gate and its file rolling |
| `ConvertRawDataLogToCsv/Program.cs` | Reading a folder of raw capture zips back out into a CSV |
| `Container.FromTcp.Consumer.ToRaw.Producer/Models/KartSpecs.cs` | The `xKartInfo` packing and the permitted type, spec and category sets |
| `KartSimulatorV2/` | The kart simulator, with `simulate.bat` alongside it for a whole fleet |
| `Telemetry.Shared/Utils/InternalSignals.cs` | The internal signal names |
| `Telemetry.Shared/Utils/TelemetryEnvironment.cs` | NATS stream and subject setup, and the environment defaults |

## Related Documentation

| Page | Relevance |
| - | - |
| [Track Operations](/software/developer/estate/systems/telemetry/track-operations) | The machine, the container stack and `ip-config.json` |
| [Charging Export](/software/developer/estate/systems/telemetry/charging-export) | What happens to the data once it is in InfluxDB |
| [Internal Signals](/software/developer/estate/systems/telemetry/internal-signals) | Every `x` prefixed signal this pipeline emits about itself |
| [Local Development](/software/developer/estate/systems/telemetry/local-development) | Driving all of this from a simulator instead of a kart |
| [Troubleshooting](/software/developer/estate/systems/telemetry/troubleshooting) | Symptoms, including a kart whose byte order is wrong |
| [Deployment And The Pipeline](/software/developer/estate/systems/telemetry/deployment) | How the receiver image is built and how it reaches a track |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.