> For the complete documentation index, see [llms.txt](https://docs.eyepop.ai/developer-documentation/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.eyepop.ai/developer-documentation/platform/response-format/predictions/tracks-and-selected-predictions.md).

# Tracks and Selected Predictions

Following an object through a video, and the predictions that report a track's best moment after the fact

A detector looks at each frame on its own: in every frame of a video, it finds *a* person, never *the same* person as before. A [tracking component](https://docs.eyepop.ai/developer-documentation/platform/pop/pop-components.md#tracking) links those detections across frames, so the person who walks in at second 0 and out at second 9 is one **track** with one identity. Tracks add three things to the response: a `trackId` on each tracked object, `track_events` that say when tracks start and end, and, with a select operator, **selected predictions** that report each track's best moment.

### Tracks

A tracked object carries a `trackId`. It is the same number on every frame the tracker follows that object:

```json
{ "timestamp": 2041666667, "objects": [{ "classLabel": "person", "trackId": 1, "x": 548.181, "y": 279.943, "width": 135.593, "height": 401.181 }] }
{ "timestamp": 2083333333, "objects": [{ "classLabel": "person", "trackId": 1, "x": 547.615, "y": 278.744, "width": 136.23, "height": 403.502 }] }
```

* A `trackId` is a number, unique within one source. Two people in view at once have two different ids.
* A track ends when its object has not been seen for a while. If the same person comes back later, they get a new `trackId`: the tracker follows objects, it does not recognize people.
* An object the tracker is not (yet) following has no `trackId`. The very first frame an object appears on can be such a frame.

### Track events

`track_events` on a prediction report tracks that started or ended. Here is the life of track 1, a person walking past an office entrance, from the predictions of a 12-second video:

```json
{ "timestamp": 41666667,   "track_events": [{ "track_id": 1, "is_ended": false, "started_at": 0 }] }
…
{ "timestamp": 9458333333, "track_events": [{ "track_id": 1, "is_ended": true, "started_at": 0, "ended_at": 9291666667 }] }
```

| Key          | Type    | Meaning                                                         |
| ------------ | ------- | --------------------------------------------------------------- |
| `track_id`   | integer | The track, the same number as the objects' `trackId`            |
| `is_ended`   | boolean | `false` when the track starts, `true` when it ends              |
| `started_at` | integer | When the track's object was first seen, as a `timestamp`        |
| `ended_at`   | integer | When it was last seen, as a `timestamp`. Only on an ended track |

Both times are on the media's timeline, like `timestamp`. A track ending is reported **after** the fact: the tracker waits to be sure the object is gone, so the event for track 1 arrives on the frame at 9.458 s and says the person was last seen at 9.292 s. Use `ended_at`, not the `timestamp` of the prediction that carries it, as the end of the track.

`track_events` can sit on a prediction with no objects: the frame where a track is declared gone may show nothing at all. At the end of a video, one last prediction, stamped one frame after the last one, ends every track still open:

```json
{ "timestamp": 12083333333, "track_events": [
    { "track_id": 2, "is_ended": true, "started_at": 9083333333, "ended_at": 12041666667 },
    { "track_id": 4, "is_ended": true, "started_at": 10000000000, "ended_at": 12041666667 }
] }
```

So a client that reads to the end of a video has seen every track start and end.

### Selected predictions

A [select operator](https://docs.eyepop.ai/developer-documentation/platform/temporal-context/select-operators.md) picks, for each track, the frame that shows its object best, and runs a deeper analysis on that frame only: a face-ID model, a vision-language model. The result comes back as a **selected prediction**: a prediction of its own, about that one frame and that one track, marked with `"selected": true`.

```json
{
  "selected": true,
  "timestamp": 8625000000,
  "seconds": 8.625,
  "duration": 41666666,
  "source_width": 1280,
  "source_height": 720,
  "objects": [
    {
      "classLabel": "person",
      "confidence": 0.9252,
      "trackId": 1,
      "x": 12.12, "y": 3.717, "width": 466.638, "height": 711.748,
      "objects": [
        { "confidence": 0.9716, "x": 137.293, "y": 69.366, "width": 168.836, "height": 168.835, "raw": [ … ] }
      ]
    }
  ]
}
```

A selected prediction differs from the live ones in three ways:

* **It is about the past.** Its `timestamp`, `seconds`, `duration` and `captured_at` are those of the selected frame, 8.625 s here, but it arrives once the selection is made: in this video after the live prediction for 9.542 s, once track 1 had ended. The predictions of a source are therefore **not in timestamp order** once selections arrive.
* **It holds one track.** `objects` holds the selected object with its `trackId`, and nested within it the results of the deeper analysis. The `trackId` is what ties a selection to the live predictions of the same track.
* **It carries no `track_events` and no `motions`.** Those belong to the live predictions.

Selected predictions need format version 3: over HTTP, ask for `version=3`. The SDKs always ask for it, from Python SDK 3.25.4 and Node SDK 3.25.3.

#### Keeping the two apart

Treat the two kinds of prediction as two streams:

* **Live predictions** are the video as it plays: one per frame, in order. Draw them, count them, and follow tracks with them.
* **Selected predictions** are insights: one per track, or one per interval, late. Store them, keyed by `trackId`.

Code that assumes predictions arrive in timestamp order, such as drawing frames, measuring the gap between frames or computing frame rates, must skip selected predictions.

### Example: each track's lifetime and its selection

{% tabs %}
{% tab title="Python" %}

```python
tracks = {}

def on_prediction(prediction):
    if prediction.get("selected"):
        for obj in prediction.get("objects", []):
            tracks.setdefault(obj["trackId"], {})["selected_at"] = prediction["seconds"]
        return
    for event in prediction.get("track_events", []):
        track = tracks.setdefault(event["track_id"], {})
        track["start"] = event["started_at"] / 1e9
        if event["is_ended"]:
            track["end"] = event["ended_at"] / 1e9

# for each prediction of the job: on_prediction(prediction), then:
for track_id, track in sorted(tracks.items()):
    print(f"track {track_id}: {track['start']:.3f}s to {track['end']:.3f}s, "
          f"selected frame {track.get('selected_at', '-')}")
```

{% endtab %}

{% tab title="Node" %}

```typescript
import { Prediction } from '@eyepop.ai/eyepop'

type TrackEvent = { track_id: number, is_ended: boolean, started_at: number, ended_at?: number }
type Track = { start?: number, end?: number, selectedAt?: number }

const tracks = new Map<number, Track>()

function track(id: number): Track {
    if (!tracks.has(id)) tracks.set(id, {})
    return tracks.get(id)!
}

function onPrediction(prediction: Prediction & { track_events?: TrackEvent[] }) {
    if (prediction.selected) {
        for (const obj of prediction.objects ?? []) {
            if (obj.trackId !== undefined) track(obj.trackId).selectedAt = prediction.seconds
        }
        return
    }
    for (const event of prediction.track_events ?? []) {
        track(event.track_id).start = event.started_at / 1e9
        if (event.is_ended && event.ended_at !== undefined) track(event.track_id).end = event.ended_at / 1e9
    }
}

function printTracks() {
    for (const [id, t] of [...tracks].sort(([a], [b]) => a - b)) {
        console.log(`track ${id}: ${t.start?.toFixed(3)}s to ${t.end?.toFixed(3)}s, selected frame ${t.selectedAt ?? '-'}`)
    }
}
```

{% endtab %}
{% endtabs %}

On the walk-by video from [Example: Best Face-ID Vector from Walk-Bys](https://docs.eyepop.ai/developer-documentation/platform/temporal-context/example-face-id-from-walk-bys.md), cut to its first 12 seconds:

```
track 1: 0.000s to 9.292s, selected frame 8.625
track 2: 9.083s to 12.042s, selected frame -
track 3: 9.125s to 10.375s, selected frame -
track 4: 10.000s to 12.042s, selected frame -
```

Track 1 was selected once it ended, as a select operator without an interval does. Tracks 2 to 4 got no selection in this clip.

### Next steps

* [Temporal Context in Videos](https://docs.eyepop.ai/developer-documentation/platform/temporal-context.md) — why and how to select
* [Select Operators](https://docs.eyepop.ai/developer-documentation/platform/temporal-context/select-operators.md) — how relevance and timing decide the selected frame
