Tracks and Selected Predictions
Following an object through a video, and the predictions that report a track's best moment after the fact
A detector looks at each frame on its own: in every frame of a video, it finds a person, never the same person as before. A tracking component links those detections across frames, so the person who walks in at second 0 and out at second 9 is one track with one identity. Tracks add three things to the response: a trackId on each tracked object, track_events that say when tracks start and end, and, with a select operator, selected predictions that report each track's best moment.
Tracks
A tracked object carries a trackId. It is the same number on every frame the tracker follows that object:
{ "timestamp": 2041666667, "objects": [{ "classLabel": "person", "trackId": 1, "x": 548.181, "y": 279.943, "width": 135.593, "height": 401.181 }] }
{ "timestamp": 2083333333, "objects": [{ "classLabel": "person", "trackId": 1, "x": 547.615, "y": 278.744, "width": 136.23, "height": 403.502 }] }A
trackIdis a number, unique within one source. Two people in view at once have two different ids.A track ends when its object has not been seen for a while. If the same person comes back later, they get a new
trackId: the tracker follows objects, it does not recognize people.An object the tracker is not (yet) following has no
trackId. The very first frame an object appears on can be such a frame.
Track events
track_events on a prediction report tracks that started or ended. Here is the life of track 1, a person walking past an office entrance, from the predictions of a 12-second video:
{ "timestamp": 41666667, "track_events": [{ "track_id": 1, "is_ended": false, "started_at": 0 }] }
…
{ "timestamp": 9458333333, "track_events": [{ "track_id": 1, "is_ended": true, "started_at": 0, "ended_at": 9291666667 }] }track_id
integer
The track, the same number as the objects' trackId
is_ended
boolean
false when the track starts, true when it ends
started_at
integer
When the track's object was first seen, as a timestamp
ended_at
integer
When it was last seen, as a timestamp. Only on an ended track
Both times are on the media's timeline, like timestamp. A track ending is reported after the fact: the tracker waits to be sure the object is gone, so the event for track 1 arrives on the frame at 9.458 s and says the person was last seen at 9.292 s. Use ended_at, not the timestamp of the prediction that carries it, as the end of the track.
track_events can sit on a prediction with no objects: the frame where a track is declared gone may show nothing at all. At the end of a video, one last prediction, stamped one frame after the last one, ends every track still open:
So a client that reads to the end of a video has seen every track start and end.
Selected predictions
A select operator picks, for each track, the frame that shows its object best, and runs a deeper analysis on that frame only: a face-ID model, a vision-language model. The result comes back as a selected prediction: a prediction of its own, about that one frame and that one track, marked with "selected": true.
A selected prediction differs from the live ones in three ways:
It is about the past. Its
timestamp,seconds,durationandcaptured_atare those of the selected frame, 8.625 s here, but it arrives once the selection is made: in this video after the live prediction for 9.542 s, once track 1 had ended. The predictions of a source are therefore not in timestamp order once selections arrive.It holds one track.
objectsholds the selected object with itstrackId, and nested within it the results of the deeper analysis. ThetrackIdis what ties a selection to the live predictions of the same track.It carries no
track_eventsand nomotions. Those belong to the live predictions.
Selected predictions need format version 3: over HTTP, ask for version=3. The SDKs always ask for it, from Python SDK 3.25.4 and Node SDK 3.25.3.
Keeping the two apart
Treat the two kinds of prediction as two streams:
Live predictions are the video as it plays: one per frame, in order. Draw them, count them, and follow tracks with them.
Selected predictions are insights: one per track, or one per interval, late. Store them, keyed by
trackId.
Code that assumes predictions arrive in timestamp order, such as drawing frames, measuring the gap between frames or computing frame rates, must skip selected predictions.
Example: each track's lifetime and its selection
On the walk-by video from Example: Best Face-ID Vector from Walk-Bys, cut to its first 12 seconds:
Track 1 was selected once it ended, as a select operator without an interval does. Tracks 2 to 4 got no selection in this clip.
Next steps
Temporal Context in Videos — why and how to select
Select Operators — how relevance and timing decide the selected frame
Last updated
