> For the complete documentation index, see [llms.txt](https://docs.eyepop.ai/developer-documentation/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.eyepop.ai/developer-documentation/platform/response-format/predictions.md).

# Predictions

The prediction document, from its top-level fields down to each kind of result

A **prediction** is what a Pop found in one image, or in one frame of a video. A single image gets one prediction. A video or a live stream gets one for every frame the Pop processes, in order, as they are ready.

The Pop below finds people, looks for a face on each person, and reads the expression of each face. On the lobby image from the [Response Format](https://docs.eyepop.ai/developer-documentation/platform/response-format.md#a-first-look) overview it returns this prediction, shortened here to its essentials:

```json
{
  "source_id": "1e5c1e5c-d4a7-11f1-9dc3-4e4f5a7da4bd",
  "source_width": 1280,
  "source_height": 720,
  "timestamp": 0,
  "seconds": 0,
  "objects": [
    {
      "classLabel": "person",
      "confidence": 0.9775,
      "x": 503.281, "y": 251.718, "width": 166.562, "height": 464.765,
      "objects": [
        {
          "classLabel": "face",
          "confidence": 0.7411,
          "x": 554.438, "y": 263.858, "width": 65.296, "height": 65.296,
          "classes": [
            { "classLabel": "neutral", "confidence": 0.9966 }
          ]
        }
      ]
    }
  ]
}
```

<figure><img src="https://1956543008-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F2tRktyRCGgE1tPlAfT4h%2Fuploads%2Fgit-blob-24c42f0ebade38a9b7c22d8998deeb116642b6f2%2Fresponse-format-person-face.jpg?alt=media" alt="The person in the lobby with a box labeled person 98% around her, and inside it a smaller box around her face labeled face 74% · neutral"><figcaption><p>The prediction drawn on the image: the person, the face found within her, and the face's expression. The person is generated.</p></figcaption></figure>

It has three layers:

1. **The source and the time** — which media, how big its frame is, and when in it this is. These fields describe the prediction as a whole.
2. **The results for the whole frame** — here `objects`, the people found in the image.
3. **The results for each object** — each person carries what was found *within* it: the face, and on the face its expression.

The results nest as deep as the Pop does. A Pop that detects people, then faces, then expressions returns a person, holding a face, holding an expression.

### Source and time

| Key                             | Type    | Meaning                                                                                                                                                                              |
| ------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `source_id`                     | string  | The source this prediction belongs to. All predictions of one source share it                                                                                                        |
| `source_width`, `source_height` | number  | The frame's size in pixels; every position is in this frame. See [Spatial Attributes](https://docs.eyepop.ai/developer-documentation/platform/response-format/spatial-attributes.md) |
| `timestamp`                     | integer | Position in the media, in nanoseconds. Always present                                                                                                                                |
| `seconds`                       | number  | The same position in seconds, for display                                                                                                                                            |
| `duration`                      | integer | How long this frame stands for, in nanoseconds. Videos and streams only                                                                                                              |
| `captured_at`                   | integer | When the camera captured the frame, in nanoseconds since the Unix epoch. Only when the camera provides it                                                                            |

[Temporal Attributes](https://docs.eyepop.ai/developer-documentation/platform/response-format/temporal-attributes.md) explains the time fields and which to use for what.

### Results

A result is one thing a model found. Each kind has its own key and its own page:

| Key         | What it holds                                                                           | Page                                                                                                                                                    |
| ----------- | --------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `objects`   | Things found at a place: a person, a car, a face, a line of text                        | [Objects](https://docs.eyepop.ai/developer-documentation/platform/response-format/predictions/objects.md)                                               |
| `classes`   | Labels for the whole frame, or for one object: a scene type, an expression, an activity | [Classes and Texts](https://docs.eyepop.ai/developer-documentation/platform/response-format/predictions/classes-and-texts.md)                           |
| `texts`     | Text that was read                                                                      | [Classes and Texts](https://docs.eyepop.ai/developer-documentation/platform/response-format/predictions/classes-and-texts.md#texts)                     |
| `keyPoints` | Named landmarks: the joints of a body, the corners of an eye                            | [Key Points and Meshes](https://docs.eyepop.ai/developer-documentation/platform/response-format/predictions/key-points-and-meshes.md)                   |
| `meshes`    | Dense 3D surfaces, such as a face mesh                                                  | [Key Points and Meshes](https://docs.eyepop.ai/developer-documentation/platform/response-format/predictions/key-points-and-meshes.md#meshes)            |
| `depth`     | A depth map of the whole frame                                                          | [Masks, Contours and Depth](https://docs.eyepop.ai/developer-documentation/platform/response-format/predictions/masks-contours-and-depth.md#depth-maps) |
| `raw`       | A model's output as plain tensors, such as an embedding vector                          | [Raw Output and Details](https://docs.eyepop.ai/developer-documentation/platform/response-format/predictions/raw-and-details.md)                        |
| `details`   | Structured JSON from an ability with a response schema                                  | [Raw Output and Details](https://docs.eyepop.ai/developer-documentation/platform/response-format/predictions/raw-and-details.md#details)                |

Only `objects` are hierarchical. An object can hold its own `objects`, and those theirs, to any depth: that is how a Pop that crops detections and runs further models on them reports what it found within each one. Every other kind of result is a leaf. `classes`, `texts`, `keyPoints`, `meshes`, `raw` and `details` sit on the prediction or on an object, and never hold results of their own. `depth` sits on the prediction only.

The same keys appear at two levels. On the prediction itself they describe the **whole frame**: a scene classifier puts its answer in the prediction's `classes`. On an object they describe **that object**: an expression classifier that ran on a face puts its answer in the face's `classes`. Which level a result lands on follows from the Pop: a component that runs on the full frame reports on the prediction, and a component that runs on a [crop](https://docs.eyepop.ai/developer-documentation/platform/pop/pop-forwarding.md) reports on the object it was cropped from.

### Fields every result shares

Most results carry the same few descriptive fields:

| Key          | Type   | Meaning                                                                                                                  |
| ------------ | ------ | ------------------------------------------------------------------------------------------------------------------------ |
| `classLabel` | string | What was found, by name: `person`, `face`, `neutral`, `left shoulder`                                                    |
| `confidence` | number | How certain the model is, from 0 to 1                                                                                    |
| `category`   | string | The group a result belongs to: the `categoryName` you give its component in the Pop, or a group the ability names itself |

`category` tells results apart when they could look alike. With `categoryName="person"` on the person detector, every person it finds carries `"category": "person"`, wherever it appears in the response.

### Results over time

Three keys are about the video rather than about one frame:

* `track_events` — when a tracked object first appeared, and when it was last seen. See [Tracks and Selected Predictions](https://docs.eyepop.ai/developer-documentation/platform/response-format/predictions/tracks-and-selected-predictions.md#track-events).
* `selected` — `true` on a prediction that reports the best moment of a track, chosen after the fact. See [Tracks and Selected Predictions](https://docs.eyepop.ai/developer-documentation/platform/response-format/predictions/tracks-and-selected-predictions.md#selected-predictions).
* `motions` — the stretch of motion this frame belongs to; see below.

### Motion

With [motion detection](https://docs.eyepop.ai/developer-documentation/platform/sources-and-options/options/motion-detection.md) on, every prediction made while the scene is moving carries the motion it belongs to:

```json
{ "timestamp": 3375000000, "motions": [{ "begin_timestamp": 3375000000 }], "objects": [ … ] }
{ "timestamp": 3416666667, "motions": [{ "begin_timestamp": 3375000000 }], "objects": [ … ] }
…
{ "timestamp": 10250000000, "motions": [{ "begin_timestamp": 3375000000, "finished_timestamp": 10250000000 }] }
```

`begin_timestamp` is when the motion began, on the media's timeline. The prediction where the motion ends adds `finished_timestamp`. A prediction made outside a motion has no `motions` key.

### A prediction with nothing found

A processed frame always gets a prediction, even when nothing was found in it. It then carries only the source and time:

```json
{ "source_id": "fbd32076-d4a7-11f1-9dc3-4e4f5a7da4bd", "source_width": 640, "source_height": 360, "timestamp": 0, "seconds": 0 }
```

So the number of predictions tells you how many frames were processed, and a frame with no `objects` is a frame where nothing was found, not one that was lost.

### Example: list everything a prediction found

Because results nest, reading a prediction is a walk down its tree. This prints every object at its depth, with its labels:

{% tabs %}
{% tab title="Python" %}

```python
from eyepop import EyePopSdk
from eyepop.worker.worker_types import CropForward, InferenceComponent, Pop

pop = Pop(components=[InferenceComponent(
    ability="eyepop.person:latest",
    forward=CropForward(targets=[InferenceComponent(
        ability="eyepop.person.face.short-range:latest",
        forward=CropForward(targets=[InferenceComponent(ability="eyepop.expression:latest")]),
    )]),
)])

def print_objects(objects, depth=0):
    for obj in objects or []:
        labels = ", ".join(c["classLabel"] for c in obj.get("classes", []))
        print(f"{'  ' * depth}{obj['classLabel']} {obj['confidence']:.0%} {labels}")
        print_objects(obj.get("objects"), depth + 1)

with EyePopSdk.sync_worker(pop=pop) as endpoint:
    job = endpoint.upload("lobby.jpg")
    while prediction := job.predict():
        print_objects(prediction.get("objects"))
```

{% endtab %}

{% tab title="Node" %}

```typescript
import { EyePop, ForwardOperatorType, PopComponentType, PredictedObject } from '@eyepop.ai/eyepop'

function printObjects(objects: PredictedObject[] | undefined, depth = 0) {
    for (const obj of objects ?? []) {
        const labels = (obj.classes ?? []).map(c => c.classLabel).join(', ')
        console.log(`${'  '.repeat(depth)}${obj.classLabel} ${Math.round((obj.confidence ?? 0) * 100)}% ${labels}`)
        printObjects(obj.objects, depth + 1)
    }
}

async function main() {
    const endpoint = await EyePop.workerEndpoint({
        pop: {
            components: [{
                type: PopComponentType.INFERENCE,
                ability: 'eyepop.person:latest',
                forward: {
                    operator: { type: ForwardOperatorType.CROP },
                    targets: [{
                        type: PopComponentType.INFERENCE,
                        ability: 'eyepop.person.face.short-range:latest',
                        forward: {
                            operator: { type: ForwardOperatorType.CROP },
                            targets: [{ type: PopComponentType.INFERENCE, ability: 'eyepop.expression:latest' }],
                        },
                    }],
                },
            }],
        },
    }).connect()
    try {
        for await (const prediction of await endpoint.process({ source: { path: 'lobby.jpg' } })) {
            printObjects(prediction.objects)
        }
    } finally {
        await endpoint.disconnect()
    }
}

main().catch(console.error)
```

{% endtab %}
{% endtabs %}

```
person 98% 
  face 74% neutral
```

### Next steps

* [Objects](https://docs.eyepop.ai/developer-documentation/platform/response-format/predictions/objects.md) — the box, identity and nesting of an object
* [Classes and Texts](https://docs.eyepop.ai/developer-documentation/platform/response-format/predictions/classes-and-texts.md) — labels and text, for a frame or an object
* [Tracks and Selected Predictions](https://docs.eyepop.ai/developer-documentation/platform/response-format/predictions/tracks-and-selected-predictions.md) — following objects through a video
