> For the complete documentation index, see [llms.txt](https://docs.eyepop.ai/developer-documentation/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.eyepop.ai/developer-documentation/platform/response-format/predictions/classes-and-texts.md).

# Classes and Texts

Labels and recognized text, for a whole frame or for one object

Not every answer has a place. A classifier says *what kind* — a scene, an expression, an activity — and a text reader says *what is written*. These answers come as **classes** and **texts**, attached to the whole frame or to one object.

### Classes

A class is a label with a confidence:

```json
{ "classLabel": "neutral", "confidence": 0.9966 }
```

| Key          | Type   | Meaning                                                               |
| ------------ | ------ | --------------------------------------------------------------------- |
| `classLabel` | string | The label                                                             |
| `confidence` | number | How certain the model is, from 0 to 1                                 |
| `category`   | string | The group the label belongs to, when the Pop or the ability names one |

#### For the whole frame

A classifier that runs on the full image puts its classes on the prediction. Asked whether the people on a street photo are walking, standing, sitting or running, [`eyepop.gemini-4-argon.classify:latest`](https://docs.eyepop.ai/developer-documentation/platform/abilities/open-vocabulary.md) answers:

```json
{
  "source_width": 810,
  "source_height": 1080,
  "timestamp": 0,
  "classes": [
    { "classLabel": "walking", "confidence": 0.9602 },
    { "classLabel": "standing", "confidence": 0.6594 }
  ]
}
```

#### For one object

A classifier that runs on a crop puts its classes on the object it was cropped from. An expression classifier run on each face:

```json
{
  "classLabel": "face",
  "confidence": 0.7411,
  "x": 554.438, "y": 263.858, "width": 65.296, "height": 65.296,
  "classes": [
    { "classLabel": "neutral", "confidence": 0.9966 }
  ]
}
```

The object's own `classLabel` says what it *is*, a face; its `classes` say what was *learned about it*, a neutral expression.

#### More than one class

`classes` is a list. An ability can return several labels for one thing, each with its own confidence. The confidences are independent: they do not have to add up to 1, and several can be high at once. Asked which of a sheepskin coat, a red scarf, sunglasses and a hat appear on the same street photo, `eyepop.gemini-4-argon.classify:latest` names two, both with high confidence:

```json
"classes": [
  { "classLabel": "a sheepskin coat", "confidence": 0.9087 },
  { "classLabel": "sunglasses", "confidence": 0.9776 }
]
```

### Texts

A text is a string that was read, with a confidence:

```json
{ "text": "YORK", "confidence": 0.9898 }
```

| Key          | Type   | Meaning                                          |
| ------------ | ------ | ------------------------------------------------ |
| `text`       | string | What was read                                    |
| `confidence` | number | How certain the reader is, from 0 to 1           |
| `category`   | string | The group, when the Pop or the ability names one |

Reading text usually takes two steps: a detector finds *where* the text is, and a reader reads each place. So texts typically arrive on the object the reader ran on. Run on a sign that says "New York Pies", the text detector finds three words, and the reader adds what each says:

<figure><img src="https://1956543008-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F2tRktyRCGgE1tPlAfT4h%2Fuploads%2Fgit-blob-c96332bee32546e79b92c9a0d7e819f6c3f60103%2Fresponse-format-sign.jpg?alt=media" alt="A red wall with the words NEW YORK PIES painted on it in black and white letters, with a box around each word"><figcaption><p>Three text objects, one per word.</p></figcaption></figure>

```json
"objects": [
  {
    "classLabel": "text", "confidence": 0.9653,
    "x": 305.937, "y": 61.699, "width": 190, "height": 110.39,
    "texts": [{ "text": "YORK", "confidence": 0.9898 }]
  },
  {
    "classLabel": "text", "confidence": 0.9614,
    "x": 133.007, "y": 104.721, "width": 146.796, "height": 100.458,
    "texts": [{ "text": "NEW", "confidence": 0.9328 }]
  },
  {
    "classLabel": "text", "confidence": 0.8891,
    "x": 233.984, "y": 166.992, "width": 192.031, "height": 128.32,
    "texts": [{ "text": "PIES", "confidence": 0.7089 }]
  }
]
```

Two confidences are at play: the object's says how sure the detector is that there *is* text at this place, and the text's says how sure the reader is of *what* it says. A clearly visible but badly lettered word has a high first and a low second.

The objects are not in reading order. To put words in reading order, sort them by their position.

### Example: the best label of each face

{% tabs %}
{% tab title="Python" %}

```python
def best_class(obj):
    classes = obj.get("classes", [])
    return max(classes, key=lambda c: c["confidence"])["classLabel"] if classes else None

def faces(objects):
    for obj in objects or []:
        if obj["classLabel"] == "face":
            yield obj
        yield from faces(obj.get("objects"))

for face in faces(prediction.get("objects")):
    print(f"face at ({face['x']:.0f}, {face['y']:.0f}): {best_class(face)}")
```

{% endtab %}

{% tab title="Node" %}

```typescript
import { Prediction, PredictedObject } from '@eyepop.ai/eyepop'

function bestClass(obj: PredictedObject): string | undefined {
    const classes = obj.classes ?? []
    return classes.length ? classes.reduce((a, b) => ((b.confidence ?? 0) > (a.confidence ?? 0) ? b : a)).classLabel : undefined
}

function* faces(objects: PredictedObject[] | undefined): Generator<PredictedObject> {
    for (const obj of objects ?? []) {
        if (obj.classLabel === 'face') yield obj
        yield* faces(obj.objects)
    }
}

function printFaces(prediction: Prediction) {
    for (const face of faces(prediction.objects)) {
        console.log(`face at (${face.x.toFixed(0)}, ${face.y.toFixed(0)}): ${bestClass(face)}`)
    }
}
```

{% endtab %}
{% endtabs %}

On the prediction from the person, face and expression Pop in [Predictions](https://docs.eyepop.ai/developer-documentation/platform/response-format/predictions.md):

```
face at (554, 264): neutral
```

### Next steps

* [Key Points and Meshes](https://docs.eyepop.ai/developer-documentation/platform/response-format/predictions/key-points-and-meshes.md) — landmarks and surfaces on an object
* [Raw Output and Details](https://docs.eyepop.ai/developer-documentation/platform/response-format/predictions/raw-and-details.md) — answers that are neither a label nor text
