For the complete documentation index, see llms.txt. This page is also available as Markdown.

Classes and Texts

Labels and recognized text, for a whole frame or for one object

Not every answer has a place. A classifier says what kind — a scene, an expression, an activity — and a text reader says what is written. These answers come as classes and texts, attached to the whole frame or to one object.

Classes

A class is a label with a confidence:

{ "classLabel": "neutral", "confidence": 0.9966 }
Key
Type
Meaning

classLabel

string

The label

confidence

number

How certain the model is, from 0 to 1

category

string

The group the label belongs to, when the Pop or the ability names one

For the whole frame

A classifier that runs on the full image puts its classes on the prediction. Asked what the people in the lobby image are doing, eyepop.image-contents:latest answers:

{
  "source_width": 1280,
  "source_height": 720,
  "timestamp": 0,
  "classes": [
    { "classLabel": "standing", "confidence": 0.99, "category": "human activity" }
  ]
}

For one object

A classifier that runs on a crop puts its classes on the object it was cropped from. An expression classifier run on each face:

The object's own classLabel says what it is, a face; its classes say what was learned about it, a neutral expression.

More than one class

classes is a list. An ability can return several labels for one thing, each with its own confidence. The confidences are independent: they do not have to add up to 1, and several can be high at once. Asked what the person in the lobby is wearing, eyepop.image-contents:latest names two pieces of clothing, both with high confidence:

Texts

A text is a string that was read, with a confidence:

Key
Type
Meaning

text

string

What was read

confidence

number

How certain the reader is, from 0 to 1

category

string

The group, when the Pop or the ability names one

Reading text usually takes two steps: a detector finds where the text is, and a reader reads each place. So texts typically arrive on the object the reader ran on. Run on a sign that says "New York Pies", the text detector finds three words, and the reader adds what each says:

A red wall with the words NEW YORK PIES painted on it in black and white letters, with a box around each word
Three text objects, one per word.

Two confidences are at play: the object's says how sure the detector is that there is text at this place, and the text's says how sure the reader is of what it says. A clearly visible but badly lettered word has a high first and a low second.

The objects are not in reading order. To put words in reading order, sort them by their position.

Example: the best label of each face

On the prediction from the person, face and expression Pop in Predictions:

Next steps

Last updated