> For the complete documentation index, see [llms.txt](https://docs.eyepop.ai/developer-documentation/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.eyepop.ai/developer-documentation/platform/response-format/predictions/masks-contours-and-depth.md).

# Masks, Contours and Depth

The exact shape of an object, as a mask or contours, and a depth map for the whole frame

A box says roughly where an object is. A segmentation ability says exactly which pixels belong to it, as a **mask**, and a contour finder turns that mask into **contours**, polygons you can draw or measure. A depth ability goes further and estimates, for every pixel of the frame, how far away it is, as a **depth map**.

### Masks

A mask is a black-and-white image the size of the object's box: a pixel is set where the object is and clear where it is not. A small diamond-shaped object, 10 pixels wide and 4 high, looks like this:

```json
{
  "classLabel": "diamond",
  "x": 100, "y": 50, "width": 10, "height": 4,
  "mask": { "width": 10, "height": 4, "stride": 2, "bitmap": "DAA/AD8ADAA=" }
}
```

<figure><img src="https://1956543008-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F2tRktyRCGgE1tPlAfT4h%2Fuploads%2Fgit-blob-04035cc41f563b7df5cc981a191ae4682d2a2885%2Fresponse-format-mask-contour.svg?alt=media" alt="The 10 by 4 pixel object, magnified, inside its dashed box from x 100 to 110 and y 50 to 54. Left: the mask marks its 16 pixels with 1 and every other pixel of the box with 0. Right: the contour is one polygon of 12 points around the same pixels."><figcaption><p>The diamond's mask and its contour, laid over the pixels of the image.</p></figcaption></figure>

| Key               | Type    | Meaning                                                                                                                                                                                                           |
| ----------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `width`, `height` | integer | The mask's size in mask pixels                                                                                                                                                                                    |
| `stride`          | integer | Bytes per row of the bitmap                                                                                                                                                                                       |
| `bitmap`          | string  | The pixels, base64-encoded, one bit each                                                                                                                                                                          |
| `world`           | string  | A point cloud for the mask, in meters, when the Pop asks for world coordinates. See [World Coordinates](https://docs.eyepop.ai/developer-documentation/platform/depth-and-world-coordinates/world-coordinates.md) |

The mask covers the object's box. Its top-left pixel is at the box's `x` and `y`, and it stretches over the box's `width` and `height`. When the mask's own `width` and `height` differ from the box's, scale it to the box to draw it.

#### How the bitmap is laid out

The bitmap is a grid of bits, one per pixel, row by row from the top:

* Each row takes `stride` bytes: `width` bits, rounded up to whole bytes.
* Within a byte, the **highest bit is the leftmost pixel**. A `1` is a pixel of the object.

The diamond's mask shows the layout. Each row needs 10 bits, so `stride` is 2 bytes, and the last 6 bits of every row are unused:

```
pixels         bytes of the row
....##....     00001100 00000000
..######..     00111111 00000000
..######..     00111111 00000000
....##....     00001100 00000000
```

#### Example: decode a mask

```python
import base64

import numpy as np

def decode_mask(mask):
    bits = np.unpackbits(np.frombuffer(base64.b64decode(mask["bitmap"]), dtype=np.uint8))
    return bits.reshape(mask["height"], mask["stride"] * 8)[:, :mask["width"]].astype(bool)

pixels = decode_mask({"width": 10, "height": 4, "stride": 2, "bitmap": "DAA/AD8ADAA="})
for row in pixels:
    print("".join("#" if p else "." for p in row))
print(f"{pixels.mean():.0%} of the box is the object")
```

```
....##....
..######..
..######..
....##....
40% of the box is the object
```

`np.unpackbits` reads the highest bit first, which is exactly the layout of the bitmap.

### Contours

A contour is the outline of a mask as a polygon, ready to draw or to measure. A [contour finder](https://docs.eyepop.ai/developer-documentation/platform/pop/pop-components.md#contour_finder) on a segmentation result adds `contours` to the object. For the diamond it is one polygon around its pixels:

```json
{
  "classLabel": "diamond",
  "x": 100, "y": 50, "width": 10, "height": 4,
  "contours": [
    {
      "points": [
        { "x": 104, "y": 50 }, { "x": 106, "y": 50 }, { "x": 106, "y": 51 }, { "x": 108, "y": 51 },
        { "x": 108, "y": 53 }, { "x": 106, "y": 53 }, { "x": 106, "y": 54 }, { "x": 104, "y": 54 },
        { "x": 104, "y": 53 }, { "x": 102, "y": 53 }, { "x": 102, "y": 51 }, { "x": 104, "y": 51 }
      ],
      "cutouts": []
    }
  ]
}
```

| Key       | Type                      | Meaning                                           |
| --------- | ------------------------- | ------------------------------------------------- |
| `points`  | array of points           | The polygon's corners, in order, in source pixels |
| `cutouts` | array of arrays of points | Holes in the polygon, each a polygon of its own   |

An object can have several contours, when its mask has separate parts. A cutout is a hole: the gap between an arm and the body, for instance, is a cutout of the person's contour.

Contour points are in the full frame, like every other position, so they draw directly on the image. With world coordinates asked for, each point also carries `worldX`, `worldY` and `worldZ` in meters.

### Depth maps

A depth ability estimates the distance of every pixel of the frame. It returns one map for the whole frame, on the prediction:

```json
{
  "source_width": 1280,
  "source_height": 720,
  "timestamp": 0,
  "depth": {
    "width": 518,
    "height": 291,
    "semantic": "canonical_metric",
    "values": "kbUFQNx/AUCjcgFAzf8AQIq9AEBYMwBA…"
  }
}
```

<figure><img src="https://1956543008-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F2tRktyRCGgE1tPlAfT4h%2Fuploads%2Fgit-blob-ff6bbc37d579f034388b04e1c6510601dac50f5f%2Fresponse-format-depth.jpg?alt=media" alt="The lobby image tinted by depth: the person and the floor in front of her in yellow, near the camera, the walls and plants in green and blue, and the doors at the back in purple, farthest away"><figcaption><p>This depth map from <code>eyepop.depth.metric.large:latest</code>, scaled to the image and laid over it: yellow is near, purple is far. The person is generated.</p></figcaption></figure>

| Key               | Type    | Meaning                                                                     |
| ----------------- | ------- | --------------------------------------------------------------------------- |
| `width`, `height` | integer | The map's size. Always the frame's aspect ratio, usually a lower resolution |
| `semantic`        | string  | What the values mean: `metric`, `canonical_metric`, `relative` or `unknown` |
| `values`          | string  | `width × height` little-endian 32-bit floats, row by row, base64-encoded    |
| `world`           | string  | A point cloud of the whole scene, when the Pop asks for one                 |

The map is smaller than the frame: here 518 × 291 for a 1280 × 720 image. To find the depth at a point of the frame, scale the point by the ratio of the sizes. [Depth Maps](https://docs.eyepop.ai/developer-documentation/platform/depth-and-world-coordinates/depth-maps.md) explains `semantic`, how to decode the values and turn them into meters, and why sky pixels are infinite.

A depth map is large: this one is close to a megabyte of text, on every frame. A Pop that needs depth only to place its objects in meters can keep the map itself out of the response; see [Depth and World Coordinates](https://docs.eyepop.ai/developer-documentation/platform/depth-and-world-coordinates.md).

### Next steps

* [Depth and World Coordinates](https://docs.eyepop.ai/developer-documentation/platform/depth-and-world-coordinates.md) — positions in meters from masks, contours and depth maps
* [Raw Output and Details](https://docs.eyepop.ai/developer-documentation/platform/response-format/predictions/raw-and-details.md) — the remaining kinds of result
