Depth Maps

The frame-level map of distance that world coordinates are back-projected from

A depth estimation ability produces one value per pixel: how far that part of the scene is from the camera. That map is what everything else in this section is built on — world coordinates are the map read through a calibration.

Only metric depth can be back-projected. See Models for the four abilities to choose from.

Asking for one

A depth map is not an inference component. A Pop names it once, in depthMap, because the platform back-projects every prediction through a single map — a second depth source would have nowhere to go.

Python

from eyepop.worker.worker_types import InferenceComponent, Pop, PopDepthMap

pop = Pop(
    components=[InferenceComponent(ability="eyepop.person:latest")],
    depthMap=PopDepthMap(ability="eyepop.depth.metric.large:latest", toWorld=True),
)

Node

const pop = {
    components: [{ type: PopComponentType.INFERENCE, ability: 'eyepop.person:latest' }],
    depthMap: { ability: 'eyepop.depth.metric.large:latest', toWorld: true },
}

Give exactly one of ability and abilityUuid. Both SDKs reject naming neither or both before the request leaves your process.

toWorld on the depth map is what reveals the map itself. Name a depthMap without it and the platform still builds the depth stage and still enriches your predictions — but keeps the map out of the response.

That is the useful default, not an oversight. A depth map is roughly a megabyte of base64 per frame; on a live stream, the difference between a few enriched key points and a megabyte a frame is the difference between a working pipeline and a saturated one. Ask for the map only when you want the map.

A depthMap with toWorld is also a complete Pop on its own. The map is the only consumer, so nothing needs to be detected for it to mean something:

Python

pop = Pop(
    components=[],
    depthMap=PopDepthMap(ability="eyepop.depth.metric.large:latest", toWorld=True),
)

Node

const pop = {
    components: [],
    depthMap: { ability: 'eyepop.depth.metric.large:latest', toWorld: true },
}

What comes back

The map arrives as a frame-level depth member on the prediction.

Field	Type	Description
width	integer	Map width in map pixels.
height	integer	Map height in map pixels.
values	string	Base64 of width * height little-endian float32 values, row-major.
semantic	string	What the values mean. See below.
world	string	The scene point cloud, present when depthMap.toWorld was asked for. See World Coordinates.

The map has the aspect ratio of the source frame but not its resolution, so a source coordinate maps onto it proportionally: (x * width / source_width, y * height / source_height). Both SDKs do that for you when you pass the source dimensions.

Sky pixels carry +Infinity.

What the values mean

semantic is always present in prediction v2, "unknown" included, so an absent one means a worker predating the field rather than a map declining to say.

semantic	Meaning	Back-projectable
metric	The value is already meters.	yes
canonical_metric	Meters up to a factor fixed by the lens, which the calibration supplies.	yes
relative	Scale- and shift-invariant: ordering is meaningful, distance is not.	no
unknown	The ability declared nothing.	no

canonical_metric values are not meters as they stand — converting them needs the focal length in pixels, rescaled to the map's own resolution. Rather than doing that arithmetic yourself, ask for world coordinates: the platform applies your calibration and returns meters directly.

A relative or unknown map is accepted and simply produces no world coordinates. There is no error — which is why the ability you name matters.

Reading it

Python

from eyepop import DepthMap

depth_map = DepthMap.from_prediction(prediction)
if depth_map is not None:
    print(depth_map.array.shape)       # (height, width), float32, +inf for sky
    print(depth_map.sky_mask)          # boolean array, True where sky
    print(depth_map.finite_min, depth_map.finite_max)

    # sample by source frame coordinate
    value = depth_map.at(x, y, prediction["source_width"], prediction["source_height"])
    if not depth_map.is_sky(x, y, prediction["source_width"], prediction["source_height"]):
        print(value)

Decoding is lazy — the base64 is not touched until .array is first read.

Node

import { decodeDepthMap } from '@eyepop.ai/eyepop'

if (prediction.depth) {
    const depthMap = decodeDepthMap(prediction.depth)
    console.log(depthMap.width, depthMap.height)
    console.log(depthMap.finiteMin, depthMap.finiteMax) // undefined if all sky

    // sample by source frame coordinate
    const value = depthMap.at(x, y, prediction.source_width, prediction.source_height)
    if (!depthMap.isSky(x, y, prediction.source_width, prediction.source_height)) {
        console.log(value)
    }
}

Called without the source dimensions, at() and isSky() index the map's own grid instead. Coordinates are clamped to the map rather than rejected.

Seeing it

Both SDKs render a depth map as a turbo heatmap over the frame: near is warm, far is cool, and sky is left transparent.

Python

import matplotlib.pyplot as plt
from eyepop.visualize import EyePopPlot

plot = EyePopPlot(plt.gca())
plot.depth(prediction, opacity=0.5)
plt.show()

Node

import { Render2d } from '@eyepop.ai/eyepop-render-2d'

const renderer = Render2d.renderer(context, [
    Render2d.renderDepth({ opacity: 0.5, renderSky: false }),
])
renderer.draw(prediction)

A source frame and its depth map from eyepop.depth.metric.large. The near pavement and foreground figures are warm, the street recedes through green, the far end is cool, and sky pixels are left untouched. Source photo: Sint Antoniesbreestraat, Amsterdam, by Fons Heijnsbroek (CC0).

Next steps

• Camera Calibration — the lens and pose that turn this map into meters

• World Coordinates — reading positions off the map

• Models — the four metric depth abilities