World Coordinates

Reading a prediction's positions in meters, and the point clouds that come with them

With a depth map and a calibration in place, predictions carry a 3D position in meters alongside their 2D one.

Opting in

Two things have to be true: the Pop names a depthMap, and the components whose predictions should be translated set toWorld.

Python

from eyepop.worker.camera import Camera
from eyepop.worker.worker_types import InferenceComponent, Pop, PopDepthMap, SourceDefaults

pop = Pop(
    components=[InferenceComponent(ability="eyepop.person:latest", toWorld=True)],
    depthMap=PopDepthMap(ability="eyepop.depth.metric.large:latest"),
    defaults=SourceDefaults(camera=Camera(hfovDegrees=72.0)),
)

Node

const pop = {
    components: [{ type: PopComponentType.INFERENCE, ability: 'eyepop.person:latest', toWorld: true }],
    depthMap: { ability: 'eyepop.depth.metric.large:latest' },
    defaults: { camera: { hfovDegrees: 72 } },
}

toWorld only means something on a component that runs its own inference — inference and tracking. Asking for it on a forward, contour_finder or component_finder is rejected when the Pop is compiled. A contour finder's points do get enriched, but they belong to the object that fed it, so the request goes on the inference component upstream.

What carries coordinates

worldX, worldY and worldZ appear on every point-shaped member:

• key points

• outline points

• contour points, including cutouts

Bounding boxes are not enriched. A box is not a point, and any single anchor — centre, nearest corner, base — would be an arbitrary choice made on your behalf.

A point the platform could not place carries none of the three members at all — not a zero, not a NaN. Test for their absence rather than for a sentinel value.

A point goes unplaced when it lands on sky, falls outside the depth map, or the map was not back-projectable in the first place.

Python

for keypoints in obj.get("keyPoints", []):
    for point in keypoints.get("points", []):
        if point.get("worldZ") is not None:
            print(point["worldX"], point["worldY"], point["worldZ"])

Node

for (const keypoints of obj.keyPoints ?? []) {
    for (const point of keypoints.points ?? []) {
        if (point.worldZ !== undefined) {
            console.log(point.worldX, point.worldY, point.worldZ)
        }
    }
}

z and worldZ are unrelated. z is model-relative depth in whatever convention the model uses — what a 3D body-points model emits, for instance. worldZ is meters. They are not convertible into one another.

Which frame the meters are in depends on whether the calibration supplied a pose — see Which frame the meters are in. Nothing in the prediction records it, so your code has to know which it asked for.

Point clouds

An object with a segmentation mask carries a dense cloud too: one xyz triple per mask pixel, laid out row-major on the bitmap's own grid, with NaN marking a pixel that could not be placed.

Python

from eyepop import PointCloud

cloud = PointCloud.from_object(obj)
if cloud is not None:
    print(cloud.array.shape)     # (height, width, 3), float32, NaN where unplaced
    print(cloud.at(0, 0))        # by cloud pixel, or None
    print(cloud.at_source(x, y)) # by source coordinate inside the object's box
    print(cloud.placed_points)   # (N, 3), holes dropped
    print(cloud.bounds)          # per-axis (min, max) in meters, or None

Node

import { cloudOfObject } from '@eyepop.ai/eyepop'

const cloud = cloudOfObject(obj)
if (cloud) {
    console.log(cloud.at(0, 0))        // by cloud pixel, or undefined
    console.log(cloud.atSource(x, y))  // by source coordinate inside the object's box
    console.log(cloud.placedPoints)    // holes dropped
    console.log(cloud.bounds)          // per-axis min/max in meters, or undefined
}

placed_points / placedPoints is usually what you want: a mask covers its object's whole bounding box, so anything not rectangular is mostly NaN.

The whole scene

Asking for depthMap.toWorld returns a cloud of the entire scene rather than one per segmented object, on the depth map's own grid.

Python

cloud = PointCloud.from_depth(
    prediction["depth"], prediction["source_width"], prediction["source_height"]
)

# every cloud in one prediction, nested objects included, scene cloud last
clouds = PointCloud.from_prediction(prediction)

Node

import { cloudOfDepth, cloudsOfPrediction } from '@eyepop.ai/eyepop'

const cloud = cloudOfDepth(prediction.depth, prediction.source_width, prediction.source_height)

// every cloud in one prediction, nested objects included, scene cloud last
const clouds = cloudsOfPrediction(prediction)

The scene cloud is indexed exactly like the depth map's values: same grid, same order, so a pixel's point and the depth it came from share an index. Both are sent, because a NaN point says only that the pixel could not be placed — the value at that index says why, whether +Infinity for sky or a reading that was never a distance.

A prediction can hold many clouds — one per masked object, plus the scene — which is why from_prediction / cloudsOfPrediction return a list while DepthMap.from_prediction returns a single map. A depth map is frame-level; a cloud belongs to one object.

Next steps

• Visualizing in 3D — plotting these coordinates

• Camera Calibration — what makes the meters trustworthy

• Components — where toWorld sits among the component attributes