Visualizing in 3D

Plotting world coordinates as a 3D scene

A depth heatmap is still a picture of the frame. World coordinates are real positions, so they are worth looking at as a scene — which is what makes a bad calibration obvious at a glance.

Python

EyePopWorldPlot scatters everything in a prediction that carries world coordinates into a 3D axes: key points, outlines, contours, cutouts and mask point clouds alike.

import matplotlib.pyplot as plt
from eyepop.visualize import EyePopWorldPlot

axes = plt.figure().add_subplot(projection="3d")
plot = EyePopWorldPlot(axes)
plot.prediction(prediction)
plot.finish()
plt.show()

It needs a 3D axes — add_subplot(projection="3d") — because these are real 3D positions rather than an overlay on the frame, which is what separates it from EyePopPlot. Call plot.point_clouds(prediction) instead for the mask clouds alone.

Key points from the four nearest people, back-projected to meters and joined into skeletons. They stand between 2.1 and 5.6 meters from the camera, and the plot preserves that spacing. Source photo: Sint Antoniesbreestraat, Amsterdam, by Fons Heijnsbroek (CC0).

Each carrier gets its own colour and legend entry, and the ones with a defined order are connected:

• contours, outlines and cutouts — closed paths in their own point order

• key points — the pose skeleton, matched on class label from the same table the 2D renderer uses

• mask clouds — left unconnected, being a grid rather than a path

Two cases would otherwise draw a plausible lie, so neither does. An unplaced point breaks the path rather than being bridged across a gap that is not there. And a key point group whose category is not a known pose gets no lines at all, rather than its points joined in arrival order.

Skeletons are matched on the key point group's category, which comes from the component's categoryName. Give the pose component categoryName="2d-body-points" (or "3d-body-points") and its joints are connected; leave it unset and the points are drawn but not joined, because nothing identifies which pose they belong to.

Reading the axes

Z is drawn into the scene and Y up the page, rather than in component order, because depth is what reads as distance from the viewer in either frame. The coordinates themselves are never altered — only which axis each component is drawn against.

The vertical axis is also flipped by default, so a camera-frame scene stands the right way up. Y grows downwards in the OpenCV camera frame, so drawn as-is a figure's feet sit above its head. That is the default because a source supplying no extrinsics — or identity ones — gets the camera frame. For coordinates already in a Z-up world frame, pass EyePopWorldPlot(axes, invert_y=False).

The point budget

A mask cloud is one point per mask pixel, so a few objects at a few hundred pixels square is already more than a scatter can usefully draw. The budget is shared across series, so adding objects thins the scatter rather than multiplying it, and sampling is strided rather than random, so the same call draws the same picture twice.

Sparse series are exempt. A 17-point skeleton and a 40,000-point mask cloud are both series, and a budget split evenly between them would thin the skeleton away to save a fraction of the cloud. Sparse ones are drawn whole and larger, dense ones strided into what is left.

Node

The Node SDK renders depth in 2D through Render2d.renderDepth(), but ships no 3D plot — a browser has better tools for that than a canvas helper.

For a worked example, examples/webpack/src/world-demo.html in the Node SDK repository draws the same four carriers with three.js: a live webcam, a calibration panel, skeletons, contours and point clouds in an orbit-controlled scene with a Z-up axes helper and a ground grid at Z = 0.

Next steps

• World Coordinates — the members being plotted

• Camera Calibration — which frame the scene is drawn in