Camera Calibration

Describing the lens and the pose that turn a depth map into measurable meters

A depth map on its own is a picture of distance. Turning it into positions you can measure takes a calibration: what the lens does to the scene, and where the camera stands in it.

Without one the worker assumes a 60° horizontal field of view. That is a development scaffold, not something to ship — and it fails in a way worth understanding, because it is quiet. For canonical metric depth the assumed focal length cancels out of X and Y and survives only in Z, so lateral measurements stay exact while every distance along the optical axis is wrong by however wrong the guess was. Two people standing side by side measure the right distance apart; how far away they are is fiction.

The lens

Exactly one of intrinsics and hfovDegrees describes the lens. Supplying both is rejected rather than resolved by precedence — two descriptions of one lens that disagree have no right answer — and supplying neither is rejected too, since defaulting a focal length would be inventing a lens.

Field	Type	Description
intrinsics.fx	float	Focal length as a fraction of the frame's width. Positive and finite.
intrinsics.fy	float	Focal length as a fraction of the frame's height. Positive and finite.
intrinsics.cx	float	Principal point across the frame, in [0, 1]. A centred lens is 0.5.
intrinsics.cy	float	Principal point down the frame, in [0, 1]. A centred lens is 0.5.
hfovDegrees	float	Horizontal field of view, in (0, 180). The shorthand alternative to intrinsics.

Intrinsics are normalised to the frame, not given in pixels. A camera calibrated at 1920x1080 and later streamed at 1280x720 keeps working. Pixel values carrying no calibration resolution are wrong by exactly the resolution ratio, and nothing downstream can detect it.

To convert an OpenCV matrix, divide by the resolution it was calibrated at: fx_normalised = fx_pixels / width, cy_normalised = cy_pixels / height.

hfovDegrees is for an operator who knows their lens's field of view but not its calibration matrix. It assumes square pixels and a centred principal point, so it approximates a real calibration rather than replacing one — but it describes the actual lens rather than the worker's fallback guess, and it composes with distortion and pose exactly as intrinsics do.

Distortion

distortion carries the OpenCV Brown-Conrady coefficients — k1, k2, p1, p2, k3, each defaulting to 0.0. All zero is a rectilinear lens, which is also what you get by omitting the field. Supply them for a wide-angle or fisheye lens, where straight lines in the scene do not arrive straight.

The pose

extrinsics places the camera in the world. Omit it and the meters come back in the camera's own frame.

Field	Type	Description
extrinsics.rotation	object	A unit quaternion, w first: { w, x, y, z }, defaulting to the identity { w: 1 }.
extrinsics.translation	object	Where the camera sits, in meters: { x, y, z }, defaulting to the origin.

The pose is camera to world:

P_world = R(rotation) · P_camera + translation

so the rotation carries the camera's axes onto the world's, and the translation is the camera's own position.

This is the inverse of what cv2.solvePnP returns. Holding its rvec / tvec, invert both halves — R = R_cv.T and t = -R_cv.T @ t_cv — then convert R to a quaternion. tvec is not the camera position, and passing it as translation puts the camera somewhere it never was.

Either half means something on its own: a rotation with no translation is a camera at the world origin, a translation with no rotation is one looking along the world axes.

Which frame the meters are in

This is the single thing to get straight, because a prediction does not say which frame it holds — and the two are oriented differently.

	Frame	Axes
With extrinsics	The world frame the pose defines	Z up, ground plane at Z = 0
Without extrinsics	The camera frame, OpenCV convention	X right, Y down, Z forward

A source that supplies no pose — or an identity one — gets the camera frame, so Y growing downwards is the common case rather than the exotic one. Anything that draws these coordinates has to know which it is looking at, or it renders the scene upside down.

Supplying it

A calibration belongs to a source. Give it per source, or once on the Pop's defaults for every source that Pop processes.

Python

from eyepop.worker.camera import Camera, CameraExtrinsics, CameraIntrinsics, Quaternion, Vector3d

camera = Camera(
    intrinsics=CameraIntrinsics(fx=0.9, fy=1.6, cx=0.5, cy=0.5),
    extrinsics=CameraExtrinsics(
        rotation=Quaternion(w=0.5736, x=-0.8192, y=0.0, z=0.0),
        translation=Vector3d(x=0.0, y=0.0, z=3.0),
    ),
)

job = endpoint.load_from("rtsp://camera.example.com/stream1", camera=camera)

The lens shorthand is Camera(hfovDegrees=72.0).

Node

const results = await endpoint.process({
    source: { url: 'rtsp://camera.example.com/stream1' },
    camera: {
        intrinsics: { fx: 0.9, fy: 1.6, cx: 0.5, cy: 0.5 },
        extrinsics: {
            rotation: { w: 0.5736, x: -0.8192, y: 0, z: 0 },
            translation: { x: 0, y: 0, z: 3 },
        },
    },
})

The lens shorthand is camera: { hfovDegrees: 72 }.

On the Pop, the same object goes under defaults:

Python

from eyepop.worker.worker_types import Pop, SourceDefaults

pop = Pop(
    components=[...],
    defaults=SourceDefaults(camera=Camera(hfovDegrees=72.0)),
)

Node

const pop = {
    components: [...],
    defaults: { camera: { hfovDegrees: 72 } },
}

Defaults merge field by field, so a source giving its own roi but no camera keeps its roi and takes the Pop's camera. The camera itself merges as one whole field: a source declaring its own lens replaces the defaulted calibration outright rather than mixing the two. There is no way to inherit a default pose and override only the lens.

With a region of interest

A roi crops the frame before inference, but a calibration always describes the source as captured — the original media resolution, not the crop. Intrinsics are normalised against the full frame, and the platform accounts for the crop itself. Do not pre-adjust a calibration to match a region of interest.

This is also why you should not recompute world coordinates yourself from a prediction's x / y and your own calibration. Under an roi the emitted coordinates are original-frame pixels while the projection happened against the cropped frame, so a hand-rolled back-projection is wrong in exactly the case where calibration matters most. Read the worldX / worldY / worldZ the platform returns.

Failing early

Both SDKs validate a calibration before the request leaves your process, so a malformed one raises locally — a ValidationError in Python, a thrown error in Node — rather than coming back as a 400 halfway through an upload. The worker validates authoritatively; the two are meant to agree.

What is checked: exactly one lens description; hfovDegrees inside (0, 180); fx and fy positive and finite; cx and cy within [0, 1]; every distortion coefficient finite; every translation component finite; and the rotation a unit quaternion, within 1e-3 of norm 1.0 so it survives a float round trip through JSON without letting an un-normalised one through.

Next steps

• World Coordinates — reading the meters a calibration produces

• Depth Maps — the map the coordinates are back-projected from

• Pop Defaults — setting one calibration for every source