Datasets And Evaluation

Build datasets and score an ability against ground truth

An evaluation runs a configured model — an ability, in CLI commands — against the accepted media in a dataset and compares its results with ground truth.

Run An Evaluation

eyepop get datasets
eyepop evaluate --ability my-namespace.find-kittens:latest --dataset people

Evaluation is asynchronous. The CLI waits up to 20 seconds — raise --timeout to wait longer; a lower value shortens only the server's own wait, since the CLI keeps polling for at least 20 seconds — and prints the metrics if the run finishes in that window. Otherwise it prints a request ID to check on:

eyepop get evals <request_id>
eyepop get evals <request_id> --watch

Narrow What Gets Scored

Score one partition or restrict the ground-truth classes:

eyepop evaluate --ability my-namespace.find-kittens:latest --dataset people \
  --partition test \
  --filter-class person

Repeat --partition or --filter-class to select more than one. Run eyepop evaluate --help for video and timeout options.

Build A Dataset

eyepop create dataset --name people
eyepop create asset --dataset people --media-path ./images --recursive --partition test
eyepop get assets --dataset people

A partition is assigned at upload, so pass --partition when you add the assets you plan to score. Uploading media does not create ground truth — annotate the assets before evaluating.

A dataset is addressed by name, by UUID, or by name with a version suffix such as people@3 or people@latest. To look at one version:

eyepop get datasets people --version 3

Next steps

• Run Inference & Evaluations — the quick path to a single evaluation

• Concepts — datasets, assets, ground truth, and partitions

• Command Reference — every evaluate and dataset flag