Choose the harness
Benchmark suites, or a closed-book QA set with per-split correct, hallucinated and abstained.
ML · ships today
Evaluate and
compare models
Score a model against your suites and flag what regressed against a baseline, direction-aware.
Ships as eval/score-a-model · training/closedbook-compare
Evaluation is the gate behind every other pipeline. Score a model on your benchmark suites, or compare a champion and a challenger on the same harness, and get a verdict a person can act on.
The pipeline
Score
Run the model, and the baseline if you supplied one, on the same questions.
Check the measurement
A gate confirms the comparison is valid before any verdict is read.
Verdict
eval/delta asks whether it gained; eval/forgetting asks whether it lost what the champion knew. A regression blocks.
What the record keeps
The models, the suite or QA set, per-split scores, the delta and forgetting verdicts, and the accept or reject decision.
Direction-aware
Lower-is-better metrics are not read backwards.
Forgetting is measured
A challenger that learned new facts but lost old ones is caught.
Measures only
Nothing is promoted or registered here; this pipeline produces evidence.
Ships as eval/score-a-model and training/closedbook-compare. Suites and judges depend on the installed operators.
See how failure can trigger a reviewed improvement cycle →Run it on your own infrastructure.
Apache 2.0 licensed, self-hosted. cairn pipeline run eval/score-a-model