Where a human approval belongs in an ML pipeline, and where it doesn't

Most ML pipelines get human approval wrong in one of two directions. Some have none: the nightly job trains, evaluates and promotes, and nobody is asked anything. Others put a person everywhere, until “approve” is a reflex and the queue is something people clear rather than read.

Our position is simple. A person decides what only a person can judge, in front of the step that spends money or changes production, with the evidence on screen. Everything a number can decide, a number decides. Cairn’s gates are built around that rule. This post explains how, and what we deliberately keep away from approvers.

Two kinds of gate, never both

A gate in a Cairn pipeline is a leg of its own, and it is answered one of two ways:

• By data, with predicate:. The expression is checked the moment its inputs exist. True, and the gate passes instantly with the verdict data. False, and the pipeline stops with the reason. Nobody is asked, so nobody can approve a run past it.
• By a person: any gate without a predicate. The pipeline pauses until someone decides. Who is asked comes from the gate’s approvers:, or the pipeline’s default.

A gate can’t be both. A file that declares both on one gate is refused when it’s parsed: “a gate is answered by data OR a human, never both. Split into two gates if you need both verdicts.” The reason is in the code next to that check: a predicate gate resolves instantly and never enters the approval queue, so pairing it with approvers would silently drop the human. And a person gate has no field that approves it automatically.

The data gate came in through ADR-0087 (Amendment 1). It stops the pipeline rather than skipping a leg, because turning a failed check into a silent skip would be fail-open.

Rule 1: if a number can decide it, use a predicate

A person asked “is the estimate under $5?” adds delay, not judgement. The serving/batch-inference pipeline puts a data gate in front of the job:

- gate: cost
  depends_on: [estimate]
  predicate: steps.estimate.usd <= input.max_cost_usd

training/retrain-cycle goes further. Before its promotion gate, four data gates have run: enough fresh rows after deduplication (enough-signal), both measurements valid (measurement-valid), and then these two:

- gate: no-regression
  depends_on: [delta]
  predicate: steps.delta.verdict == 'pass' || steps.delta.verdict == 'warn'

- gate: no-forgetting
  depends_on: [forgetting]
  predicate: steps.forgetting.passed

Note how no-regression is written: as a list of verdicts that may pass, not != 'fail'. The comment in the file explains why: a missing verdict isn’t equal to 'fail' either, so the shorter form would open the gate when the data is missing. A predicate should fail closed.

By the time the promotion gate fires, the questions a number can answer have already been answered. The person is left with the one that needs judgement.

Rule 2: put the person in front of the spend, not after it

An approval only matters if it comes before the thing it approves. In dataset-readiness/pull-and-admit, the gate sits before the leg that reads a production database with real credentials, not after it. The comment there says it plainly: approving a read that already happened is theatre.

- gate: approve-pull
  show:
    - input.connection
    - input.table
    - input.cursor_field
    - input.max_rows

- id: pull
  task: ingest/pull
  depends_on: [approve-pull]

The same goes for promotion: in training/finetune, the step that registers the model is the one that waits for a person. Cairn also checks for the gap. cairn validate and cairn pipeline explain warn about any leg whose flow can write externally and has no approval gate on the path in front of it.

Rule 3: an approver who can’t see the evidence is guessing

A gate’s show: list names the values the approver sees, taken from the outputs of earlier legs. Each entry must be a <step>.<field> reference, and that’s checked when the file is parsed, not when an approver opens the gate.

retrain-cycle’s promotion gate shows the old and new model’s scores side by side on every split, the regression and forgetting verdicts, the replay mix, and the judge model used. It also shows the abstention rate as a pair, and the comment explains why that one is shown rather than gated: abstaining more is good if the alternative was a confident wrong answer and bad if it was a right one. No predicate can tell those apart. That’s exactly the kind of call a person should make.

Next to the show: values, the console’s Approvals panel shows what the pipeline has spent against its budget, and the admission decision made at launch.

Rule 4: a pause is a state, not a process

Approvals take hours, sometimes days. A gate that only lives in a running process is lost on the next deploy. In Cairn the pipeline record holds the plan, every leg’s state and the cursor, and that record is what owns the pipeline. On restart, the bridge picks it up again and the waiting gate is still waiting.

The notification is durable too: each pause is announced once, even across a restart.

Rule 5: the approver is whoever signed in

When a gate is decided through the API with authentication on, the approver recorded is the identity behind the token. An approver name in the request body is accepted for older clients but ignored, and a mismatch is logged. So the audit trail says who actually clicked, not who someone typed.

Rule 6: some gates belong to the environment, not the author

A pipeline author can’t be the only one who decides where people sit. A deployment policy whose applies_to.pipeline_legs patterns match a leg acts when the run is launched, before any record exists:

• require_hitl inserts an approval gate immediately before each matching leg. The pipeline author can’t remove it, because it isn’t in their file. The gate carries the name of the policy that added it, so the approver can see why it’s there.
• deny refuses the launch outright and names the policy.

If a person already approves upstream of a matching leg, no second gate is added. One decision covers it. This came in as part of ADR-0087 (phase P0b).

What we keep away from approvers

• Anything a predicate can decide. Cost under a cap, rows over a minimum, a verdict from a deterministic check.
• Waits that aren’t waiting on a person. A step waiting for a schedule or an event doesn’t notify anyone. Only the two human waits do. The comment explains why: paging people for the other kinds would train them to ignore the channel.
• The same decision twice. Policy gates don’t double up on an approval that’s already on the path.
• Decisions after the fact. If the work has already happened, the approval is a record, not a gate.

Approval fatigue isn’t a discipline problem. It’s a design problem. Each extra pause makes every pause count for less. Spend them where a person adds something a number can’t.

What to do next

Pick the pipeline that worries you most and run cairn pipeline explain <pack>/<name>. It lists every gate on each leg’s path and who decides it, so you can see where people sit and whether a predicate could do the job. New to Cairn? Start with the quickstart.