Fine-tune with a cost gate

A fine-tune usually goes wrong in one of two places. Either the run costs far more than expected, or doesn’t fit the GPU at all and fails after the money is spent. Or the run finishes, and the model gets promoted before anyone has checked it properly.

Cairn’s training/finetune pipeline adds a check at both points. This guide walks through each step and the commands that run it.

The shape of the pipeline

training/finetune is a pipeline with one leg, and that leg runs the agentic finetune flow. The pipeline file is short. It declares only what the flow can’t decide for itself:

name: finetune
kind: pipeline
inputs:
  $inherit: run          # take the flow's own input contract
governance:
  budget: {usd: 40}
  series: finetune
  approvers: ml-leads
steps:
  - id: run
    template: finetune

Inside the flow, the steps run in this order:

Dataset readiness: the dataset-readiness check runs first. If its verdict is fail, nothing after it runs: no estimate, no GPU, no training. The run ends with a report saying why it was blocked.
Cost estimate → cost gate → audit: Cairn prices the run before any GPU is provisioned.
Train: a QLoRA adapter.
Actual cost: the measured training time is priced with the same table the estimate used, so the money is recorded and not left at zero.
Model gate: the trained adapter is checked against held-out metrics.
Promote: this step pauses until a person approves it.
Lineage, model card, report: one record from dataset to promotion.

Pricing the run before a GPU starts

The estimate step (compute/estimate) uses the base model, the method, the number of rows, the epochs and the GPU type. From those it works out:

the VRAM the run needs
the VRAM the GPU has
whether the run fits
the GPU hours
the dollar cost
the assumptions behind the figures

The cost gate (cost/gate) turns those figures into a pass, warn or fail verdict with reasons:

A run that won’t fit the GPU fails. This is on by default.
An estimate above max_usd fails, and one above warn_usd warns.
No usable estimate is a warning, never a silent pass.

The thresholds belong to the deployment, not to the run. You set them once in the environment’s profile under [profile.config."cost/gate".thresholds]. Both default to 0, which means no dollar cap until you set one. A run can’t loosen them. Every verdict is written to a hash-chained audit log.

The money is also checked when the run is admitted. The pipeline declares a $40 budget for the finetune series. When a run is admitted, Cairn reserves its estimate against that budget in one atomic step. If what has already been spent plus this estimate would go over the cap, the run is refused before it starts.

Making cost a hard stop in your own pipeline

Gates in a pipeline come in two kinds: one asks a person, the other checks data. A data gate is decided by a predicate as soon as its inputs exist. Nobody approves it, so nobody can approve a run past it. This example comes from the serving pack’s batch-inference pipeline, which prices a job before any token is spent:

steps:
  - id: estimate
    task: cost/tokens
    with:                             # abridged
      model: $input.model
      rows: $input.rows
      prompt_tokens: $input.est_prompt_tokens
      completion_tokens: $input.max_tokens   # priced as if every row uses the full allowance

  - gate: cost                        # data, not a person
    depends_on: [estimate]
    predicate: steps.estimate.usd <= input.max_cost_usd

  - id: infer
    flow: batch-infer
    depends_on: [cost]

An estimate above max_cost_usd stops the pipeline before the job starts. cost/tokens refuses to price a model it doesn’t know rather than reporting $0, so the gate can’t pass by accident. The same pattern puts a hard cost stop in front of the training leg of your own fine-tune pipeline.

Promotion needs both a passing check and a person

The model gate (model/gate) gives a fixed pass, warn or fail on the adapter’s held-out metrics, and lists the reasons. You can set stricter bars with model_gate_thresholds, for example minimum scores on the public benchmark if you turned it on. If you supply the scores of the model you ship today, a quality regression against that model fails the promotion.

Promotion itself is model/register, which requires approval. The run pauses at that point, and the pause survives a restart, until a person approves or rejects it. The approver sees the model gate’s verdict. If the promotion is rejected, the adapter still exists, but it doesn’t ship.

Running it

List what’s installed, then see what a launch would do before anything is spent:

cairn pipeline ls
cairn pipeline show training/finetune
cairn pipeline explain training/finetune \
  -i dataset_path=databricks/databricks-dolly-15k \
  -i base_model=HuggingFaceTB/SmolLM2-135M

explain does a dry run of how each step will be triggered: which engine runs it, every gate on the path and who decides it, and where each setting comes from.

Then launch it:

cairn pipeline run training/finetune \
  -i dataset_path=databricks/databricks-dolly-15k \
  -i base_model=HuggingFaceTB/SmolLM2-135M \
  -i dataset_rows=1000 -i gpu=A100

The command stays attached and prints each leg as it changes state. When the run reaches the promotion step, decide it in the console under Approvals, or from a terminal connected to the server:

cairn remote approve <run-id>

Afterwards, read the provenance record and the run’s trace:

cairn lineage <output-name>   # dataset → config → adapter → verdicts → cost → promotion
cairn replay <run-id>         # the run's timeline, read-only

In the console, the launch dialog’s Try with sample data has three ready-made sizes, from Dolly-15k on SmolLM2-135M up to UltraChat on Qwen2.5-7B. Use them to try the whole path before you point it at your own data.