From a DAG of scripts to a governed pipeline

Many ML teams already have a pipeline. It’s a DAG: a scheduler starts a data-prep script, then a training job, then an evaluation script, and someone checks a dashboard before the model ships. That works until someone asks one of these questions:

• What did that run cost?
• Who approved the release?
• Which data went in?
• Why did last night’s job start a GPU on a dataset that had failed its checks?

This post is for teams in that position, whether the DAG runs on Airflow, Kubeflow or a cron file. It covers what Cairn adds, and what it leaves alone.

What Cairn doesn’t replace

Your engines still run the work. In Cairn, a flow leg goes to the engine your environment names: the built-in runner on a local process or an SSH box, Metaflow, Prefect or SkyPilot. Training runs where it ran before, on your GPUs. Your data stays in your storage, such as S3, MinIO, GCS or your own databases.

Cairn doesn’t import your DAG. There is no converter. You describe the pipeline in a short YAML file, and the code your steps run becomes a flow stage or an operator in a pack. cairn init operator <name> scaffolds one. It’s a deliberate rewrite of the orchestration layer, usually small, and the step code stays the same.

What it adds

Cairn’s vocabulary is pipeline → leg → flow → step. A pipeline is the unit you launch. Each leg is one governed unit of work. A leg can be a flow (a list of steps that runs on an engine), a single operator task, an agentic template, or a gate. Here is a fine-tune pipeline from the training pack, shortened:

name: e2e-finetune
kind: pipeline
inputs:
  dataset: {type: Dataset}
  base_model: {type: str, default: "Qwen/Qwen2.5-0.5B"}
governance:
  budget: {usd: 300}
  series: e2e-finetune
  approvers: ml-leads
steps:
  - id: readiness
    flow: readiness                 # data leg, on the environment's engine
    with: {raw: $input.dataset}
    consumes: {raw: Dataset}
    produces: {export: Dataset}
  - id: train
    flow: sft-train                 # ML leg, on the environment's engine
    depends_on: [readiness]
    with:
      dataset_uri: readiness.export
      base_model: $input.base_model
    consumes: {dataset_uri: Dataset}
    produces: {adapter: Model, metrics: Report}
  - gate: promote-model             # a person decides, with the evidence in front of them
    depends_on: [train]
    show: [readiness.rows, train.metrics]
outputs:
  model_uri: train.adapter

Compared with a plain DAG, that file gives you the following.

One record per leg. Every leg runs as its own run, with its own admission, its own budget hold and its own approvers. Its inputs, outputs, artifacts and cost are recorded where the leg ran. A gate leg records a verdict, or the decision a person made.

Typed wires, checked before anything runs. consumes and produces declare what each leg passes on. If a Dataset is wired where a Model is expected, the plan is refused. You find out before the job starts, not after.

Cost caps that apply before money is spent. The governance block gives the pipeline a budget and a series. When a run is admitted, its estimate is reserved against that budget. If what’s been spent plus the estimate would go over the cap, the run is refused.

Gates of two kinds. A data gate is a predicate that is checked as soon as its inputs exist, so nobody can approve a run past it. A human gate waits in the console’s Approvals list, and the approver sees the values listed under show. Policy can also add gates when a run is launched. A deny refuses the launch before any record exists.

Clean-up that always runs. A leg marked finalizer: true runs whether the pipeline finishes, fails or is cancelled. For example, it can cancel a rented GPU job. when: lets one plan include alternative legs, such as one for each provider, without hiding which one ran.

Triggers on the pipeline. A schedule is declared on the pipeline (on: [{cron: "0 3 * * *"}]). Enabling the pipeline turns the schedule on, and disabling it turns it off.

A trace you can read back. cairn replay <run-id> shows a past run’s timeline from its record. It is read-only. cairn lineage <subject> reads the hash-chained provenance record, checking the chain as it goes, and points to the run that produced it.

Before you spend anything

The habit that changes most is checking what a launch will do before it does it:

cairn pipeline show training/e2e-finetune      # legs, gates, wires, and the runtime for each leg
cairn pipeline explain training/e2e-finetune   # the engine, every gate and who decides it, where each value comes from
cairn pipeline run training/e2e-finetune -i dataset=<your-dataset>

The run you start from the CLI is the same governed object as one started from the console. It goes through the same launch path, with the same gates and the same record.

Where to start

Pick the DAG that costs the most or worries you most, usually the one that ends in a GPU or a release. Move that one first. Keep the scripts. Put a data gate in front of the expensive step and a person in front of the release. Then compare: when someone next asks “what did that cost, and who said yes?”, the answer is in the run record, not in a Slack thread.