From failure to the next candidate

A failed run can start
a better-informed next run.

Recover the process, protect the current release and turn reviewed failures into useful training or configuration changes.

01 / CAPTURE

A run or evaluation fails

Record the failure, affected inputs and artifacts. Preserve the original run as evidence.

run_id · failure type · evidence refs
02 / QUALIFY

Decide whether to react

A trigger you set up picks which failures count. Duplicate events are ignored and the usual checks apply.

event id · policy · retry limit
03 / DIAGNOSE

Choose the right response

Retry a transient failure, correct invalid data or propose a model/configuration candidate.

rules you write · limited investigation
04 / BUILD

Create a new candidate

A separate declared pipeline prepares corrected inputs or runs a targeted improvement.

parent run · dataset revision · candidate
05 / EVALUATE

Test the proposed improvement

Compare against a baseline and held-out criteria. A regression blocks advancement.

quality · cost · safety · baseline
06 / DECIDE

Approve, reject or escalate

Someone with approval rights can advance a passing candidate. Otherwise retain the current version.

approval · promotion · rollback evidence
Example / demand forecasting

A forecast misses the quality threshold.

Evidence from the failed evaluation starts a declared review-and-improvement pipeline. The current approved forecast remains in use while a candidate is tested.

Ready for review, not automatically released.

The candidate meets the configured evaluation criteria. A reviewer with approval rights decides whether to publish it; the evidence links back to the failed run.

Deduplicate failure eventsBound attempts and spendKeep held-out evaluationPreserve approval requirementsEscalate repeated failure

Interactive illustration, not a live self-healing system. Trigger wiring and recovery behavior must be verified for the selected pipeline and backend. Retrying infrastructure is not the same as improving a model. The workflow does not rewrite itself or relax its own policy.

Run it on your own infrastructure.

Apache 2.0 licensed, self-hosted. The quickstart gets a control plane with demo data running; a call gets your first pipeline onto it.