ETL · a self-healing demo

Back to risshabh.com

Self-healing ETL

A pipeline that survives a renamed or retyped column — remap when sure, hold when not

Rename or retype an upstream column below and watch the staged engine score it against its confidence floor: remap when sure, hold the batch for review when not, and log every decision.

Staged scenario — illustrates the decision logic, not measured results. This project keeps no outcome numbers.

Staged scenario — illustrates the decision logic, not measured results.

Type a column name above, or pick a staged example. The engine scores it live.

  1. No decisions yet — the log fills as the engine runs.

Scores are staged for this page. The rule is the real one: remap at or above 0.92, hold below it, log every decision.

How this works

A pipeline breaks the usual way: an upstream source renames or retypes a column and everything downstream quietly corrupts. This engine maps each upstream column onto a master schema through exact, learned, and embedding matching behind a 0.92 confidence floor.

At or above the floor it remaps the column itself. Below the floor it holds the batch for review instead of guessing, so bad data stops rather than propagates. Every mapping decision and its confidence goes to a replayable audit trail — the log on this page is its stand-in. The scoring here is a small lexical stand-in written for the page; the floor and the rule are the real ones.

Limits

  • Staged demonstration with synthetic confidences, not measured results.
  • It makes no claims about speed or correctness: this project keeps no outcome numbers.
  • Built with Python, Polars, PostgreSQL, and dbt.

Project page · risshabh.com

How this works

What this page shows

A live mini-pipeline: you edit an upstream column name and type, and the staged engine scores it against every master-schema column at once, compares the best score against a 0.92 floor, and decides. Three staged examples (a rename, a retype, an ambiguous column) fill the editor in one tap; running the engine appends the decision to the visible log.

The rule it demonstrates

Map each upstream column onto a master schema through exact, learned, and embedding matching behind a 0.92 confidence floor. Remap the column itself when sure; hold the batch for review when not, so bad data stops rather than propagates. Record every mapping decision and its confidence to a replayable audit trail.

What it does not claim

It makes no claims about speed or correctness: the scenarios and confidences are staged, and this project keeps no outcome numbers. Built with Python, Polars, PostgreSQL, and dbt.