Design for Determinism

I was reading Rohit Anand's article on data contracts v2 recently. He makes the case that schema validation alone isn't enough. Freshness, distribution stability, and downstream impact all belong in a richer specification. The problem space is right.

But reading it pulled me toward a different question. Whether those concerns are solvable depends on something earlier in the design: was the system built to be reconstructable in the first place?

Most aren't. And the moment that gap becomes visible is usually a system upgrade.

A major upgrade completes. The new system is live. The migration team has done good work. The schema is correct, the logic has been ported, the outputs look right.

Then someone asks for reconciliation. Prove that the new system produces the same results the old system produced for the last submission.

And it can't be done. Not from any fault in the migration. The prior state cannot be recreated. The inputs the old system read aren't versioned. The source tables have moved on. The reference data it joined to has been updated. The configuration files that governed its behaviour have changed. No one captured exactly which version of each of these was live at the time.

This isn't only a regulated-environment problem. Any engineer who has tried to reconstruct what a pipeline saw during a failed run has felt a version of this. The migration didn't create the gap. It made a pre-existing design gap impossible to ignore.

The system was designed to produce outputs. Not to reproduce them. That's the gap.

A system designed for determinism can do both. Determinism, in this context, means: known inputs, pinned logic, known reference state, reconstructable output. The same inputs, run through the same logic against the same reference data, always produce the same result. And that result can be proved.

Most pipelines aren't built this way. Inputs aren't snapshotted. Reference data changes silently between runs. The transformation logic is tied to whatever is deployed at the time, not to a specific release. Incremental state is merged forward without a recoverable history. There is no mechanism for point-in-time reconstruction.

A deterministic system isn't automatically correct. It makes correctness much easier to prove.

Designing for determinism costs more. That's the honest version of this argument. It costs more at design time. More at build time. More in the early discipline of snapshotting inputs, pinning transforms to releases, and versioning reference data.

The argument isn't that it's free. The argument is that the alternative costs more, and delivers less.

When a system is built for determinism, the evidence base exists as a product of how the system operates. Immutable inputs where the risk justifies them. Versioned reference data. Logic tied to a specific release. Lineage that records which sources, which transforms, and which outputs belonged to which run. These aren't governance add-ons. They're the properties that make the system reconstructable. The evidence base is a by-product.

This is the standard engineering argument: design it right, spend the time upfront, end up with a better product. Governance is no different. Design for provable correctness, and the evidence needed to prove correctness exists naturally. Review, approval, ownership, and exception management still need explicit design, but they attach to real system facts rather than reconstructed stories.

If you don't build for determinism, you pay to build the system. Then you pay again to bolt governance onto something that was never designed to support it.

Determinism exists on a spectrum. My context sits at the formal end: regulated reporting environments where inputs, logic, approvals, and releases all need to be traceable. Most teams don't need the full model, and they shouldn't be made to feel they do.

But there are steps toward more determinism that pay back in contexts far less formal than mine.

Start with the inputs. If a pipeline reads live from a source table, it sees whatever that table contains at the time of the run. Two runs on different days may see different data. If something goes wrong, reconstructing what the pipeline actually read requires trusting that the source hasn't changed, and it may have. Snapshotting inputs at the start of a run, even to a staging layer, changes this. You can look at what was actually processed.

Then there's the transform code itself. Most pipelines run whatever version of the code happens to be deployed. That makes "which logic produced this output?" a harder question than it should be. Pinning execution to a specific release gives it a clean answer.

The same principle applies to reference data: the mapping tables, dimension lookups, threshold values, and configuration files that govern how data is transformed. If these are versioned alongside the pipeline, a historical run becomes reconstructable. If they aren't, the answer to "which version of this mapping was active last quarter?" is usually a shrug.

Then there's lineage. The lightest version is recording, at each run, which source tables were read, which transforms ran, and which outputs were written. Most orchestration tools capture some of this already. Treating it as a first-class artefact rather than an operational log changes what you can do with it. It's the bridge between "what happened?" and "what was supposed to happen?"

The cost comes down to timing.

A system built for determinism carries its governance evidence naturally. The audit trail is a by-product of how it runs. The lineage is a release artefact, not a reconstruction. The transforms were pinned. The inputs were snapshotted. The system was designed to answer the question.

A system that wasn't pays twice: once to build, and once to bolt governance onto something that was never designed to support it. That layer is expensive, fragile, and always slightly wrong. It's reconstructing what was never recorded. It drifts from reality. It requires maintenance that never quite happens.

Where in your pipeline could you design for more determinism? Not as a compliance exercise. As a correctness one.