Reducing Failure Latency in the System You Already Have

Article 5 ended with a practical question: how do you get from an existing system to one that works this way?

Five articles of argument can leave a wrong impression. Boundaries, contracts, gates, run records. Read together, they can look like a target architecture, the kind you'd build if you were starting again.

Most teams aren't starting again. They run a system that grew over years. Validation is scattered through it, some in code, some in scripts, some in the head of the person who checks the file every morning. Ownership is partial. Engineering capacity is spoken for.

This article is for that system. The claim is simple: you don't need a new platform to reduce failure latency. You need to reduce the distance between failure and awareness, one failure mode at a time.

The Rebuild Trap

The common response to an architecture argument is to scope a rebuild. The future platform will have proper boundaries. It will validate at source receipt, hold failures in named states, and produce a run record. It goes on the roadmap.

Then two things happen. The improvements get attached to the future platform, so nobody touches the current system. And the late failures keep arriving on schedule, in the system that actually runs.

A rebuild is a multi-year bet. A failure latency improvement is available this month, in the system you have.

There's a second problem with the rebuild path, and it's the more expensive one. Years of implicit validation live in the old system. A join that quietly drops malformed rows. A reconciliation script someone runs when the numbers look off. A person who knows that a small file on the fifth of the month means the upstream extract failed. None of that is written down. A rebuild has to rediscover all of it, usually by failing in production.

Incremental adoption isn't the compromise option. It's the path that works. The realistic path is not replacing the system. It's progressively shortening the distance between where failures exist and where they become visible.

Start With One Failure

Don't start with an inventory. Auditing every validation gap in a mature system produces a backlog, not an improvement. Backlogs of that size get prioritised into nothing.

Pick one recurring late failure instead. Every team can name one without looking. The month-end file that breaks the downstream load. The reference data that goes stale and nobody notices until the numbers drift. The feed that arrives malformed twice a quarter and costs a day each time.

Recurring matters. A one-off failure teaches less than a failure with a known shape and a known cost. You want one where the team can already describe how it goes wrong, what it breaks, and what the cleanup looks like. The failure everyone groans about is the right first candidate.

Map the Distance

With a failure chosen, map how it plays out today. Four questions, answered honestly:

Where does the failure exist? The earliest point the problem is present in the data. Often that's the moment a file lands, before anything has processed it.

Where does it become visible? The point where someone or something notices. A downstream job falls over. A report looks wrong. A consumer complains.

Who finds out? The actual person. And whether they're someone who can act on it, or someone who has to find the person who can.

What happens next? Rerun, patch, escalation, apology. The full cleanup, including the parts that don't make it into the incident ticket.

The gap between the first two answers is the failure latency for this failure mode. It's usually larger than the team expects. A structural problem that existed at 06:10 becomes visible as a reporting question at 16:40, and the map makes that gap explicit for the first time.

The last two answers expose the routing problem. Failures tend to surface to whoever is nearest the symptom, not whoever owns the cause. The person who can fix a malformed source file is rarely the person staring at the broken report.

Nobody designed that routing. In most systems there isn't any. The failure travels with the data until someone notices, and whoever notices owns it by accident.

This map is your baseline. The improvement gets measured against it.

The Earliest Useful Boundary

Article 2 made the argument: validate each thing at the earliest boundary where it can meaningfully run. Now apply it to the mapped failure.

Walk back from where the failure becomes visible towards where it exists. Somewhere along that path is the earliest boundary with enough context to detect it. For a structural problem, that's usually source receipt. For a reference data problem, it's the enrichment step. For a completeness problem, it might be the point where all expected sources have arrived.

Earliest useful, not earliest possible. A check needs the context to run meaningfully. There's no point checking referential integrity before the reference data is available to check against.

In an existing system, the boundary you need usually exists already as an implicit seam. The step after the file lands. The staging table load. The handover between the overnight jobs. You're not building a boundary. You're making a seam explicit and giving it a job.

The Smallest Check That Changes Behaviour

Now add the check. Small in scope, not in consequence.

The scope is one failure mode at one boundary. Not a validation framework. Not a data quality initiative. One check that detects the failure you mapped, at the seam you found. That's deliverable without a project, which is the point.

The consequence is where the discipline from article 5 applies. A check that logs a warning and lets the run continue hasn't changed anything. The failure still travels downstream and still surfaces in the same place to the same person. You've added observability to the old latency, not reduced it.

The check has to change what happens next. Stop the stage. Hold the source in a named state. Quarantine the affected rows. Route the failure to the person who can act, with a decision required before the run proceeds. Any of these will do. A check that does not change what happens next is not an improvement. It's a log line with ambitions.

And record what it does. What it found, what state it set, who was told. The run record from article 5 doesn't arrive in one release. It accretes, one consequential check at a time. Your first check is the first entry.

Capture First, Then Improve

One discipline is specific to existing systems, and it deserves its own section.

The system already validates. Implicitly, partially, and without records, but it does. That join dropping malformed rows is a validation rule. The morning eyeball check is a validation rule. The new explicit check doesn't land in a vacuum. It lands next to existing behaviour that nobody has ever written down.

So capture the existing behaviour first. Work out what the implicit validation actually catches, and write the explicit check to match it before you improve on it. Then run them side by side. The new check holds nothing yet. It just reports, and you compare what it catches against what the old behaviour catches. Parity first, then consequence.

This sounds slower than cutting over. It's slower the way checking the rope is slower than falling. The new check earns the right to govern the pipeline by demonstrating it sees what the old behaviour saw.

Duplicate execution is fine. Duplicate ownership is dangerous. Two checks running side by side is a safe state, and you can stay in it as long as you need. Two owners each assuming the other one handles the failure is how failures get dropped. When the new check takes over, move ownership deliberately: a named handover on a known date, not a gradual fade where the old script stops being run and nobody quite decided that.

A Lens, Not a Project

Then repeat. Pick the next recurring failure, map its distance, find its boundary, add the smallest consequential check, capture and hand over what it replaces. Each pass shortens one real distance and leaves a recorded control behind it.

The lens works in both directions. It shows you which distance to shorten next, and it shows you when a distance is short enough. Awareness has a human cadence. If the person who acts on a failure checks twice a day, detection within the hour is already ahead of them. Shortening it to minutes changes nothing they can use, the same way there's no point building a real-time pipeline for a report a human refreshes every two hours. When the system tells people sooner than they can act, that distance is done. Spend the effort on the next one.

There's no finished state, and that's not a weakness of the method. Failure latency isn't a property a system achieves. It's a lens for looking at an imperfect system and finding the next improvement worth making. The system you have keeps its history, its quirks, and most of its architecture. What changes is how long it believes bad data is good.

That's the series. Failure latency names the problem: the time between a data problem existing and the system making it visible. Validation belongs at boundaries. Contracts make the boundaries executable. The implementation patterns make them operable. The evidence falls out of running honestly. And none of it requires starting again.

The goal is not to build the perfect pipeline. The goal is to make failure visible soon enough to do something useful about it.