Shift Left Isn't Enough: Validation Belongs at Boundaries

The first article in this series named the problem: failure latency.

A data problem becomes detectable at one point. The pipeline makes it visible at another. The gap between those points is where time, compute, trust, and sanity get lost.

The obvious response is to move validation earlier.

That instinct is right, but it isn't precise enough. "Shift left" gets us moving in the right direction. It doesn't answer the design question.

Take the same example from the first article. A scheduled pipeline runs for three hours, then fails. One source file contained an invalid business code. The value was already present in the file at the start of the run. The pipeline didn't spend three hours creating the problem. It spent three hours failing to notice it.

So yes, the check should happen earlier.

But where?

A file-present check can run at source receipt. A schema check can run during parsing. A row-count check can run after landing. A business-code check needs the reference set that defines valid codes. A reconciliation check needs transformed or aggregated data.

Those aren't the same boundary.

Earlier isn't the same as useful.

The Scope of This Argument

The examples here come from batch and scheduled data pipelines: bounded runs, known inputs, staged processing, validation gates, retries, handoffs, and operational state.

The same principle can transfer to event-driven systems, but those systems have different failure boundaries. I'm not giving implementation guidance for streaming, pub/sub, or continuously running event pipelines here.

This article is about placing validation in batch pipelines. The goal isn't to validate everything at the start. It's to put each check where the pipeline first knows enough to act.

The Limit of Shift Left

Late validation creates high failure latency.

A check that runs after hours of work can still be correct. It can produce a clear error. It can stop the output from being published. But it still lets the system spend hours treating bad data as usable.

That's why "shift left" has appeal. It pushes teams away from final-stage validation and after-the-fact reconciliation. It asks the system to find problems sooner.

That phrase gets weaker after that.

It doesn't say what each check needs to know. It doesn't say who owns the result. It doesn't say what the pipeline should do after a failure. It doesn't distinguish a cheap structural check from a rule that needs business context.

A check can run too early.

Run a reference-data check before the reference set is available, and the result is either impossible or fake. Run a semantic check before the data has the right shape, and the check becomes noisy. Run every rule at intake, and the intake layer becomes a confused mixture of file checks, business rules, consumer checks, and partial guesses.

That doesn't reduce failure latency in a useful way. It just moves the confusion to a different place.

Saying "shift left" is fine as far as it goes. It just doesn't tell you where the check should actually live.

Boundaries Are Where the System Learns Something

A boundary is a point where data, responsibility, trust, or system state changes.

A file landing is a boundary. The system can now tell whether the source arrived.

Parsing is a boundary. The system can now tell whether the file is readable and has the expected shape.

Landing data into a table is a boundary. The system can now count rows, check emptiness, detect duplicates, and capture a stable copy.

Staging is a boundary. The system can now apply source-level rules against materialised data.

Enrichment is a boundary. The system can now join to reference data and judge mapped values.

Aggregation is a boundary. The system can now compare totals, balances, and cross-record relationships.

Publication is a boundary. The system can now decide whether an output is fit to release.

Handoff is a boundary. Responsibility can now move to another team, process, system, or consumer.

At each boundary, the system knows something it didn't know before. It can make a better decision than one step earlier.

That's why boundaries matter. They aren't just places in the code. They're points where the system can make a better decision.

A boundary matters when it changes what the system knows, or what the system is allowed to do next.

Match the Check to the Boundary

A check belongs at the first point where the system has enough information to answer it properly, and where the answer can still change what happens next.

That means three things.

The rule has the facts it needs. The answer is worth trusting. The result can change the next step.

A missing file check doesn't need a database load. It belongs at source receipt.

A malformed CSV doesn't need transformation logic. It belongs at parsing.

A missing required column doesn't need a full pipeline run. It belongs at parsing or landing.

An invalid currency code needs a set of valid currencies. It belongs where the source value and reference set first meet.

A balance check needs the data to have the right shape. It belongs after the transformation or aggregation that creates that shape.

A publication completeness check belongs before release, not after a consumer has already picked up the output.

Check type	Required context	Likely boundary	Useful next action
Source arrived	Expected delivery and file location	Source receipt	Mark late, notify source owner, hold run
File readable	File bytes, format, delimiter, header	Parsing	Reject file, request replacement
Required columns	Parsed structure	Parsing or landing	Stop source processing, report missing fields
Empty or duplicate file	Landed file metadata or row count	Landing	Quarantine input, block dependent work
Null or simple domain rule	Materialised source data	Staging	Reject rows, stop source stage, assign owner
Reference value valid	Source value and reference set	Enrichment	Reject, quarantine, request mapping update
Total or balance matches	Transformed or aggregated data	Aggregation	Stop output build, route for review
Output complete	Finished output and release rules	Publication	Hold release, mark output not ready
Consumer notified	Published state and owner list	Handoff	Send state, evidence, and next action

This isn't a call for more validation everywhere.

It's a call to stop putting checks wherever they happen to be convenient.

The pipeline doesn't need one giant gate at the start. It needs the right checks at the points where they mean something.

The Same Failure, Placed Properly

Return to the invalid business code.

The file arrives before the scheduled run. The receipt boundary checks that the expected source appeared in the expected window. That passes.

The parsing boundary checks that the file is readable, has the expected header, and contains the required columns. That passes too.

The landing boundary captures the file and checks basic facts: row count, empty file, duplicate delivery, and source metadata. Nothing fails there.

The staging boundary applies simple field rules. The business code is present. It has the right type. It matches the expected pattern. Still no failure.

The enrichment boundary joins the business code to the reference set for that reporting date. The code doesn't resolve.

That's the first useful place to fail.

The failure didn't belong at publication. By then the pipeline has already done too much work.

It didn't belong at raw file receipt either. The file alone didn't contain enough context to judge the code.

It belonged at enrichment, where the source value and the reference set first existed together.

At that point the system can do something useful. It can stop the affected source. It can quarantine the unresolved rows. It can route the failure to the source owner, reference-data owner, or mapping owner. It can hold downstream publication until the state is resolved.

That's the point of boundary placement. It reduces failure latency without pretending every problem can be found at intake.

Checkpoints and Gates Are Different

A checkpoint records state.

A gate controls progression.

A checkpoint can say passed, failed, pending, skipped, quarantined, or not ready. It gives operators and downstream users a clearer view of the run.

A gate decides what the pipeline is allowed to do next. It can reject an input, stop a stage, pause a run, quarantine records, require a decision, or block publication.

Both matter.

A checkpoint without consequence can still help. It can show where the run stands. It can create evidence. It can make diagnosis faster.

But a checkpoint alone doesn't reduce operational risk very much. The pipeline can still carry on after learning that something is wrong.

A gate without visibility creates a different problem. The pipeline stops, but nobody understands the state. Operators see a failed job, not a clear decision point.

Reducing failure latency needs both parts. The system has to make the failure visible, and that visibility has to affect what happens next.

A checkpoint tells you where the pipeline is. A gate decides what the pipeline can do next.

Ownership Belongs With the Boundary

Validation placement isn't only a technical choice.

A failure isn't really visible until it reaches someone who can do something about it.

A source receipt failure belongs with the source owner or ingestion owner. A parse failure belongs with the team that owns the source format or ingest code. A reference mismatch belongs with the owner of the source value, the reference set, or the mapping rule. A reconciliation failure belongs with the team that can judge the business result.

Routing matters.

The system can detect a failure early and still waste time after detection. That happens when the result lands in the wrong queue, the wrong Slack channel, the wrong dashboard, or a log nobody reads.

That's still failure latency. The system knows, but the right person doesn't.

Good boundary design includes the audience for the failure. It says what failed, where it failed, who owns the next action, and what work now has to wait.

This is where observability starts to matter, but the boundary comes first. Without a clear boundary, the message has no useful shape.

Boundaries Make Recovery Smaller

Monolithic jobs make failures large.

One job starts. It loads data, transforms it, joins it, aggregates it, builds outputs, and publishes results. A failure deep in the run often means a full rerun. Operators have to work out what already happened, what can be reused, what needs clearing down, and what state downstream users have seen.

Staged pipelines make failures smaller.

One source can fail and the rest can wait. A file can be quarantined before enrichment. A staging table can preserve the last good load. An output can remain held until one upstream issue is fixed. A downstream handoff can show a clear not-ready state instead of silence.

Boundaries give the system places to stop without forgetting what already happened.

That matters for recovery. It matters for trust too.

A late failure says: something went wrong somewhere in the run.

A boundary failure says: this source passed receipt and parsing, failed reference resolution at enrichment, and didn't reach publication.

That's a different operational conversation.

Boundaries don't just catch failures earlier. They make the recovery smaller.

A Better Placement Question

The first article asked:

Where does this failure become visible today, and where is the earliest useful place we could make it visible instead?

Here, that becomes a placement question:

What does this check need to know, and what should change if it fails?

That breaks down into a few smaller questions:

• What rule failed?
• What facts does the rule need?
• Where do those facts first exist?
• What work has already happened by that point?
• Who can act on the result?
• What should the system do next?
• What evidence should the system keep?
• What downstream work should wait?

You don't need a new platform to ask those questions.

Pick one late failure. Map the place it appears now. Map the first boundary where the rule has the facts it needs. Add the smallest useful check there. Make the result visible to the right owner. Make the pipeline do something different after failure.

That's boundary design in practical terms: not a grand redesign, just a better answer to where the check belongs.

What Comes Next

Once you look at boundaries, failure latency stops being a vague timing complaint. It becomes a design question.

Once the boundaries are clear, the next question is what each boundary expects.

What should a source look like at receipt? What should a file contain after parsing? What rules should staging enforce? What reference values must enrichment resolve? What must be true before publication?

Those expectations often exist already. They're just scattered across code, wiki pages, test suites, runbooks, tickets, analyst memory, and incident history.

The next step is to make them explicit and executable.

That's where data contracts enter the series. Not as documentation. Not as a vendor pitch. Not as a magic fix for bad data.

Data contracts help when they express boundary expectations in a form the system can run.

The point isn't to validate sooner. The point is to validate where the system knows enough to act.