Refresh Cadence Is a Design Decision

I've seen versions of the same conversation several times on LinkedIn.

"I'm building a near-real-time pipeline."

"What needs the data that quickly?"

"The business wants it."

That's a composite, not a quote from one person. The pattern is real.

Someone has been asked to build a pipeline with a very short refresh interval. The requirement sounds clear. Then one question exposes the gap: what needs the data at that speed?

"The business wants it" can start a requirements conversation. It can't finish one.

The requested frequency might be right. A decision might need fresh data every few minutes. An automated process might act on every new output. The cost of stale data might be high enough to justify the work.

But none of that follows from the request alone.

Refresh cadence is a design decision, not a default.

This article concerns batch and scheduled pipelines: bounded runs, expected inputs, known handoffs, and outputs produced at set intervals.

Near-real-time requests make useful examples by forcing the frequency question into view. This article isn't implementation guidance for streaming, pub/sub, or continuously running event pipelines.

The Frequency Question Nobody Asked

Pipeline schedules often look more deliberate than they are.

Hourly sounds reasonable. Every 15 minutes sounds responsive. Daily sounds old-fashioned. A cron expression or scheduler setting gives the decision a precise shape, even when nobody has explained the need behind it.

A new pipeline copies the schedule of an older one. A platform team chooses the shortest interval that runs comfortably. A stakeholder asks for fresher data, and the requested interval becomes the requirement. A team runs more often to make a slow process feel responsive.

Each route can produce a valid schedule. None proves that the schedule is useful.

The missing design facts are straightforward:

• What changes between runs?
• Who uses the changed output?
• What becomes late if the next run waits?
• What happens when a run fails?

Those questions rarely appear in the scheduler configuration. Cadence then looks like a technical setting after the design decision has been forgotten.

Frequency creates behaviour across the whole system. It controls how often sources are read, transformations run, outputs are written, downstream jobs start, alerts fire, and operators inspect state.

A schedule isn't just a timer. It's a recurring decision to make the system do all of that work again.

Fast Execution and Frequent Execution Are Different

Teams often mix up two separate ideas.

Execution speed is how long one run takes.

Refresh frequency is how often a new run starts.

Pipeline A starts every hour and takes 50 minutes to finish.

Pipeline B starts twice each day and takes 10 minutes to finish.

Pipeline A is more frequent. Pipeline B is faster.

That distinction matters. Fast execution is nearly always useful. Users wait less. Reruns finish sooner. Failures leave more time for recovery. Compute stays occupied for less time. A shorter run gives the operating team more room to respond.

None of those benefits chooses the schedule.

A fast daily pipeline is still daily. A slow hourly pipeline is still hourly. Improving the runtime doesn't prove that another trigger creates value.

System design still treats the two as one idea. A request for "faster data" turns into a shorter schedule. The platform can process the workload every five minutes, so five minutes starts to feel natural.

It isn't.

A fast pipeline gives you options. It doesn't choose the schedule for you.

The useful interval depends on the system around the pipeline. Sources need to change. Consumers need to act. Recovery needs to fit inside the available window. Each run needs to earn the work it creates.

Freshness Needs a Named Use

"Fresh data" sounds like an obvious good. It's still incomplete as a requirement.

A useful requirement identifies the consumer, the use time, the acceptable data age, and the consequence of delay.

A weak requirement says:

Data must be refreshed every 15 minutes.

A stronger requirement says:

The 14:00 funding decision needs positions no older than 30 minutes.

The second version gives the engineering team something to design against: the consumer, decision, acceptable data age, and consequence of delay.

It can still lead to a 15-minute schedule. It can lead somewhere else too.

Perhaps the source publishes at 13:20 and 13:40. Perhaps one run after the second publication meets the need. Perhaps the pipeline takes 25 minutes, so a 15-minute trigger creates overlapping work without improving delivery. Perhaps the decision uses the latest accepted output and ignores every intermediate version.

The stated use lets the team ask those questions.

Human review is one kind of consumption. A risk analyst might review a report at 09:00. New output produced at 09:15 can't improve that review.

Automated processes have consumption points too. A downstream batch might start at 14:00 and require an accepted dataset by 13:45. It doesn't gain anything from earlier versions that it never reads.

Some automated systems genuinely act on every new output. Their consumption cadence supports a high refresh frequency. That's a real requirement, not a preference for newer timestamps.

A freshness requirement should name the decision that becomes late without it.

Four Constraints Shape Useful Cadence

Four constraints help expose the real design.

They don't form a universal formula. Each one answers a different question.

Source Cadence

Source cadence describes how often the source produces meaningful change.

Suppose a source file arrives once each day at 06:00. An hourly pipeline can run against that file all day. It still has one source state.

Running the same input again doesn't create fresher source data. It creates another processing attempt.

If the consumer hasn't changed its behaviour either, the run may produce a newer timestamp without producing a newer decision.

Repeated runs can have another purpose. A team might run readiness checks, reconciliation, or controlled recovery. That purpose needs a name. "The schedule is hourly" isn't enough.

Consumption Cadence

Consumption cadence describes how often a person or downstream process can use changed output.

A report reviewed at 09:00 and 15:00 has two clear use points. An automated handoff at 14:00 has another.

Those points don't always require separate production runs. One accepted output can serve several uses when the source hasn't changed and the data remains fit for them.

Systems often optimise for the production of outputs rather than the consumption of decisions.

A dashboard refreshed every five minutes doesn't create value if the decision it supports still happens once a day.

Recovery Time

Recovery time covers detection, diagnosis, repair, rerun, validation, and release after failure.

A five-minute schedule doesn't create five-minute recovery.

Suppose an incident takes two hours to resolve. New triggers might continue during that period. They might fail in the same way, queue behind the blocked run, publish suspect output, or produce repeated alerts.

The answer depends on the control design. The schedule still determines how much activity can accumulate during the recovery window.

Fast execution helps here. It shortens reruns and leaves more time before a deadline. High frequency can work against recovery when each trigger creates fresh operational work.

Run Cost

Run cost covers every resource and operational action created by a trigger.

Compute is the easiest part to measure. It isn't the only cost.

Each run can read sources, write storage, execute validations, trigger downstream processing, emit alerts, create output versions, and add another state for operators to understand.

A query that costs pennies can start a chain of work that costs far more in attention.

Cheap to run isn't the same as free to operate.

The Daily Source on an Hourly Schedule

Consider a scheduled pipeline with a simple operating pattern.

A daily source file arrives at 06:00.

The pipeline runs every hour.

Business users review the output at 09:00 and 15:00.

A failed run takes about two hours to diagnose, fix, rerun, and release.

What does the hourly schedule contribute?

The 06:00 run processes the new source. The 07:00 and 08:00 runs process the same source state. One of those later runs might provide recovery from a transient technical failure. It might produce the accepted output for the 09:00 review. That's useful.

The 09:00 run starts as the morning review begins, so it can't improve that review.

Every run between 10:00 and 14:00 still processes the same daily source. The 15:00 review might need a later accepted output. It might use the same output as the morning review. The requirement needs to tell us which.

Any runs after the second review still read, process, validate, write, log, and trigger whatever sits downstream.

There might be a sound reason for every run. The source might receive corrections during the day. The output might feed consumers that act hourly. A contractual commitment might require regular publication. The hourly schedule might form part of a recovery model.

The problem isn't the interval. The problem is the absence of a reason.

Now consider failure.

A bad reference value enters the 06:00 source. Each hourly run sees the same unresolved problem.

The first failure might stop later triggers. Later runs might repeat the failure or alert the same people. The control design decides whether any output can reach the 09:00 review.

The schedule affects all of those outcomes. It can support recovery, or it can multiply the same fault into repeated work.

This article isn't choosing the final cadence for that pipeline. The evidence isn't complete. We need to know whether the source changes again, which outputs the reviewers use, what the downstream consumers need, and how failed runs affect later triggers.

That's the point.

"Hourly" isn't the answer until those facts support it.

Capability Is an Input, Not the Answer

Platform capability matters.

The orchestration system must support the interval. The pipeline must finish inside the available window. Source systems must tolerate the reads. Storage and downstream processes must handle the output rate.

Those facts tell us what is feasible.

They don't tell us what is useful.

A platform might handle a run every five minutes. A pipeline might finish in two. The source might change once each morning. The consumer might act once each afternoon.

Nothing in the technical capability closes that gap.

Return to the opening request:

"I'm building a near-real-time pipeline."

The useful response isn't an automatic challenge to near-real-time processing. It's a request for the missing design facts:

• Which decision needs the data?
• How old can the data be?
• What changes every 15 minutes?
• Who acts on each new output?
• What happens if the data is one hour old?
• How long does recovery take?

Clear answers might support a short interval. They might support a schedule aligned to source arrival. They might support one production run plus a controlled rerun path.

The design comes from the answers, not from the shortest interval the platform accepts.

There's another consequence when production runs faster than the surrounding system can usefully respond. Output starts to accumulate ahead of review, acceptance, or consumption. More versions exist, but confidence hasn't increased.

That's the subject of the next article.

The platform tells you how often the pipeline can run. The system around it tells you how often it should.

Fast execution is an engineering goal.

Frequent execution is a design choice.

Refresh cadence is a design decision, not a default.