Slow processing, the answer that came too late
Where operational data ends up
Some data is good, connected and correct, and still useless, because it arrives after the moment it could have changed. A batch job that runs nightly cannot warn anyone at two in the afternoon. Lateness is a quality problem wearing an infrastructure costume.
What it means
What late means, measured against the decision
Data is late relative to a decision, not to a clock. A monthly figure can be perfectly timely for a board and uselessly late for an operator watching a temperature climb.
Most industrial estates run on batch rhythms designed when storage was expensive: extract nightly, aggregate weekly, report monthly. The equipment got faster. The rhythms did not.
- Nightly batches mean today's problem is visible tomorrow.
- Each hop between systems adds its own delay and its own failure mode.
- Aggregation throws away the minutes where the event actually lives.
- Alerts computed on stale data alert about the past.
What it costs
What an hour of delay actually costs
The cost of lateness is the gap between when reality changed and when anyone could act. Everything in that gap runs on assumption.
Interventions become post-mortems
The trip that could have been prevented at five past two is investigated at nine the next morning. Same data, same analysis, entirely different value.
Operators stop looking
A dashboard known to be an hour behind trains people to ignore it. The habit persists even where the data underneath was actually fresh.
Automation inherits the lag
An agent acting on stale state acts wrongly, with confidence. Automation makes lateness worse, because it removes the human who knew the dashboard lied.
Our approach
What real time actually requires
Real time is not a faster batch. It is a pipeline where data streams as it is produced, and where the platform is honest about how fresh every value is.
Stream first, batch as the exception
Values flow through the pipeline as they are measured, and subscriptions push them onward. Batch remains for what is genuinely periodic, instead of being the default everything inherits.
Freshness as metadata
Every value carries when it was measured and when it arrived. A dashboard that shows its own lag is a dashboard people can calibrate their trust against.
Compute where the data lands
Aggregations and detections that run inside the platform see values seconds after measurement, not after the next scheduled export. The query engine is fast enough that raw history stays queryable directly.
Common questions
Asked and answered
What is slow data processing?
Slow processing is any pipeline where data arrives after the decision it should have informed: nightly extracts, hourly aggregations and multi-hop integrations that turn a live measurement into next-day history.
When does data lateness matter?
Whenever the decision window is shorter than the pipeline delay. An operator reacting to a rising temperature needs seconds to minutes, and any pipeline measured in hours makes that class of intervention impossible.
How do you make operational data real time?
Stream values as they are produced instead of batching them, push updates through subscriptions, record freshness on every value, and run detection inside the platform so the alert fires seconds after the measurement, not after the next export.
The rest of the split
Where the other data points go
The drain has five outlets, and each one fails an operation in its own way. This page covers one of them. The other four are worth ten minutes each.