Stream processing
The four pages before this one quietly assumed the data was sitting still. It is not. Readings arrive every second of every day, and the choice is whether your computation waits for them to pile up or meets them as they land.
Stream processing is computing on data as it arrives. The shift it asks for is not really about speed: it is that the question is asked once and keeps answering, instead of being asked again every time somebody wants a current number.
Batch and stream, and why the difference is not speed
Batch collects the data and then computes. A stream computes as the data lands. Both are legitimate, and the honest difference is how old the answer is when somebody reads it.
Batch is not obsolete and nothing here suggests replacing it. A monthly close, a model trained on three years of history, an annual emissions submission: all of these want the whole set, and there is no advantage in computing them continuously. Batch is periodically right. What it costs is that between runs, nobody knows how much of the story is missing from the number in front of them.
A rule of thumb that survives contact with real operations:
- If a decision waits on the number, stream it. Anything feeding an alarm, an intervention, a dispatch decision or a control-room screen.
- If the number is read once a month, batch it and enjoy the simplicity. Streaming machinery has running costs, and a stream nobody watches is an expensive way to store data.
The trap in between is assuming a streamed answer is a better answer. It is the same answer, sooner, and sooner is only worth something if somebody or something can act on it.
A stream has no end, so a window manufactures one
You cannot take the average of a stream, because it has not finished. Every continuous computation therefore cuts time into pieces, and the piece is called a window.
Choosing the length is a claim about the process, not a preference. Too short and the window reports noise as though it were news; too long and it reports late, and both failures look like the method not working. The length that is right is set by how fast the thing being watched can actually change, which is the same reasoning that decides how often you sample it.
The same idea drawn as a computation →
When it happened is not when it turned up
Here is the part that generic streaming advice, most of it written for web traffic, will not prepare you for. In an industrial setting a reading's timestamp and its arrival are two different facts, and they can be hours apart.
Links drop. A vessel sails out of coverage and reports when it is back. A rig batches uploads to save bandwidth. A technician's handheld syncs at the end of the shift. None of this is a fault, it is normal operation, and it wrecks any computation that treats arrival as time.
Three rules follow, and they are worth writing into whatever you build:
- Compute on the timestamp the reading carries. Arrival time is a fact about the network, not about the process, and it belongs in the record as such rather than in the calculation.
- Decide how long a window stays open. Late data forces a trade with no clean answer: hold the window and the result is more correct but later, close it early and you will sometimes have to correct a number you already published. The industry's name for where you draw that line is a watermark.
- Assume a reading can arrive twice. A consumer restarts, a source replays, a connection is re-established mid-delivery. The cheapest defence is to make the computation give the same result whether it sees a reading once or twice, which is worth designing for rather than discovering. This is not hypothetical here, and the section on the SDK below says why.
What a computation has to remember
The dividing line that decides how hard a piece of stream processing will be is not the volume. It is whether the answer depends on anything other than the reading in hand.
| The question | What it has to remember |
|---|---|
| Is this reading above 4.5? | Nothing. Each reading answers on its own |
| Convert this to another unit | Nothing |
| What is the average over 15 minutes? | The readings in the current window |
| Has this been above 4.5 for 20 minutes? | When it first went above, and whether it has dropped since |
| Did a start follow an alarm within 5 minutes? | The alarm, until either the start arrives or the five minutes pass |
| How does this shift compare with the last one? | A summary of the previous shift |
Everything genuinely worth detecting is in the lower half of that table. State is what makes stream processing valuable and it is also what makes it hard, because state has to survive a restart. A rolling average that resets every time a process is redeployed is not a rolling average, it is a rumour.
When the stream outruns the reader
If data arrives faster than something can handle it, one of three things must give: readings are dropped, they queue up somewhere, or the producer is slowed down. Every streaming system picks one, and the only real mistake is not knowing which one yours picked. In practice the failure is rarely a heroic flood. It is one consumer that quietly stopped reading days ago while its backlog grew.
How the platform supports this today
Data arrives continuously, is stored in one queryable model, and can be read continuously. The delivery mechanism is a subscription: push rather than poll, with a position that survives a reconnect. The console's live mode is the same machinery with a chart on the end of it, which is a useful thing to know, because it means anything you can watch live you can also consume in code.
The computing part, today, is yours to run, and the SDK is what makes that a short job rather than a project.
What a subscription gives you through the SDK
There are SDKs for Java, Python and Rust, and the shape is the same in all three. A subscription names a set of time series; a live connection then delivers each new datapoint as it lands, with no polling anywhere in the loop.
Four properties matter more than the syntax:
- You acknowledge what you have handled. Anything unacknowledged is redelivered, so a crash cannot lose data. If a downstream system is unavailable you can decline a message instead, and it comes back later rather than disappearing.
- Which means delivery is at least once, not exactly once. That is the honest guarantee, and it is why the rule above about a reading arriving twice is a design constraint rather than a caution: make the handler idempotent.
- The interest set can change while it runs. Series can be added to or removed from a live listener without reconnecting, so a standing question can be re-aimed at new equipment as the model grows.
- Reconnection is handled for you, which is exactly the case that produces the burst in the figure above. The data is not lost, it arrives together, carrying the timestamps it had all along.
The whole consumption loop is about this long, and it is here to show how little there is rather than because you have to write it:
with client.subscriptions.listen(["engine_room"]) as listener:
for msg in listener:
handle(msg.payload) # your computation goes here
listener.ack([msg.message_id])
A windowed rule, seeded and verified end to end →
The loop that works today
A subscription is the boundary of what your computation sees, so it is worth scoping the way a data set is: narrowly, and for one purpose.
Windowing, filtering, feature extraction, scoring a model. This is ordinary code in an ordinary process, and it can run wherever you like as long as it can reach the platform.
A rolling average becomes a time series; a crossing, a pattern change or a detection becomes an event attached to the equipment it concerns.
The derived signal is a first-class series in the model. Whoever charts it, subscribes to it or trains on it need not know it was computed outside, which is what keeps this a sound arrangement rather than a workaround.
The soft sensor in the oil and gas example is exactly this loop, running against a real process today.
Running the computation inside the platform is the roadmap part. Windowed streaming computation, defined once and executed by the platform rather than by your process, is what functions will add. Until then the loop above is the supported answer, and it produces the same data in the same model, so nothing has to be redone when execution arrives.
Where agents help
The obvious idea, an agent subscribed to a stream, is the wrong shape economically. A language model called once per reading is absurd at any real data rate, and the cost is not the only problem: a model asked the same question a thousand times an hour will eventually answer it differently.
The arrangement that works is two-tier. A cheap detector runs on every reading; the expensive reasoner runs on the finding. The threshold, the window or the small model decides that something deserves attention, and only then does an agent gather the context around it: what this equipment is, what happened to it recently, which similar units did the same thing last quarter, and whether anybody is already looking at it.
That division is also what makes the agent's output reviewable, because the trigger is a rule somebody can read. Building AI agents →
- Data subscriptions: the delivery mechanism, in the console and in code
- Time series: what arrives, and how often it should
- Events: what a detection becomes once it is worth recording
- Functions: windowed computation inside the platform, on the roadmap
- Feature extraction: the computations most worth streaming
- What is machine learning?: models that watch a sequence rather than a reading
- Change data capture: getting a source system's changes onto a stream in the first place
- Architecture: where the streaming platform sits in the stack