Data cleaning
Bad data almost never looks bad. A frozen sensor reports a perfectly reasonable number, over and over. A drifting calibration is right to two decimal places and wrong by four per cent. Both flow into reports, models and decisions without anybody noticing, and the answer that comes out is confident, well sourced and untrue.
Cleaning is not the tidying you do before the interesting work. It is what decides whether any of the interesting work is true.
What dirty actually looks like
| Defect | What you see | Where it comes from |
|---|---|---|
| Frozen | The same value, hour after hour, while the plant carries on | A failed transmitter, a stuck buffer, a lost connection where the last value is repeated rather than dropped |
| Spike | A single impossible value | Electrical noise, a comms glitch, a sensor briefly out of range. Harmless to look at, ruinous to any average |
| Drift | Slowly wrong, and never obviously so | Calibration ageing, fouling, a sensor moving out of position |
| Gap | Missing hours or days | An outage upstream, a filter, a network drop. Dangerous mostly for what gets filled in afterwards |
| Wrong unit or scale | A step change that looks like an event | A configuration change, an instrument replacement, an integration mapping bar where the source meant psi |
Two more do not draw well but cost just as much:
- Duplicates. The same measurement arriving twice through two routes, quietly doubling a count or halving an average. Usually a symptom of matching on names instead of external ids.
- Timestamps in the wrong zone. An offset that turns a night shift into a day shift and destroys any comparison against something recorded properly. The favourite of anyone who has ever tried to correlate two source systems.
A signal that looks like a slow, clean wave may be an artefact of sampling too rarely rather than anything the plant did. That one is chosen at the instrument and cannot be repaired afterwards. What undersampling invents →
The model is what convicts a reading
Here is the part that matters most, and the reason this page sits in the concepts section rather than in a manual.
A value is rarely wrong on its own terms. A tank level rising on a warm afternoon is a plausible number. It becomes suspicious only when you know that nothing was delivered, that the tank is outdoors, and that the ambient temperature rose at the same time. A vibration reading of zero is fine, unless the pump it belongs to is running.
Every one of those judgements is made by looking at the neighbours of the reading: the equipment it belongs to, what feeds that equipment, what else was happening at the time. In other words, cleaning is a knowledge graph problem wearing a statistics costume.
Two consequences worth acting on:
- Model first, clean second. Teams that try to clean a data set before describing what the data is about end up applying thresholds they cannot justify. The context is what makes a defect visible.
- Cleaning is never finished. Data keeps arriving, so detection has to keep running. This is standing work, not a project with an end date, which is exactly why it suits an agent.
Finding what moves with what →
Three rules that keep a correction honest
A correction is a claim about reality, and it deserves the same scepticism as the reading it replaces.
A correction is a new series beside the original, never an edit to it. The measurement is a historical fact even when it is a wrong one, and you will want it the day somebody disputes the correction, or the day you discover the correction itself was wrong.
What was changed, when, by whom or by which agent, and on what grounds, recorded as an event against the resource. A cleaned number with no record of the cleaning is worse than a dirty one, because it cannot be argued with.
Smoothing noise is safe. Declaring a sensor faulty and substituting a modelled value is a judgement with consequences, and it belongs in front of a person the first several times. An automatic substitution nobody sees becomes an assumption nobody remembers.
Where agents help
Cleaning is continuous, judgement-heavy and dull, which is an unusually good description of work to delegate. An agent fed by subscription sees each reading as it lands, can compare it against everything the model says is related, and raises a typed event when something looks wrong instead of quietly rewriting it.
The guardrails are the same as for any acting agent: a narrow account, a step limit, and a human gate before anything irreversible. How agents do this work →
Why it is worth the trouble
Every later capability inherits this. A feature computed from a frozen sensor is a feature that measures the sensor. A model trained on drifted readings learns the drift. An agent reasoning over gap-filled values produces gap-filled conclusions, and cites its sources while doing so.
The cost of dirty data is not the error, it is the confidence.- Data cleansing
- Calibration
- Missing data
- Outlier
- Mars Climate Orbiter: the most expensive unit-conversion error on record
- Data quality
- Feature extraction: what you build once the readings can be trusted
- What is machine learning?: why a model trained on dirty data learns the dirt
- Lineage and data quality: what the platform will eventually track for you
- Relationship analysis: the statistics behind "does this reading fit what everything else did?"
- Time series: where readings live, and the sampling trap that cannot be cleaned away
- Building AI agents: delegating the standing work