Direct Answer

The platform cleaning layer runs every day — what data problems is it actually correcting? Will it wash away real faults? The four-stage cleaning pipeline handles four types of problems respectively: denoising handles noise spikes, deduplication handles repeated reporting, anomaly flagging handles out-of-limit jumps, and interpolation handles missing periods — each stage manages its own domain with its own boundaries. The most critical discipline is "flag anomalies, do not silently change the numbers": the waveform of a real fault must be kept intact and labeled, not smoothed away; cleaning rules leave traces throughout and are auditable; raw data is read-only, never written. Only in this way can the cleaning layer reduce false alarms without washing away responsibility. FEXCloud's IoT cloud platform makes this four-stage cleaning the first gate of data quality.

A scene: a park platform reported "current over limit" for three days in a row, but the site check found everything normal — opening the cleaning records showed that all three days were the same gateway resending the same frame; the deduplication rule had not taken effect, and three false alarms were generated. After fixing the deduplication rule, similar false alarms dropped to zero. Cleaning is not "something quietly done in the background"; it is the first gate of data quality — if the rules do not land, no algorithm can save you.

A lesson of "washing away a real fault": a platform applied a smoothing window to all current waveforms, and a real arc event was smoothed into a "normal fluctuation" — the fault did not alarm, and the switch burned out before it was discovered. Post-review found that the smoothing window happened to cross the mutation segment of the arc ignition and erased the most critical evidence. Every boundary of a cleaning rule carries such a lesson behind it — better to keep more than to kill by mistake.

1. Denoising: The Boundary Between the Sliding Window and Mutation Detection

Noise is the "background sound" of sensors and channels — electromagnetic interference and quantization jitter all create spikes. The first thing in denoising is to distinguish "noise" from "real mutation":

Sliding-window smoothing — replace the current value with the mean/median of the N points before and after, smoothing small random jitter. Suitable for: steady-state measurements (the daily fluctuation of temperature and voltage); Mutation detection — for points whose rate of change exceeds the threshold, do not apply smoothing; keep them directly as events. Suitable for: fault waveforms (arc, inrush current) — the feature of a real fault is exactly "mutation".

Boundary discipline: the smoothing window must not cross events — at the moment a mutation signal is detected, smoothing must pause, otherwise the spike of a real fault will be "washed flat". Denoising handles only "small high-frequency jitter" and never "large low-frequency mutations" — if you cannot tell the two apart, you will wash away the fault.

A denoising example: a temperature point fluctuated daily around 45°C ± 0.3°C; one day a 47°C spike appeared and fell back — after the smoothing window flattened the spike, the temperature curve looked "normal"; but the site check found the cooling fan had stopped and restarted, and the 47°C was a real event. There is no absolute value between "small high-frequency" and "real event" — mutation detection must classify them first; for an uncertain spike, better to keep it as an event than to smooth it as noise.

2. Deduplication: The Merge Rules for the Same Timestamp / Same Frame

Duplicate data comes from three scenarios: gateway resend (communication retry), multi-channel repeated reporting (redundant acquisition), and abnormal device repetition (the same state reported repeatedly). The deduplication rule judges by "data source + timestamp":

Same-source same-timestamp deduplication — duplicate frames of the same device at the same timestamp keep the latest frame (or the first frame); Cross-source merge — redundant acquisition of the same measuring point from different channels is merged by priority (primary channel first, backup channel fills gaps); Repeated-alarm folding — repeated alarms generated by the same state repeatedly reported are folded into one record with the count noted.

Boundary discipline: deduplication looks only at "whether it is the same source at the same moment", not at the value — if the values at the same moment genuinely differ (for example, two channels read slightly differently), that is "merge", not "deduplication"; the deduplication rule must not swallow the difference information.

A gateway-resend example: a gateway with unstable communication resends the same frame 3 times within 2 seconds — without deduplication, the feature computation counts the same event 3 times and the alarms triple; after deduplication only one frame remains and the event count is correct. Deduplication solves not "whether the data is right" but "how many times the event is counted" — if the rule leaks, false alarms multiply.

A cross-source merge example: a measuring point has a primary channel and a backup channel, each with its own readings; when the primary channel fails, the backup channel takes over — the readings of the two channels at the same moment differ by 0.1%; this is not duplication but "difference information that backs each other up": merge by priority keeping the primary-channel value, and the difference record is used to judge channel health. Treating merge as deduplication swallows the channel-health information too — deduplication swallows "exactly the same", and merge keeps "reasonable difference".

3. Anomaly Flagging: Hard Limits vs Soft Trends — Flag, Not Delete

The most common dispute in the cleaning layer: "this value exceeds the limit — is it a data error?" — the answer is: out-of-limit data is most likely not an error but an event. So the discipline of anomaly handling is "flag, do not delete, do not rewrite":

Hard-limit flagging — points exceeding the equipment rating or the normative limit (such as the GB/T 14549 harmonic limits) are labeled "out of limit", with the original value retained; Soft-trend flagging — points that do not exceed the hard limit but deviate from their own baseline (such as an abnormal temperature-rise rate) are labeled "trend anomaly"; No silent modification — no anomaly point is allowed to be "corrected" into a normal value by a cleaning rule; the waveform of a real fault must remain checkable in its original form.

Boundary discipline: flagging is for the downstream to see; deletion is sentencing the data to death — the cleaning layer only flags; whether to remove is decided by the diagnosis model and by humans, which both keeps the data complete and does not wash away responsibility.

Why "deletion" is more dangerous than "flagging": deletion is a one-way road — once data is deleted, "what actually happened at that time" can never be found during post-review; flagging is a two-way road — the label travels with the data, and the downstream can "reverse the verdict" at any time. When a real fault is deleted, the raw evidence of the diagnosis is destroyed; when a real fault is flagged, even if it did not alarm at the time, it can be traced afterwards. The professional conduct of the cleaning layer is to always leave the choice to the downstream.

4. Interpolation: Linear / Spline Each Fits Its Own Missing Duration

Device offline and communication interruption both produce missing periods. The principle of completion is "short gaps are filled, long gaps leave traces":

Linear interpolation — for a few missing points (within seconds), connect the points before and after with a straight line, simple and reliable; Spline interpolation — for slightly longer gaps (tens of seconds), fit a smooth curve with neighboring points to avoid an abrupt polyline; Long gaps are not filled — gaps exceeding the threshold (minutes and above) are not interpolated; the period is directly marked "data missing" and skipped or degraded by the diagnosis logic.

Boundary discipline: interpolation only fills "measurement continuity", not "event-judgment basis" — interpolated points must not be used as the basis of alarms, otherwise a missing period will be "filled" with a fake event. Interpolated data carries the "interpolated" label and is treated differently downstream.

A hard-limit vs soft-trend example: a loop current is 320 A with a rating of 300 A — the hard limit flags "out of limit"; at the same time its own baseline is 250 A and the temperature-rise rate is twice normal — the soft trend flags "trend anomaly". The hard limit answers "whether the specification is exceeded", and the soft trend answers "whether it deviates from itself" — with both labels present, the diagnosis model can distinguish "occasional over-limit" from "continuous deterioration".

A missing-duration example: a device went offline for 30 seconds; linear interpolation filled the missing points and the curve remained continuous and viewable; if it were offline for 2 hours and still interpolated, every "filled" point in between would be fictional — the correct approach is to mark "data missing" and let the diagnosis logic skip that period rather than fabricate a stretch of fake data. Interpolation has a ceiling; when the gap exceeds it, admit the gap — admitting the gap is more professional than fabricating data.

5. Cleaning Traces: Raw Data Read-Only, Cleaning Rules Auditable

The engineering foundation of the four-stage cleaning is "leaving traces":

Raw data read-only — no writing is allowed into the raw library; cleaning results are stored separately in a cleaning library; Rules versioned — every cleaning rule carries a version number and an effective time; rule adjustments are traceable; Cleaning records checkable — for every data point, which rule handled it and how (smoothing/deduplication/flagging/interpolation) can be checked, with before-and-after comparison.

Boundary discipline: "explainability" is the bottom line of data quality — for any alarm the platform reports, it can be traced back to "what the raw data looked like and what the cleaning rules did"; only then does the customer trust the system.

An audit scene: a platform reported a "cable overheating" alarm, and the customer questioned "maybe the cleaning messed up the data" — the platform pulled up the processing records: the raw temperature curve, the smoothing rule hitting 12 times, 3 points filled by interpolation, 1 out-of-limit flag, with the before-and-after comparison clear at a glance. Traces are not for the platform itself; they are for trust — every cleaning withstands retrospection, and every alarm produces evidence.

6. Landing the Four-Stage Cleaning Pipeline

Step 1: access by data source — the data of every measuring point passes through the cleaning layer first, processed in the order of "denoise → dedupe → flag → interpolate" — the order matters: first remove fake data (noise, duplicates), then mark real events (over-limit), and only last fill the gaps; a reversed order mixes flagging and interpolation together; Step 2: rules configured per measuring point — different rules for different points: smoothing for steady-state temperature, no smoothing for fault current, emphasis on deduplication for gateway data; Step 3: cleaning results stored — cleaned data enters feature computation while the processing records are kept; Step 4: quality monitoring — regularly count each rule's hit rate and mislabel rate; when a rule drifts, tune the parameters or take it offline.

In one sentence: access → configure → store → monitor; with the four steps done, the cleaning layer changes from "can run" to "trustworthy".

The rule hit rate is the physical-check indicator of the cleaning layer: a sudden rise in the deduplication hit rate may mean the gateway network is deteriorating; the anomaly-flag rate continuously above baseline may mean the device is really deteriorating, or the thresholds are too loose; a hit rate of smoothing that is too low means the measuring point itself jitters a lot — solve the measurement problem first before talking about cleaning. The cleaning layer is not configured once and finished — the rules must evolve together with the site.

7. Back to FEXLINK: How the Cleaning Discipline Lands

To land this logic on products: FEXCloud's IoT cloud platform makes the four-stage cleaning the first gate of data quality — denoising manages steady state, deduplication manages repeated reporting, anomaly flagging manages over-limit events, and interpolation manages missing periods, with traces and auditability throughout. The value of the cleaning layer is not "making the data beautiful" but "making the data trustworthy" — real faults are kept intact, fake noise is handled in time, and the processing is checkable; only then does the customer dare to use the diagnosis conclusions as a basis. During selection, ask "does your platform delete or label abnormal data" — the one answering "only flag, do not silently change the numbers" is the one treating data quality as systems engineering; the one answering "automatically correct abnormal values" should raise the alarm that it is washing away both responsibility and evidence together.

Scope and Limitations

  • The conclusions in this article are intended for LV distribution and electrical-safety sites. Standard numbers (GB/T 14549) refer to their currently effective editions, and product parameters follow the latest FEXLINK selection handbook and product knowledge base.
  • Quantitative criteria (protection zone, discharge and residual voltage, earth resistance, thresholds and grading) hold under the preconditions stated in this article. Where the site voltage level, earthing arrangement, environmental category or load profile differs, the figures must be recalculated. Specific configurations involving FEXCloud IoT cloud platform should be verified against actual circuit parameters.
  • Specific product models and parameters are subject to the latest selection handbook and product knowledge base; this article does not constitute selection or procurement advice.
  • Standard numbers cited here refer to the currently effective editions; when a standard is revised, the related quantitative conclusions must be re-checked.
  • Deployment and implementation for specific projects must be carried out by qualified engineers based on site survey and design drawings.

Frequently Asked Questions

The Four-Stage Cleaning Pipeline

The platform cleaning layer runs every day — what data problems is it actually correcting? Will it wash away real faults