Direct Answer

The data is connected, but the feed is full of spikes, duplicates and gaps — feeding it straight into an AI model will skew the results. Which cuts come first? The Taiyi L2 cleansing layer splits cleaning into four fixed-order stages: denoise (fixes sensor glitches and EMI), dedup (fixes multi-source duplicate reporting at the same instant), outlier-flag (labels over-limit and jump values instead of deleting them), and interpolation (fills short network gaps). The order cannot be reversed: you must denoise before you can recognize outliers correctly, and you must flag before you lose the signal that "an anomaly is a hazard warning". Cleansing does not wash data until all anomalies are gone; it makes data trustworthy for the model.

A scenario: An industrial park platform onboarded 500 measurement points. Before model launch, the data engineers found current curves full of glitches, two channels reporting the same instant twice, and a one-hour overnight gap. Modelling on the raw feed would treat glitches as real surges, count duplicates twice, and treat the gap as zero. The Taiyi L2 applied the four cuts in order — denoise, dedup, outlier-flag, interpolation — and the training set was clean, with a clear drop in false alarms.

1. Where the Data Diseases Come From: Spikes, Duplicates, Gaps, Outliers

Sensor glitches — loose probes, EMI: single-point jumps with normal neighbours; Multi-source duplicates — the same point reported by two channels, master/backup meters sampled together: several records at the same instant; Short gaps — network drops, gateway restarts: a period with no data; Real outliers — overload, over-limit, abnormal swings: the data itself is saying "something is wrong"; All four mixed together — uncleansed, one-size-fits-all handling makes the feed dirtier.

In one sentence: there are four data diseases — glitches, duplicates, gaps and real outliers — with different causes and different cures, so cleansing must classify first and treat each disease with its own cut.

A disease example: One switch-room current curve — three 8,000 A spikes at 3 p.m. (loose probe), the gateway and the collector each reporting the same instant (duplicate), a 40-minute overnight network drop (gap), and a 30-minute transformer overload (real outlier) — all four diseases on one curve; mixed washing would mis-handle every one. Classify before cleaning — four diseases, four cuts.

2. The Four-Stage Overview: What Each Cut Fixes

Stage 1 denoise — fixes glitches: single-point anomalies from sensor jitter and EMI; Stage 2 dedup — fixes duplicates: multi-source reporting at the same instant; Stage 3 outlier-flag — handles real outliers: over-limit and jumps are labeled and kept, not deleted; Stage 4 interpolation — fixes gaps: fill short network gaps; mark long ones as missing.

StageActionDisease fixedTypical exampleResult
L2-1denoisesensor glitches and EMIsingle 8,000 A current spikeglitch removed, normal value kept
L2-2dedupmulti-source duplicates at the same instantgateway and collector both reportingone record kept, dedup logged
L2-3outlier-flagreal outliers such as over-limit and jumps30-minute transformer overloadlabeled and kept, enters alarms
L2-4interpolationshort network gaps40-minute overnight droppoints filled or gap flagged

In one sentence: the four stages are denoise, dedup, outlier-flag and interpolation — each cut cures one disease, every result is traceable, and the order is fixed.

A comparison example: The same curve — denoise first (glitches gone), then dedup (one record per instant), then flag (the overload segment labeled "outlier-overload"), then interpolation (the gap filled) — after the four cuts: clean normal segments, labeled outlier segments, and flagged missing segments. Four cuts leave data both usable and still carrying the "something went wrong" signal.

3. Stage One: Denoise — Fixing Sensor Glitches and EMI

The look of a glitch — single-point, absurd amplitude, normal neighbours: clearly against physical rules; The source of a glitch — loose probes, poor wiring, EMI: a sensor-layer problem, not a real on-site event; The denoise method — isolated-point detection with neighbourhood checks: a point beyond the plausible neighbourhood range is judged a glitch; The boundary of denoise — multi-point segments are not glitches: 5-7x startup current is a real event and must not be removed; The value of denoise — stop fake spikes fooling the model: uncleansed glitches make the model read noise as surges.

In one sentence: denoise fixes "fake disease" — a single-point glitch is a sensor problem, not an on-site problem — wash it away so the model sees no hallucinations.

A denoise example: A bus current showed a single 8,000 A spike between 500 A neighbours — judged a loose-probe glitch and removed; another curve showed 20 consecutive 2,500 A points during motor startup (5x of the 500 A rating) — judged a real startup event and kept. The hard part of denoise is not removing glitches; it is not washing away real events with them.

4. Stage Two: Dedup — Fixing Multi-Source Duplicates

The source of duplicates — master/backup meters, gateway and collector double channels, replay mechanisms; The harm of duplicates — the same data counted twice: accumulated energy doubles, statistics distort; The dedup method — three-dimension judgment on timestamp, point and source: one record per instant per point, from the authoritative source; The authoritative source — master meter over backup, gateway over collector: the rule is configurable; The dedup log — every dedup leaves a trace: what was removed and what was kept — auditable.

In one sentence: dedup fixes the "duplicate disease" — one record per instant per point, and when master and backup disagree, the authoritative source wins.

A dedup example: One meter's energy: the gateway and the collector each reported the same value at the same instant — by the "gateway first" rule, the gateway record was kept and the collector record was marked "duplicate-deduped". Dedup is not deleting at random — it defines rules, keeps records, and can always explain what was removed and why.

5. Stage Three: Outlier-Flag — Label, Do Not Delete

The nature of outliers — real events: overload, over-limit, abnormal swings — the data is saying "something is wrong"; The common mistake — deleting outliers as dirty data: the hazard signal disappears with them; The right way — label and keep: mark the segment "outlier-overload" or "outlier-over-limit"; The value of labeling — an outlier is a hazard signal: overload segments feed alarms, over-limit segments are rectification evidence; The link to alarms — flagged outlier segments enter the alarm engine: nothing is lost, alarms have evidence.

In one sentence: outlier-flag handles "real disease" — over-limit and jumps are not dirty data but hazard signals — label and keep them instead of deleting.

A labeling example: A transformer current stayed over-limit for 30 minutes. Deleting it would make the segment vanish from reports and leave alarms without evidence; labeling it "outlier-overload" sends it into the alarm chain as rectification evidence. The word "anomaly" is information, not an error — cleansing keeps it, not kills it.

6. Stage Four: Interpolation — Fixing Short Gaps

The source of gaps — network drops, gateway restarts, collector disconnects: a period with no data; The harm of gaps — treated as zero: the algorithm reads "no data" as "value is zero" and every result skews; The interpolation method — linear fill: short gaps are bridged with the neighbouring values; The boundary of interpolation — long gaps are not filled: beyond the threshold, mark the "missing segment"; The interpolation log — every filled point is tagged: the model knows what is measured and what is computed.

In one sentence: interpolation fixes the "gap disease" — fill short gaps, flag long ones — a gap must never be read as zero.

An interpolation example: One meter lost 40 minutes overnight: current at 300 A/305 A around the gap, linear fill bridged the curve; if the outage were two days, no fill — mark "missing segment" and raise a network-down alarm. Interpolation is not fabricating data to fool the model — it fills short holes and flags long ones so the model knows "this part was computed".

7. Why the Order Cannot Be Reversed

Denoise before dedup — with glitches still in the feed, dedup may keep or remove the wrong record at a glitch instant; Dedup before flag — with duplicates still in, the outlier segment is counted twice and flags duplicate; Flag before interpolation — real outliers are flagged first so interpolation is not biased by them (outlier segments are not interpolated); The cost of wrong order — interpolating first treats a glitch as a true value and pulls neighbours up; deleting first washes the hazard away; The guarantee — a fixed pipeline: data entering L2 always walks the four cuts in order.

In one sentence: the order is fixed — denoise, dedup, flag, then interpolate — if the dirt of the previous stage remains, every later judgment is wrong.

An order example: Interpolate before denoise and the 8,000 A glitch is treated as a true value, pulling the neighbours up and ruining a good curve; delete before flag and the overload segment is gone, leaving alarms without evidence. The order is the lifeline of the four stages — each stage paves the way for the next.

8. How Cleansed Data Flows Back to Qianzhi and Tianyan

What Qianzhi receives — clean data with labels: outlier segments carry tags, so diagnosis has evidence; What Tianyan receives — continuous computable series: glitches removed and gaps filled, so trend calculation is undisturbed; What reports receive — authoritative deduped data: statistics do not double-count and energy reconciliation holds; The delivery form — the cleansing layer outputs a standard data interface: models, reports and dashboards consume the same feed; The closed loop — models discover new outlier patterns and feed them back into the cleaning rules: cleaning and models evolve together.

In one sentence: post-cleanse data is a clean base (denoised and deduped) plus kept hazard signals (flagged) plus flagged gaps — Qianzhi, Tianyan and reports each take what they need.

A flow-back example: One platform — after L2 cleansing, every outlier segment Qianzhi diagnoses carries an "overload" tag (solid alarm evidence); Tianyan's trend calculation uses the continuous filled curve (fewer false alarms); the monthly report uses deduped energy (reconciliation passes first time). The cleansing layer is not an isolated step — it is the foundation of the data production system, and every downstream consumes its output.

9. Common Pitfalls: Four Traps in Cleansing

Trap 1 — deleting outliers as dirty data — the overload segment disappears and the hazard signal vanishes — flag, do not delete; Trap 2 — reading gaps as zero — the network-drop segment is computed as zero and conclusions distort — interpolate or flag; Trap 3 — arbitrary order — interpolating before denoising treats a glitch as truth — the order is fixed; Trap 4 — no trace — dedup, fill and flag leave no record — you cannot explain yourself at audit time.

In one sentence: the four traps are delete outliers, zero gaps, wrong order and no trace — the first three ruin data, the fourth ruins trust.

A trap example: In one project's review, the cleaning rule deleted an over-limit segment outright, leaving alarms without evidence and the rectification report rejected by the grid operator; after switching to "outlier-flag", the same batch passed. Every cleansing action must be explainable — what was deleted, flagged and filled is all recorded.

10. Back to YESLON: How the Taiyi L2 Cleansing Layer Lands

The Taiyi central control system L2 cleansing layer — ships with the four-stage pipeline: denoise, dedup, outlier-flag and interpolation, in a fixed order; Configuration — each stage's rules are adjustable: denoise neighbourhood range, authoritative-source priority, flag thresholds, and the interpolation duration cap; Standard output — cleansed data carries a labeling system: outlier tags, dedup records and fill marks, fully auditable; Model coordination — Qianzhi and Tianyan consume the cleansing layer directly: training and inference both use trustworthy data; Place in the data production system — cleansing is one link of data production: collection, cleansing, storage and consumption form a complete chain.

In one sentence: the Taiyi L2 is a four-stage pipeline with configurable rules and full traceability — it moves data from "viewable" to "model-trustworthy".

A landing example: An industrial park onboarded 500 points; after the L2 four-stage cleansing, model false-alarm rates dropped, the monthly report reconciled on the first pass, and every anomaly alarm had labeled evidence. Cleansing is not optional preprocessing — it is the foundation of trustworthy AI judgment; the four stages are each indispensable and the order cannot be shuffled.

Scope and Limitations

  • The conclusions in this article are intended for LV distribution and electrical-safety sites. Criteria are compiled from public codes and FEXLINK product documentation, and product parameters follow the latest selection handbook and product knowledge base.
  • Quantitative criteria (protection zone, discharge and residual voltage, earth resistance, thresholds and grading) hold under the preconditions stated in this article. Where the site voltage level, earthing arrangement, environmental category or load profile differs, the figures must be recalculated.
  • Specific product models and parameters are subject to the latest selection handbook and product knowledge base; this article does not constitute selection or procurement advice.
  • Standard numbers cited here refer to the currently effective editions; when a standard is revised, the related quantitative conclusions must be re-checked.
  • Deployment and implementation for specific projects must be carried out by qualified engineers based on site survey and design drawings.

Frequently Asked Questions

G5 Four-Stage Data Cleansing

The data is connected, but the feed is full of spikes, duplicates and gaps — feeding it straight into an AI model will skew the results