Statistical Process Control in IIoT: Control Charts, Cpk, Western Electric and Nelson Rules

A modern production line can emit a thousand readings per second from a single machine, and most plants still decide whether the process is healthy by looking at a red or green tag on a dashboard. That tag is usually a fixed threshold against the specification, which is the wrong question. Statistical process control (SPC) asks something different: is the process behaving the way it behaved when it was last known to be stable, and if not, when did it change? A part can sit inside its tolerance for weeks while the process underneath it drifts toward the edge, and a threshold on the specification will stay silent until the first scrap bin fills.

SPC is a century-old discipline, but the streaming era exposes two sharp problems. High-rate sensors violate the independence assumptions behind textbook limits, which inflates false alarms by an order of magnitude. And alarms that never reach a manufacturing execution system (MES) are just decoration. This article derives the mathematics from first principles, states the Western Electric and Nelson rules exactly, shows runnable numpy code, and then wires everything into a Kafka, Flink and TimescaleDB pipeline that can place a lot on quality hold.

What this covers: common versus special cause variation, the chart family and when to use each, control versus specification limits, ARL arithmetic, rule sets, EWMA and CUSUM, Cp/Cpk/Pp/Ppk with a worked example, autocorrelation pitfalls, and a streaming reference architecture with MES integration.

Context and Background

Walter Shewhart introduced the control chart at Bell Telephone Laboratories in the 1920s. His central insight was not a formula but a classification: variation that comes from the stable system itself (common cause, sometimes called chance cause) versus variation that comes from a specific, identifiable event (special cause, or assignable cause). Deming later popularised the practical corollary: reacting to common-cause variation as if it were special, called tampering, makes the process worse, while ignoring special causes lets defects ship.

The tooling that followed is well documented. The NIST/SEMATECH e-Handbook of Statistical Methods describes a control chart as a plot of a quality characteristic against sample number or time, with a center line and upper and lower control limits, and notes that points inside the limits must also show a random pattern for the process to count as in control. The Western Electric Company codified supplementary run rules, which appeared in its 1956 Statistical Quality Control Handbook, and Lloyd Nelson published an extended set of eight tests in the Journal of Quality Technology in 1984. Page’s CUSUM (1954) and Roberts’ EWMA (1959) followed for small, persistent shifts.

On the standards side, the automotive industry’s reference is the AIAG Statistical Process Control manual, whose second edition dates to 2005 (I could not retrieve the publisher page this run, so treat edition details as to be confirmed against your copy). Internationally, the ISO 22514 series, “Statistical methods in process management: Capability and performance”, defines capability and performance indices. The ISO catalogue entry for ISO 22514-2:2017 describes it as covering process capability and performance of time-dependent process models, and lists its status as withdrawn at the time of writing, so check the current edition of the series before citing it in a quality manual. The control-chart methods themselves sit in the ISO 7870 series.

What changed is the data. In 1985 a technician measured five parts every hour. In 2026 an IO-Link sensor, a PLC register, or an OPC UA subscription gives you every cycle. That abundance sounds like a gift, but it breaks rational subgrouping, introduces serial correlation, and creates a streaming-systems problem: windowing, late events, state, and recovery. For the anomaly-detection side of the same problem, see our complete guide to anomaly detection in manufacturing; SPC is the transparent, auditable baseline that those learned models should be measured against, not replaced by.

Why keep SPC when machine learning exists? Because SPC gives a defensible statement a quality engineer and an auditor can both read: “this point violated a documented rule at a known false-alarm rate.” A neural network anomaly score has no such contract. In regulated or automotive settings (automotive quality systems built on IATF 16949 and the AIAG core tools routinely ask for SPC evidence on special characteristics), that contract is the product.

Reference Architecture: SPC as a Streaming Pipeline

In short: an SPC streaming pipeline ingests sensor readings with event timestamps, groups them into rational subgroups with windowing, computes subgroup statistics, evaluates control limits and run rules against frozen Phase I parameters, and emits alarm events that an MES consumes to place lots on hold. Limits are data, not code, and are versioned.

Figure 1: Reference statistical process control pipeline, from PLC and IO-Link sensors through MQTT, Kafka and Flink to the rule engine, TimescaleDB and the MES quality hold.

The diagram shows a deliberately boring flow. Readings leave the PLC or an IO-Link master (see our piece on IO-Link sensor integration with OPC UA and IODD), travel through an edge gateway that publishes MQTT, and land in a Kafka topic keyed by machine and characteristic. Keying matters: all readings for one chart must reach one Flink task in order, or the run rules, which depend on sequence, become meaningless.

Why the architecture separates statistics from rules

The Flink job produces subgroup statistics (mean, range, count, window boundaries) and writes them to TimescaleDB. A separate rule-engine stage consumes those statistics and evaluates Western Electric and Nelson tests against the active limit set. The separation gives you three things. You can replay the rule stage with different rule sets without recomputing windows. You can audit exactly which statistics fed which alarm. And you can run EWMA and CUSUM detectors in parallel on the same statistics, merging into one alarm topic with a rule identifier.

Why limits live outside the job

Phase I limits (the baseline) are a product of an analysis, usually done by an engineer on a clean historical dataset. Phase II is monitoring against those limits. If limits are hard-coded or recalculated continuously from the live stream, the chart silently adapts to the very drift it is supposed to detect. We store limits in a versioned table keyed by machine, characteristic, recipe, and effective date. The job loads them as broadcast state and the alarm event carries the limit version ID. Recalculation is a human-approved action, shown at the end of the sequence in Figure 4.

Where it runs: edge or cloud

For a single characteristic at one sample per second, the whole chart fits comfortably on an edge gateway. The reasons to push SPC to the cloud are fleet-wide capability comparison and long retention, not compute. The reason to keep a fallback at the edge is connectivity: if the broker link drops, subgroup statistics should buffer and the rule engine should keep running. Our article on MQTT store-and-forward at the edge covers the buffering mechanics, and the SPC-specific rule is that buffered data must be replayed in event-time order so run rules see the true sequence.

Variation, Limits, and the Mathematics of False Alarms

Figure 2: Classifying variation. Common-cause variation is left alone and feeds capability studies; special-cause variation is detected by rules, investigated, and removed before capability is computed.

Common cause versus special cause

Every process has noise: tool wear on a microscopic scale, ambient temperature, material lot variation within tolerance. If you remove nothing and change nothing, that noise forms a stable distribution. That is common-cause variation, and a process showing only that is in statistical control. It is predictable, though predictable does not mean acceptable: a stable process can still be incapable of meeting the specification.

Special-cause variation is a new source: a worn cutter, a wrong coolant concentration, a new material lot, an operator changing a setting. It changes the distribution’s mean, spread, or shape, and the chart is the instrument that detects the change. The order in Figure 2 matters. You detect and remove special causes first, then compute capability. Capability computed on an unstable process is a number without meaning, because there is no stable distribution to describe.

Control limits are not specification limits

This is the most common confusion in plants, and it has expensive consequences. Specification limits (LSL, USL) come from the customer or the design: they define what a conforming part is. Control limits (LCL, UCL) come from the process’s own history: they define what the process normally does. They are computed from different information and answer different questions. A process can be perfectly in control while producing every part out of specification, and it can be wildly out of control while every part is still inside tolerance.

Equally important, control limits for subgroup means are narrower than limits for individual values by a factor of the square root of n. Drawing the specification limits on an X-bar chart is therefore wrong: the means of five parts vary far less than the parts themselves, so a mean can sit inside the spec while individual parts are outside it. Keep the specification on the capability histogram, not the control chart.

Deriving 0.27 percent and ARL0

Shewhart limits sit three standard deviations from the center line. For a normally distributed, independent statistic, the probability that a single in-control point falls outside is the two-sided tail P(|Z| > 3) = 0.0027, about 0.27 percent. The number of points until the first false alarm follows a geometric distribution, so the mean is the reciprocal of that probability: ARL0 = 1 / 0.0027 ≈ 370.4. I computed this directly from the complementary error function and got 370.4, matching the figure quoted in the literature.

The practical meaning is sobering at streaming rates. If every sample is charted and you chart one point per second, a false alarm arrives on average every 370 seconds, about 6.2 minutes. At one point per minute it is about every 6.2 hours, or roughly four false alarms per day per chart. A plant with 500 charts at that rate would be flooded. This is why subgrouping and sensible sampling intervals are not optional details: they set the alarm budget.

Out-of-control ARL: how fast does it catch a shift

The mirror image is ARL1, the expected number of points to detect a real shift. For a mean shift of d standard deviations on an individuals chart with 3-sigma limits, the detection probability per point is Φ(−3 + d) + Φ(−3 − d), and ARL1 is its reciprocal. My calculation gives:

Mean shift (sigma units)	P(detect per point)	ARL1 (points)
1.0	0.0228	43.9
1.5	0.0668	15.0
2.0	0.1587	6.3

A one-sigma shift takes about 44 points to flag on average, which is why supplementary run rules and the memory-type charts later in this article exist. Rational subgrouping helps dramatically: with n = 5, a 1.5-sigma shift in the individual values is a shift of 1.5 × √5 ≈ 3.35 standard errors in the subgroup mean, which gives a per-subgroup detection probability of about 0.638 and an ARL1 of roughly 1.6 subgroups.

Rational subgrouping

A rational subgroup is a set of readings taken so that variation within the subgroup is only common-cause, while any special cause shows up between subgroups. Consecutive parts from one machine, one tool, one fixture, taken over a short span, fit this definition. Mixing four cavities of a mold into one subgroup puts cavity-to-cavity differences inside the subgroup, inflating R-bar and the limits, and the chart will never signal the cavity problem. The cure is one chart per cavity, or a subgroup structure that makes cavity the between-group factor. Subgrouping is an engineering decision about the physics of the process; no algorithm can pick it for you.

The Chart Family and How to Choose

Figure 3: Choosing among SPC control charts. Variables data lead to X-bar with R or S, I-MR, or memory charts; attribute data lead to p, np, c, or u depending on what you count and whether the sample size is constant.

Variables charts

For continuous measurements (diameter, torque, temperature), the workhorse is the X-bar and R chart, with subgroups of size n (typically 2 to 10). The statistics for each subgroup are the mean and the range. With grand mean X-double-bar and average range R-bar, the limits are:

X-bar chart: CL = X-double-bar, UCL/LCL = X-double-bar ± A2 × R-bar
R chart: CL = R-bar, UCL = D4 × R-bar, LCL = D3 × R-bar

The constants come from the distribution of the range of normal samples. For n = 5 the standard table values are A2 = 0.577, D3 = 0, D4 = 2.114, and d2 = 2.326. They are widely published; I used them in the code below, but verify against your own table before relying on them for a controlled document. The within-subgroup sigma estimate is R-bar / d2. For larger subgroups (commonly n above about 8 to 10) the range becomes an inefficient estimator and the X-bar and S chart is preferred.

When rational subgrouping is impossible, either because production is slow (one part per hour) or the measurement is a continuous reading, use the individuals and moving range (I-MR) chart. The moving range between consecutive points estimates short-term variation, and sigma is estimated as MR-bar / 1.128, where 1.128 is d2 for n = 2. Limits on the individuals chart are the mean ± 2.660 × MR-bar (that is 3 / 1.128). I-MR is the default for most IIoT process variables such as furnace temperature, and it is also the chart most affected by non-normality and autocorrelation, as we will see.

Attribute charts

When you count rather than measure, the underlying distribution is binomial or Poisson and the chart changes:

p chart: proportion defective, variable sample size. Limits are p-bar ± 3√(p-bar(1 − p-bar)/n), recalculated per subgroup when n varies.
np chart: number defective, constant sample size.
c chart: count of defects in a constant inspection unit (Poisson, limits c-bar ± 3√c-bar).
u chart: defects per unit when the inspection area varies.

The distinction between a defective unit and a defect is the usual stumbling block. A circuit board that fails inspection is one defective unit (p or np); the three solder bridges on it are three defects (c or u). With low defect rates the lower limit on attribute charts falls below zero and is set to zero, which means the chart cannot signal an improvement, a good reason to prefer variables data wherever a gauge exists. The NIST handbook also warns that for skewed counts the 3-sigma false-alarm risk changes: its example is a Poisson process with np = 0.8 where the upper limit is about 3.48 and the chance of exceeding it rises to roughly 0.009.

A note on normality

The three-sigma convention is robust for averages of subgroups, because of the central limit theorem, but not for individuals. Skewed measurements such as vibration amplitude or particle counts have heavier upper tails, so the I-MR upper limit fires too often and the lower limit never fires. Transform (log is common) or use limits based on the empirical distribution, and document the choice.

Run Rules: Western Electric and Nelson, Stated Exactly

A single point beyond three sigma is a crude detector. The run rules add sensitivity to patterns that are improbable under pure common-cause variation, at the cost of a higher combined false-alarm rate. Both sets work on a chart divided into zones: Zone C is within one sigma of the center line, Zone B is between one and two sigma, and Zone A is between two and three sigma, on each side.

The four Western Electric rules

The rules, applied to one side of the center line at a time, are as follows. These were codified by a Western Electric committee and appeared in the 1956 handbook, per the summary on Wikipedia’s Western Electric rules page, which also attributes the exact false-alarm results to Champ and Woodall (1987, Technometrics).

One point beyond the 3-sigma limit.
Two out of three consecutive points beyond 2 sigma on the same side.
Four out of five consecutive points beyond 1 sigma on the same side.
Eight consecutive points on the same side of the center line.

Rule 1 alone yields a false alarm about once per 370 points. Evaluating all four together raises the false-alarm rate to about 1 in 91.75 points, according to the same source. That is a more than fourfold increase in noise, bought in exchange for a much shorter time to detect small shifts.

The eight Nelson tests

Nelson’s 1984 set contains the tests below. I am stating them from my knowledge of the Journal of Quality Technology paper (The Shewhart Control Chart: Tests for Special Causes) and cross-checked their overlap with Western Electric; the fetched summary did not cover Nelson, so confirm against the original or your SPC software documentation.

One point more than 3 sigma from the center line.
Nine points in a row on the same side of the center line.
Six points in a row steadily increasing or decreasing.
Fourteen points in a row alternating up and down.
Two out of three points more than 2 sigma from the center on the same side.
Four out of five points more than 1 sigma from the center on the same side.
Fifteen points in a row within 1 sigma of the center, on either side.
Eight points in a row more than 1 sigma from the center, on either side, none within one sigma.

Note the family resemblance: Nelson 1, 5 and 6 equal Western Electric 1, 2 and 3, and Nelson 2 is a longer-run (nine rather than eight) version of Western Electric 4. Nelson tests 3, 4, 7, and 8 add trend, oscillation, stratification and mixture detection.

What each test is actually telling you

Rules are not interchangeable alarms; each maps to a physical story. Tests 2 and Western Electric 4 flag a sustained level shift, such as a new material lot or a setpoint change. Test 3 is a monotonic trend, the signature of tool wear, filter clogging, or thermal drift, which is the pattern worth catching early in predictive maintenance. Test 4, fourteen alternating points, suggests over-adjustment by two operators or an automatic controller hunting, or two alternating streams (such as two fixtures) being plotted as one.

Test 7, fifteen points hugging the center line, is the counterintuitive one. It often means the limits are too wide because the subgroups mix several streams, or the data have been rounded, filtered, or smoothed upstream. A chart that looks beautifully calm is a warning, not a victory. Test 8, eight points on both sides but none near the center, signals a mixture of two distributions, typically two machines or two heads feeding one chart.

Choosing a rule set for streaming

More rules do not mean better protection; they mean more alarms. Let me put numbers on this. Take one chart receiving one subgroup per minute. Rule 1 alone gives one false alarm per 370 subgroups: about 3.9 per day. The four Western Electric rules combined give one per 91.75 subgroups: about 15.7 per day on that single chart. Across a hundred charts, that is over a thousand false alarms a day, and the operators will learn to ignore all of them.

My recommendation: run Western Electric 1, 2 and 4 as alarm-grade rules on critical-to-quality characteristics, keep the trend and stratification tests as advisory events that go to a maintenance queue and not a quality hold, and set the sampling interval from a false-alarm budget (alarms per shift per line), not from whatever the PLC scan time happens to be. That is an opinion grounded in the arithmetic above, not a standard.

EWMA and CUSUM: Memory for Small Shifts

Shewhart charts use only the latest subgroup (plus run rules that bolt on pattern memory). A one-sigma shift takes about 44 points on an individuals chart, as computed earlier. Charts with built-in memory accumulate evidence and respond sooner to small persistent shifts.

EWMA

The exponentially weighted moving average, per the NIST handbook’s EWMA section, is EWMA_t = λ·Y_t + (1 − λ)·EWMA_(t−1), with EWMA_0 set to the target or historical mean and 0 < λ ≤ 1. NIST states that λ is usually set between 0.2 and 0.3 and that λ = 1 reduces the chart to a Shewhart chart. The control limits use the variance approximation σ²·λ/(2 − λ) for large t, with k set to 3 or taken from the Lucas and Saccucci (1990) tables, which also give the ARL performance I am not reproducing here because I did not retrieve those tables.

For early samples, the exact time-varying limit is L·σ·√(λ/(2−λ)·(1−(1−λ)^(2t))), which is what the code below uses. The narrower early limits matter after a restart. Small λ gives long memory and the best sensitivity to small shifts but slow response to large ones; large λ behaves like Shewhart. A common engineering choice is to run a Shewhart chart alongside an EWMA with λ around 0.2, so large jumps and small drifts are both covered.

CUSUM

The tabular CUSUM accumulates deviations beyond an allowance k from the target, in sigma units: C⁺t = max(0, C⁺(t−1) + z_t − k) and C⁻t = max(0, C⁻(t−1) − z_t − k), signalling when either exceeds a decision interval h. The textbook design for detecting a one-sigma shift uses k = 0.5 and h in the range of 4 to 5; I use k = 0.5 and h = 5 below as a design choice, and you should select h from published ARL tables for your target ARL0, which I did not retrieve in this run.

The intuition: k is half the shift you want to detect, so deviations smaller than that are forgiven, and the max with zero resets the sum when the process looks fine. CUSUM tends to be slightly better than EWMA for a specific, known shift size, and it has a handy by-product: the point where the sum last left zero estimates when the shift began, useful when you need to know which lots to quarantine.

Observed behaviour on a synthetic shift

In the code that follows I generate 60 in-control points from N(10.000, 0.02²) and then 40 points from a mean of 10.015, a 0.75-sigma shift. With the seed used, the Western Electric rule 3 (four of five beyond one sigma) first fires at sample 71, CUSUM at sample 70, and EWMA (λ = 0.2, L = 3) at sample 71. These are one run of one synthetic stream, not a benchmark: a different seed will shift each by several samples. What the example does show is the qualitative pattern: a lone 3-sigma rule 1 would need roughly 80 points on average for a 0.75-sigma shift (1 / Φ(−2.25) ≈ 82), while the supplementary and memory detectors respond in about ten samples.

Capability: Cp, Cpk, Pp, Ppk with a Worked Example

Control charts tell you whether the process is stable. Capability indices tell you whether a stable process fits the specification. The order is non-negotiable.

The formulas

With specification limits LSL and USL, process mean μ, and a sigma estimate:

Cp = (USL − LSL) / (6σ). Potential capability if the process were centered.
Cpk = min(USL − μ, μ − LSL) / (3σ). Actual capability, penalising off-center processes.
Pp / Ppk use the same formulas with the overall (long-term) standard deviation of all the data, whereas Cp and Cpk use a within-subgroup sigma such as R-bar/d2 or MR-bar/1.128.

The difference between Cpk and Ppk is itself diagnostic. If Ppk is much smaller than Cpk, between-subgroup variation (drift, shifts) is inflating the overall sigma, and the process is not as stable as its short-term capability suggests. This is exactly the gap that a high-rate IIoT stream will expose quickly.

Worked example

Take a shaft diameter with LSL = 9.90 mm and USL = 10.10 mm. Suppose the process mean is 10.004 mm and the within-subgroup sigma is 0.020 mm. Then Cp = 0.20 / 0.12 = 1.67. The distances to the limits are 10.10 − 10.004 = 0.096 and 10.004 − 9.90 = 0.104, so the minimum is 0.096, and Cpk = 0.096 / 0.060 = 1.60. The small gap between Cp and Cpk says the process is well centered.

Now suppose the overall standard deviation, including shifts between subgroups, is 0.026 mm. Ppk = 0.096 / 0.078 = 1.23. A plant reading only Cpk sees a comfortable 1.60; one reading Ppk sees a process short of the common 1.33 target. The honest statement is that the process has capability of 1.60 when it is stable and performs at 1.23 over the observed period because it does not stay stable.

Indices to defect rates

For a centered, normal, stable process, Cpk = 1.00 corresponds to about 2,700 ppm out of specification (two-sided), Cpk = 1.33 to about 66 ppm (my calculation, using Cpk exactly 1.33 rather than 4/3, which gives about 63 ppm), and Cpk = 1.67 to roughly 0.5 ppm. Those translations assume normality and a perfectly known sigma. In practice, estimation error is large with small samples, which is why quality procedures demand 25 or more subgroups, and why a Cpk without a confidence interval should be treated as a rough indicator. The commonly cited minimums (1.33 for existing processes and higher for new or safety-related characteristics) are customer requirements, not laws of nature; read your customer-specific requirements.

Capability before control is a trap

If you compute Cpk on data containing special causes, the standard deviation includes shifts and the index is pessimistic, or, if the process happens to be centered between two shifts, deceptively optimistic. Compute capability only on the Phase I data after special causes have been removed, and recompute after any process change. A streaming system should therefore publish Cpk with the stability status next to it, never alone.

Runnable Python: Limits, Cpk, Rules, EWMA, CUSUM

The following is a self-contained numpy script. I ran it in the sandbox and it produced the numbers quoted in this article (Phase I limits for the n = 5 subgroup, Cp 1.67 and Cpk 1.60, ARL0 370.4, rule hits, and EWMA and CUSUM alarm indices).

import numpy as np
from math import erfc, sqrt
rng = np.random.default_rng(7)

# 1. Phase I limits for X-bar/R with n=5 (table constants)
A2, D3, D4, d2 = 0.577, 0.0, 2.114, 2.326
phase1 = rng.normal(10.000, 0.020, size=(25, 5))
xbar, R = phase1.mean(1), np.ptp(phase1, axis=1)
xbb, Rb = xbar.mean(), R.mean()
print("Xbar: CL=%.4f UCL=%.4f LCL=%.4f" % (xbb, xbb + A2*Rb, xbb - A2*Rb))
print("R: CL=%.4f UCL=%.4f LCL=%.4f" % (Rb, D4*Rb, D3*Rb))
sigma_hat = Rb / d2

# 2. Cp and Cpk
def cpk(mean, sigma, lsl, usl):
    return (usl - lsl) / (6*sigma), min(usl-mean, mean-lsl) / (3*sigma)
print("Cp, Cpk = %.2f, %.2f" % cpk(10.004, 0.020, 9.90, 10.10))

# 3. ARL0 for 3-sigma limits
p = erfc(3/sqrt(2)); print("ARL0 = %.1f" % (1/p))

# 4. Rule checks on standardised data
def rules(x, cl, sigma):
    z = (np.asarray(x) - cl) / sigma
    hits = {}
    def add(name, i): hits.setdefault(name, []).append(i)
    for i in range(len(z)):
        if abs(z[i]) > 3: add("WE1/N1", i)
        if i >= 2:
            w = z[i-2:i+1]
            if (w > 2).sum() >= 2 or (w < -2).sum() >= 2: add("WE2/N5", i)
        if i >= 4:
            w = z[i-4:i+1]
            if (w > 1).sum() >= 4 or (w < -1).sum() >= 4: add("WE3/N6", i)
        if i >= 7:
            w = z[i-7:i+1]
            if (w > 0).all() or (w < 0).all(): add("WE4", i)
        if i >= 8:
            w = z[i-8:i+1]
            if (w > 0).all() or (w < 0).all(): add("N2", i)
        if i >= 5:
            d = np.diff(z[i-5:i+1])
            if (d > 0).all() or (d < 0).all(): add("N3", i)
        if i >= 13:
            d = np.diff(z[i-13:i+1])
            if (d[1:] * d[:-1] < 0).all(): add("N4", i)
        if i >= 14 and (np.abs(z[i-14:i+1]) < 1).all(): add("N7", i)
        if i >= 7 and (np.abs(z[i-7:i+1]) > 1).all(): add("N8", i)
    return hits

x = np.concatenate([rng.normal(10, 0.02, 60), rng.normal(10.015, 0.02, 40)])
print({k: v[:3] for k, v in rules(x, 10.0, 0.02).items()})

# 5. EWMA with exact time-varying limits
def ewma(x, mu, sigma, lam=0.2, L=3.0):
    z, out = mu, []
    for t, v in enumerate(x, 1):
        z = lam*v + (1-lam)*z
        w = L*sigma*np.sqrt(lam/(2-lam)*(1-(1-lam)**(2*t)))
        out.append((t, z, abs(z-mu) > w))
    return out
print("EWMA first alarm after shift:",
      next((t for t, z, a in ewma(x, 10.0, 0.02) if a and t > 60), None))

# 6. Tabular CUSUM, k=0.5, h=5
def cusum(x, mu, sigma, k=0.5, h=5.0):
    cp = cm = 0.0
    for t, v in enumerate(x, 1):
        zz = (v-mu)/sigma
        cp = max(0, cp + zz - k); cm = max(0, cm - zz - k)
        if cp > h or cm > h: return t
print("CUSUM first alarm:", cusum(x, 10.0, 0.02))

Two details deserve comment. The rule implementation checks each window ending at the current index, so it is naturally streaming: state is just the last 15 standardised values per chart. And Western Electric 4 (eight on one side) is checked alongside Nelson 2 (nine) so you can see the earlier signal; in the shifted stream above WE4 fired at sample index 73 (zero-based), a point earlier than Nelson 2 at 74. In the run, Nelson 4 also fired once at index 24 during the in-control segment, a reminder that a rule fourteen points long still has a nonzero false-alarm rate across many charts.

Autocorrelation: Why High-Rate Sensor Data Breaks the Textbook

Every formula above assumes independent observations. A temperature sampled at 10 Hz from a furnace with thermal inertia is nothing like independent: each reading is largely the previous reading plus a little noise. This is positive autocorrelation, and it has two effects that compound.

First, the moving-range estimate of sigma shrinks, because consecutive values are close together, so MR-bar / 1.128 understates the true spread. Second, the true spread of the series around its mean is larger than an independent model would predict. Limits that are too narrow combined with a wandering series produce a flood of false alarms.

I simulated an AR(1) process, y_t = φ·y_(t−1) + ε_t, with 200,000 points, estimated sigma from the moving range as an I-MR chart would, and counted points beyond 3 naive sigma. The results come from one seeded run of my own script, so treat them as illustrative:

φ (lag-1 autocorrelation)	Naive sigma from MR	True standard deviation	False alarm per point	Effective ARL0
0.00	1.00	1.00	0.0026	about 383
0.50	0.82	1.15	0.033	about 30
0.80	0.75	1.66	0.178	about 6
0.95	0.72	3.18	0.500	about 2

At φ = 0.5, which is mild for sensor data, the nominal ARL0 of 370 collapses to about 30. At φ = 0.8 you receive an alarm every six points. This is a mechanism, not a rounding error, and it is the commonest reason IIoT SPC projects are abandoned: operators lose trust in a chart that cries wolf.

Remedies, in order of preference

Sample less often. If the autocorrelation decays in 30 seconds, chart one point per minute. Thinning is the simplest and most defensible fix, and it reduces the data volume too. Estimate the autocorrelation function on Phase I data and pick an interval at which lag-1 correlation is small.
Subgroup across time, not within a burst. Consecutive readings in a one-second burst are not a rational subgroup of independent observations; they are one observation measured repeatedly. Take summary statistics per interval and treat each interval as a point.
Model-based residual charts. Fit an AR or ARIMA model on Phase I data and chart the one-step-ahead residuals, which should be close to independent. This works but couples your SPC to a model that must itself be maintained, and it blurs the simple audit story.
Adjust the limits. Some methods widen the limits using the estimated autocorrelation. They are acceptable for slowly varying processes but are harder to explain to an auditor.
Use EWMA with care. An EWMA chart is also sensitive to correlation, and the limit formula above assumes independent data, as NIST explicitly notes.

A related trap is cross-correlation among variables: monitoring ten correlated temperatures with ten univariate charts multiplies the false alarm rate and ignores correlation structure. Multivariate charts such as Hotelling’s T² exist for that problem; they are beyond this article but belong in the same pipeline, and they give up the easy diagnosis that a univariate chart provides.

Streaming Implementation with Kafka, Flink, and Timescale

Figure 4: From reading to quality hold. The Flink job closes a subgroup window, the rule engine evaluates Western Electric and Nelson tests, the alarm topic drives the MES hold, and limit recalculation only happens after engineer approval.

Windowing and event time

A subgroup in a stream is a window. For time-based subgroups (one point per minute) use tumbling event-time windows with watermarks tolerant to the edge gateway’s buffering delay. For count-based subgroups (five consecutive parts) use a count window keyed by machine. Count windows are usually the better fit for discrete manufacturing because the subgroup is defined by parts, and a machine that stops produces no windows rather than empty ones.

Late data deserves an explicit policy. If a buffered reading arrives after its window has closed, either route it to a side output for a reconciliation job, or hold windows open for an allowed lateness equal to the longest expected outage and accept the delay in alarm latency. For SPC, silently dropping late readings is the worst choice, because run rules depend on the sequence and a missing point can break or invent a run.

A Flink-style skeleton

The sketch below is illustrative pseudocode in Java-like form to show where state lives; I have not compiled it. The essential idea is a keyed process function that holds the last N standardised values and the active limit version.

// Illustrative sketch, not compiled. Keyed by machineId + characteristicId.
class SpcRuleFunction extends KeyedProcessFunction<String, Subgroup, SpcAlarm> {
    ValueState<Limits> limits;           // broadcast-updated, versioned
    ListState<Double> lastZ;             // last 15 standardised subgroup means

    public void processElement(Subgroup sg, Context ctx, Collector<SpcAlarm> out) {
        Limits lim = limits.value();
        double z = (sg.mean - lim.centerLine) / lim.sigmaXbar;
        lastZ.add(z); trimTo(lastZ, 15);
        for (Rule r : activeRulesFor(sg.characteristicId)) {
            if (r.violated(lastZ)) {
                out.collect(new SpcAlarm(sg, r.id(), lim.version, z));
            }
        }
    }
}

Checkpointing the state means a job restart resumes the last 15 values rather than starting a blank run history, which would otherwise suppress alarms for 8 to 15 subgroups after every deployment, a failure mode that is quiet and unpleasant.

Persisting subgroup statistics in TimescaleDB

Store each subgroup as one row: machine, characteristic, window start and end, n, mean, range, min, max, limit version, and a flag list. A hypertable partitioned by time with a compound index on (machine, characteristic, time) serves both the dashboards and the Cpk reports. Continuous aggregates can roll up Ppk per shift and per week without re-reading raw data. Keep raw readings on a short retention (days) and subgroup statistics long (years): the subgroup rows are tiny and carry the audit trail.

-- Illustrative schema, adjust to your naming conventions
CREATE TABLE spc_subgroup (
  ts_end timestamptz NOT NULL,
  machine_id text NOT NULL,
  characteristic_id text NOT NULL,
  n int, mean double precision, range double precision,
  limit_version int, rules_fired text[]
);
SELECT create_hypertable('spc_subgroup', 'ts_end');
CREATE INDEX ON spc_subgroup (machine_id, characteristic_id, ts_end DESC);

Phase I versus Phase II

Phase I is retrospective: collect 20 to 25 subgroups (a commonly used guideline, not a theorem), plot them, remove subgroups with identified assignable causes, recompute, and repeat until the chart shows control. Phase II is prospective monitoring against the frozen limits. The pipeline should enforce the boundary. A characteristic whose limit record is in “Phase I” status produces charts and no alarms; once an engineer promotes the limits to Phase II, they become immutable and alarms become live. Recalculate only after an intentional process change (new tool, new material specification, maintenance), with the reason logged.

One subtlety: estimates from Phase I carry sampling error, so Phase II false-alarm rates are somewhat higher than nominal, especially with fewer than about 25 subgroups. This is why quality manuals ask for more data before limits are trusted.

Linking Alarms to MES and Quality Holds

An SPC alarm has value only if it changes what happens to the product. The alarm event should contain machine, characteristic, rule identifier, subgroup time window, the lot or serial range produced in that window, limit version, and a link to a chart snapshot. The MES consumes the topic and applies a policy by severity: rule 1 on a critical characteristic places the current lot on hold and notifies the quality engineer, whereas a trend or stratification test creates a maintenance work order without blocking production.

The lot genealogy is where IIoT earns its keep. Because the alarm carries the window boundaries, the MES can resolve which serial numbers were produced in the suspect interval. For a CUSUM alarm, the estimated change point extends the hold backwards to when the sum last left zero. On release, the engineer records a disposition (release, rework, scrap) and a corrective-action reference, which closes the loop; this record is what an automotive or medical auditor will ask to see.

Two integration hazards recur. First, alarm storms during a restart: after a changeover, run history from the previous recipe makes rules fire spuriously, so chart state should reset on a recipe-change event, and the first subgroups after startup should be flagged as warm-up. Second, idempotency: Kafka delivery is at least once, so duplicate alarm events must not place a lot on hold twice or send repeated pages; key the hold on (characteristic, window end, rule).

Trade-offs, Gotchas, and What Goes Wrong

Tampering. Adjusting a process in response to a single in-control point increases variation. Funnel experiments by Deming showed the effect. A pipeline that triggers automatic setpoint corrections from every deviation can double the variance. Close the loop only on rule violations, not on points.

Wrong sigma. Computing control limits from the overall standard deviation of all data, instead of the within-subgroup estimate, widens limits to include the very shifts the chart should catch. Use R-bar/d2, S-bar/c4 or MR-bar/1.128 for limits, and reserve the overall sigma for Pp and Ppk.

Specification limits on the chart. Operators then react to spec proximity and ignore the process signal. Keep them on separate displays.

Overlapping or rolling windows. Sliding windows reuse readings, so consecutive subgroup statistics are correlated, which invalidates run rules. Use non-overlapping subgroups for rule evaluation; use sliding windows only for display smoothing.

Quantisation and resolution. If the gauge resolution is coarse relative to sigma (a rule of thumb is that sigma should span at least several resolution steps), range values collapse to a few discrete values and the R chart misbehaves. Check measurement system capability first; an SPC chart on a poor gauge faithfully charts measurement noise.

Non-stationary processes by design. Batch processes, tool-wear sawtooth patterns, and recipes with multiple setpoints do not have a single center line. Use separate limits per recipe, or chart residuals from a model of the planned variation, and expect the simple Nelson trend test to fire constantly on legitimate tool-wear ramps.

Alarm fatigue. The combined-rule false-alarm arithmetic earlier is the quantified version of this old problem. Budget alarms per shift, and review rule hit rates monthly.

What SPC cannot do. It detects that something changed, not what. It is poor on high-dimensional interactions, which is where learned models complement it. And it says nothing about whether the specification itself is right.

Practical Recommendations

Start small and rigorous. Pick the five to ten characteristics that matter most to the customer, ideally the special characteristics already named in your control plan, and instrument those before building a platform. For each, verify the measurement system, determine whether rational subgrouping is possible, and decide on the chart type using Figure 3.

Estimate autocorrelation before choosing the sampling interval. Collect a Phase I dataset, clean it, and compute limits from within-subgroup sigma. Run Western Electric rules 1, 2 and 4 as alarm-grade, EWMA at λ about 0.2 beside a Shewhart chart for drift, and keep other Nelson tests advisory. Publish Cpk and Ppk together with a stability flag.

In architecture terms, keep limits versioned and external to the job, checkpoint rule state, key by machine and characteristic, process event time, and make alarm events idempotent. Wire alarms to the MES so a rule violation has an owner, a disposition, and a record.

A short checklist:

Control limits come from within-subgroup variation; specification limits stay off the chart.
ARL0 budget is set per line (alarms per shift), not left to the sampling rate.
Autocorrelation estimated; sampling thinned or residuals charted.
Phase I and Phase II are separate states; limits are immutable in Phase II.
Capability reported only on stable data, with both Cpk and Ppk.
Alarm events carry rule ID, window, lot range and limit version.
State survives restarts; chart state resets on recipe change.
Rule sets, constants, and edition references verified against your own standards.

Frequently Asked Questions

What is statistical process control in simple terms?

Statistical process control is the use of control charts and statistical rules to tell whether a process is behaving consistently over time. It separates normal, built-in variation from unusual events. You plot measurements against limits calculated from the process’s own stable history and act only when a point or pattern is unlikely to be chance, which avoids both over-reacting and missing real problems.

What is the difference between control limits and specification limits?

Control limits are computed from the process’s own data and describe what the process normally does. Specification limits come from the customer or design and describe what is acceptable. They are unrelated: a process can be in control yet produce nonconforming parts, or be out of control yet still within tolerance. Capability indices such as Cpk compare the two, but only for stable processes.

What is a good Cpk value?

A frequently used minimum is 1.33 for an existing process, which for a centered normal process corresponds to roughly 63 to 66 parts per million outside specification. Higher values, often 1.67, are demanded for critical or new processes. These are customer requirements and depend on your contract, so confirm them with the customer-specific requirements. A Cpk is also only meaningful if the process is stable and the estimate is based on enough data.

What is the difference between Western Electric and Nelson rules?

Both are run rules that supplement the 3-sigma limit. Western Electric has four rules, while Nelson has eight. Nelson tests 1, 5 and 6 match Western Electric 1, 2 and 3, and Nelson 2 is a nine-point version of the eight-point run. Nelson adds trend, alternation, stratification and mixture tests. More rules mean more sensitivity but also a higher combined false-alarm rate.

What is ARL0 and why is it 370?

Average run length in control, ARL0, is the average number of points plotted before a false alarm. For a normal, independent process with 3-sigma limits, each point has a 0.27 percent chance of being outside the limits, so the expected wait is 1 / 0.0027, about 370 points. Run rules and autocorrelation both lower this figure.

When should I use EWMA or CUSUM instead of a Shewhart chart?

Use EWMA or CUSUM when you care about small, sustained shifts, roughly under 1.5 sigma, that a Shewhart chart takes dozens of points to detect. They accumulate evidence over time, so they respond faster to subtle drift. Shewhart charts remain better for large, sudden jumps, so many plants run both side by side and merge their alarms.