Trade Surveillance Architecture: Detecting Spoofing, Layering and Wash Trading Under MAR and Dodd-Frank (2026)

Systems and architecture analysis only. This article is not legal, compliance or investment advice; regulatory interpretation belongs with qualified counsel.

In May 2010 a single futures trader kept sell orders worth roughly 20 to 29 percent of the visible sell side of the E-mini S&P 500 order book alive for hours, modified them more than 81,000 times, and had only 81 lots execute. The orders were never meant to trade. A decade later one bank paid $920.2 million to resolve spoofing allegations that spanned eight years and hundreds of thousands of orders. Both stories end the same way: the behaviour was visible in the order-event stream long before anyone acted on it.

That is the engineering problem behind trade surveillance. The abusive pattern is not a single bad trade but a relationship between orders, cancellations, fills and the book around them, and regulators in the EU and US now expect firms and venues to detect it systematically. This post walks through the legal definitions that shape requirements, the data model, rule-based scenarios with worked numbers, the limits of machine learning, alert economics, case handling, reporting and the platform that carries it all.

What this covers: legal definitions under MAR and the US Commodity Exchange Act, the abusive behaviours, the surveillance data model, alert scenarios with a numeric layering example and a Python cancel-ratio detector, ML limits, false-positive economics, STOR and SAR workflows, reference architecture, retention and model validation.

Context and Background

Market abuse regulation has two halves. The first defines prohibited behaviour; the second obliges firms and venues to detect and report it. Engineers mostly work against the second half, but the first half dictates what the data must be able to show.

In the European Union the core text is the Market Abuse Regulation, Regulation (EU) No 596/2014, known as MAR. Article 12 defines market manipulation as four families of activity: transactions or orders that give false or misleading signals or secure prices at an artificial level; transactions using a fictitious device or deception; disseminating misleading information; and manipulating a benchmark. A non-exhaustive second paragraph lists examples that matter directly for surveillance, including securing a dominant position, trading at the open or close to mislead, and placing, cancelling or modifying orders, including through algorithmic or high-frequency trading, in ways that disrupt the trading system, obscure genuine orders or create a false signal such as initiating or exacerbating a trend. Annex I then supplies indicators of manipulative behaviour, including orders that change the best bid or offer or the order book and are removed before execution (the layering indicator) and orders or trades around reference-price calculation times (the marking-the-close indicator). These indicators are factors to weigh, not proof.

The detection duty sits in Article 16: market operators and investment firms operating a trading venue must maintain effective arrangements, systems and procedures to prevent and detect insider dealing, market manipulation and attempted abuse, and anyone professionally arranging or executing transactions must report orders and transactions that give rise to a reasonable suspicion. The report is a Suspicious Transaction and Order Report, or STOR, filed with the competent authority. The ESMA peer review of STOR supervision, published in December 2019, found uneven quality across national authorities and recorded 11,130 STORs received in 2018, up from 10,653 in 2017, with investment firms the dominant submitters. It also flagged defensive reporting and cross-border delays as live issues.

In the United States the anti-spoofing prohibition arrived with the Dodd-Frank Act of 2010, which added Section 4c(a)(5) to the Commodity Exchange Act, codified at 7 U.S.C. 6c(a)(5). Subparagraph (C) makes it unlawful to engage in trading, practice or conduct that “is, is of the character of, or is commonly known to the trade as, ‘spoofing'”, defined as bidding or offering with the intent to cancel the bid or offer before execution. The Commodity Futures Trading Commission (CFTC) interpretive guidance states that recklessness is not enough, that a single instance can be a violation, and that partial fills are not automatically exempt. Criminal cases are also brought under the wire fraud statute, as in the 2020 JPMorgan deferred prosecution agreement.

Trade surveillance teams therefore face a structural mismatch. The legal test turns on intent, which no feed contains. What engineers can build is a system that surfaces behavioural evidence from which a human can infer intent, and that preserves that evidence in a form an investigator or regulator can replay. For the hardware and event-sequencing side of the venues generating this data, see our deep dive on exchange matching engine architecture. For the regulator’s own framing of STOR quality, the ESMA STOR peer review report is the primary source.

Reference Architecture for a Trade Surveillance Platform

A modern trade surveillance platform ingests every order event, rebuilds the market state around it, computes behavioural features, runs rules and models, and routes ranked alerts to analysts whose decisions can end in a regulatory report. The design goal is not speed alone but completeness, replayability and defensible evidence at each step.

Figure 1: Reference architecture for a trade surveillance platform, from raw feeds to regulatory filing.

Figure 1 reads left to right. Two streams enter: public exchange market data (for example Nasdaq ITCH style depth feeds or CME market data) and the firm’s own order flow from FIX gateways and order management systems. Both pass through a normalisation and timestamping layer, land in an immutable event log, feed an order-book reconstruction stage, and from there populate a feature store. Reference data and communications metadata join at the feature stage. A rules engine and an ML scoring service both write to an alert store, which case management consumes, and confirmed suspicion exits as a STOR or SAR.

Ingest and the immutable event log

The event log is the system of record. Every order new, amend, cancel, fill and reject is appended with its original venue timestamp, a firm receive timestamp, the source system and a content hash. Nothing is updated in place. This matters for two reasons: investigators must be able to show what the surveillance system saw at the time, and model validators must be able to replay history through a changed rule and compare results.

Streaming transport is typically a partitioned log such as Apache Kafka, partitioned by instrument so that a single consumer sees a coherent per-instrument sequence. Ordering within a partition is the property that matters; cross-instrument ordering is reconstructed from timestamps when needed. Clock quality is a first-order concern because layering and spoofing are defined by sub-second sequences. MiFID II RTS 25 sets clock synchronisation requirements for trading venues and participants, with the tightest tier for high-frequency algorithmic trading at 100 microseconds divergence from UTC. Trade surveillance cannot recover precision the sources never recorded, so capture hardware or gateway timestamps wherever possible.

Order-book reconstruction

Trade surveillance needs the book as it looked at each event, not just the events. Reconstruction replays the depth feed per instrument to maintain price levels, queue positions where the feed allows it, and the best bid and offer. It enables questions such as “what share of the visible offer side did this participant’s resting orders represent when the buy order executed?” That share is the single most useful quantity in layering detection.

Reconstruction must be deterministic. Given the same input log it must produce the same book, which means handling feed gaps, sequence-number recovery, auction phases and trading halts explicitly. Teams that treat reconstruction as a convenience rather than an engineered component discover during their first regulatory inquiry that they cannot reproduce what the market looked like. Our article on order book imbalance features covers the same reconstruction machinery from the alpha-research side, and the two workloads can share a library.

Feature store and the batch/stream split

Features are per-participant, per-instrument aggregates over sliding windows: order count, cancel count, order-to-trade ratio, mean order lifetime, share of visible depth, price improvement or deterioration after cancellation, and so on. A feature store gives rules and models a consistent definition and lets validators recompute historical values.

Most firms run a hybrid. A streaming path computes windowed features within seconds to catch live patterns, which matters for pre-emptive supervisor intervention. A batch path recomputes end-of-day with complete data, including late-arriving venue corrections, and is the version of record for regulatory filings. Where the two disagree, the batch result wins and the discrepancy is itself a data-quality metric.

Reference data and communications

Behaviour only looks suspicious against context. Reference data supplies instrument liquidity, tick size tables, trading calendars, auction schedules, benchmark fixing windows, restricted and watch lists, and the mapping from trader IDs to desks, accounts and beneficial owners. Wash-trade detection in particular depends on resolving different account identifiers to a common beneficial owner. Communications surveillance (email, chat, recorded voice) supplies the intent evidence that order data cannot. Joining a chat message that says “let’s push it up into the close” to the order pattern in the same minutes is what turns a statistical alert into a case, and it is why mature programmes integrate trade and communications surveillance rather than running them as separate silos.

What the Abusive Behaviours Look Like in Data

Before writing rules, it helps to be precise about what each behaviour is, because each leaves a different footprint. The definitions below are working engineering definitions; the legal tests are in MAR and the CEA and are applied by lawyers and regulators, not by code.

Spoofing is entering orders with the intent to cancel them before execution to create a false impression of supply or demand, typically to move the price in favour of a genuine order on the opposite side. Layering is the multi-level variant: several orders at successive price levels on one side, creating the appearance of depth, followed by a genuine order on the other side and cancellation of the layers. In the CFTC consent order for Navinder Sarao, an automated “dynamic layering” program placed four to six very large sell orders, each one price point from the next, starting three or four price points away from the best ask, and modified them hundreds of times so they stayed away from execution. The two terms overlap heavily; this post uses “layering and spoofing” together where the distinction does not change the detection logic.

Wash trading is trading with oneself, or with a counterparty under common control, to create the appearance of volume without a change in beneficial ownership. It matches Annex I of MAR, which lists transactions that do not result in a change of beneficial ownership as an indicator. The ESMA peer review noted that some national authorities treat wash trades as minor infringements handled through moral suasion and sometimes refer them to anti-money-laundering authorities, which tells you the supervisory appetite varies by jurisdiction even though the footprint is easy to detect.

Marking the close is placing orders or trades near the end of a session or around a benchmark fixing window to move the reference price. MAR names it in Article 12(2) as trading at the open or close that misleads investors relying on displayed prices, and Annex I lists orders or trades at or around times when reference, settlement or valuation prices are calculated. The US statute separately covers conduct that demonstrates intentional or reckless disregard for the orderly execution of transactions during the closing period.

Front-running is trading ahead of a known client order, using the information about that order. It is an inside-information abuse in EU terms and is detected by joining client order timestamps to proprietary or personal-account activity in the same instrument shortly beforehand. Insider trading is trading on inside information; surveillance can only flag statistical correlates such as unusually profitable, well-timed positions ahead of announcements, joined to restricted lists, deal calendars and personal-dealing records. Insider dealing detection is the weakest of the set from a pure data standpoint, because the key variable, whether the person possessed inside information, lives outside the market data.

Behaviour	Core footprint	Primary data needed	Main false-positive source
Spoofing / layering	High cancel rate, large depth share, opposite-side execution	Full-depth order events, book state	Market makers, algo hedging
Wash trading	Trades with no ownership change	Account-to-owner mapping, fills	Legitimate internal crossing
Marking the close	Aggressive flow near fixing	Order flow, fixing calendars	Index rebalancing, hedging
Front-running	Own-account trade before client order	Client order timestamps, personal accounts	Coincident research views
Insider trading	Profitable trades before news	Trades, news, restricted lists	Informed but legal research

The table shows why one model cannot serve all behaviours. Spoofing detection is primarily a time-series-over-order-book problem. Wash and front-running detection are primarily entity-resolution and join problems. Insider detection is an event-study problem with heavy reliance on non-market data.

Rule-Based Alert Scenarios with Parameters

Rule-based scenarios remain the backbone of regulated surveillance because they are explainable, testable and map cleanly onto regulatory language. A scenario is a pattern, a set of parameters and a severity logic. Parameters are not given by regulation; they are calibrated per asset class, venue and participant type, and documented as part of the model governance file.

The layering scenario

A typical layering scenario evaluates each participant and instrument over a rolling window and looks for four ingredients occurring together: a burst of orders on one side that adds a large share of visible depth; a high proportion of those orders cancelled with a short lifetime; an opposite-side order that executes while the burst is resting; and a cancellation of the burst soon after that execution. Illustrative starting parameters, to be tuned against your own data, might be: window of 5 seconds, at least 3 price levels, cancel ratio of at least 90 percent, order lifetime under 2 seconds, depth share of at least 25 percent of the visible side, and a genuine opposite-side execution within the window.

Worked example (illustrative numbers, not from a real case). A liquid futures contract trades with a tick of 0.01. At time zero the best bid is 100.00 and the best ask 100.01. Visible offer depth across the first five levels is 3,000 lots. A participant has a resting buy order for 500 lots at 100.00.

1. Over 1.2 seconds the participant enters five sell orders of 400 lots each at 100.02 through 100.06. Visible offer depth rises from 3,000 to 5,000 lots; the participant’s share is 2,000 / 5,000 = 40 percent.
2. Other participants see a heavier offer side. Several sell into the bid and the best bid drops to 99.99. The participant’s 500-lot buy order fills at 99.99, one tick better than where it started.
3. Within 0.4 seconds of the fill the participant cancels all five sell orders. Lifetime of each layer averaged about 1.6 seconds. Fills on the sell layers: zero. Cancel ratio: 2,000 / 2,000 = 100 percent.
4. The participant later sells the 500 lots into a recovered market. Net, the 500 lots were bought one tick below the pre-layering price.

Every parameter above trips: three or more levels, cancel ratio above 90 percent, lifetime under 2 seconds, depth share above 25 percent, opposite-side execution, cancellation within seconds of that execution. A single episode proves nothing on its own: a hedging algorithm might cancel quotes for legitimate reasons. Repetition does the work. If the same participant produces 14 such episodes in a day, always with an opposite-side fill landing between layer entry and cancellation, the aggregate score rises sharply and the case is routed with a higher priority.

Figure 2: One layering episode in sequence, which the scenario encodes as ordered events within a time window.

Figure 2 matters for design: the rule is a sequence, so the engine needs ordered per-instrument state, not just counters. The ordering constraint (layers resting before the genuine fill, cancellation after it) separates layering from ordinary quote refreshing, where cancels and fills have no stable relationship.

Order-to-trade and cancel ratios

Two ratios appear in almost every scenario library. The cancel ratio is cancelled quantity divided by entered quantity over a window. The order-to-trade ratio (OTR) is the number of order messages divided by the number of trades. MiFID II requires trading venues to calibrate order-to-transaction ratio limits to the nature of their market, and many venues impose their own thresholds, but regulation does not prescribe a universal figure; the ratio is a signal, not a verdict. Market makers legitimately show very high OTRs. A market maker with 1,200 messages and 10 trades in a minute (OTR of 120:1) is normal behaviour for a quoting algorithm, which is why OTR needs peer-group normalisation by participant type, instrument liquidity and time of day.

A cancel-ratio detector in Python

The sketch below flags participant-instrument windows combining a high cancel ratio, short lifetimes, large depth share and an opposite-side fill. It is deliberately minimal and uses an in-memory list of order events; a production version would run as a stateful stream processor.

from dataclasses import dataclass
from collections import defaultdict

@dataclass
class Evt:
    ts: float        # seconds
    pid: str         # participant
    sym: str
    oid: str
    kind: str        # NEW, CANCEL, FILL
    side: str        # B or S
    qty: int
    px: float

def detect(events, window=5.0, min_cancel=0.90, max_life=2.0,
           min_depth_share=0.25, visible_depth=None):
    """Flag layering-like windows. visible_depth(sym, side, ts) -> lots."""
    alerts, by_key = [], defaultdict(list)
    for e in sorted(events, key=lambda x: x.ts):
        by_key[(e.pid, e.sym)].append(e)

    for (pid, sym), evs in by_key.items():
        for side in ("B", "S"):
            opp = "S" if side == "B" else "B"
            news = [e for e in evs if e.kind == "NEW" and e.side == side]
            for start in news:
                w_end = start.ts + window
                layer = [e for e in news if start.ts <= e.ts <= w_end]
                if len({e.px for e in layer}) < 3:
                    continue
                ids = {e.oid for e in layer}
                canc = [e for e in evs if e.kind == "CANCEL" and e.oid in ids]
                fills = [e for e in evs if e.kind == "FILL" and e.oid in ids]
                entered = sum(e.qty for e in layer)
                cancelled = sum(e.qty for e in canc)
                if entered == 0 or cancelled / entered < min_cancel or fills:
                    continue
                born = {e.oid: e.ts for e in layer}
                life = [c.ts - born[c.oid] for c in canc]
                if max(life) > max_life:
                    continue
                share = entered / max(visible_depth(sym, side, start.ts), 1)
                if share < min_depth_share:
                    continue
                opp_fill = [e for e in evs if e.kind == "FILL" and e.side == opp
                            and start.ts <= e.ts <= w_end]
                if not opp_fill:
                    continue
                alerts.append({"pid": pid, "sym": sym, "side": side,
                               "ts": start.ts, "cancel_ratio": cancelled/entered,
                               "depth_share": round(share, 2),
                               "opp_fill_qty": sum(e.qty for e in opp_fill)})
    return alerts

Even this toy shows the design decisions that matter. The window and lifetimes are parameters, so they can be versioned and tuned. The depth share needs book reconstruction, hence the callback. The opposite-side fill requirement is the condition that most sharply cuts false positives, because it ties the cancellations to a benefit. Production code would also deduplicate overlapping windows, handle amendments (price modifications are often how layers are kept away from execution, as the Sarao order described), and attach evidence pointers to the event log so an analyst can open the exact sequence.

Machine Learning for Market Abuse Detection and Its Limits

The case for machine learning in market abuse detection is real. Rules capture patterns someone already thought of; abusers adapt, splitting activity across accounts, venues and instruments, or stretching timing just beyond a threshold. Models that score behaviour against a participant’s own history and against peers can surface activity no rule anticipated.

Three approaches are in common use. Unsupervised anomaly detection (isolation forests, autoencoders, clustering on feature vectors) scores how unusual a participant-day is. Supervised classification trains on historical alerts labelled by analysts as escalated or dismissed, effectively learning analyst judgement, and works as an alert-prioritisation layer on top of rules. Graph and sequence models examine relationships among accounts for wash-trade rings and cross-venue patterns, or learn order-lifecycle sequences directly.

The limits are equally real, and they are mostly about data, not algorithms:

• Label scarcity and bias. Confirmed abuse is rare. Most labels are analyst dispositions, which reflect what the existing rules surfaced, so supervised models inherit the blind spots of the scenario library. Training on dismissals teaches the model to reproduce today’s analysts, not to find tomorrow’s abuse.
• Class imbalance and unstable base rates. With positives measured in single units per million events, precision and recall estimates carry wide uncertainty, and a model that looks good in backtest may not hold.
• Explainability. An investigator must explain to a compliance head, and potentially a regulator, why something was flagged. A score of 0.93 without feature attribution is not evidence. Techniques such as per-alert feature contributions help, but a rules-based alert with named conditions is simpler to defend.
• Adversarial adaptation. Once thresholds are inferred, behaviour moves. Models trained on yesterday’s pattern are partially a map of what abusers have already stopped doing.
• Drift. Volatility regimes, tick-size changes and venue microstructure alter normal behaviour, and models silently decay.

The pragmatic architecture, and the one in Figure 1, runs rules and models in parallel into a single alert store. Rules provide the explainable, regulator-aligned core. Models provide anomaly coverage and a ranking signal that orders the queue, so the highest-value alerts reach analysts first. This is a defensible position: ML is used where its uncertainty is tolerable (prioritisation) and not where an unexplained output would become the stated basis for a STOR.

Alert Tuning, False-Positive Economics and Case Management

Trade surveillance is a queueing system with a regulatory floor. Every alert consumes analyst minutes, and almost all alerts are benign. The economics decide whether a programme is sustainable, and they explain why tuning is a permanent activity rather than a launch task.

Figure 3: The alert funnel. Each stage trades coverage against analyst workload, and outcomes feed back into thresholds.

The arithmetic of false positives

Consider an illustrative desk-level setup (the numbers are assumptions for the arithmetic, not industry benchmarks). A scenario library generates 2,000 alerts a day. Assume a first-level review takes six minutes on average. That is 12,000 minutes, or 200 analyst-hours, per day, roughly 25 full-time reviewers. If 98 percent are dismissed, then 1,960 of those reviews produce nothing but a documented rationale.

Now tune. Add peer-group normalisation so market makers are compared with market makers, suppress alerts that duplicate an open case, and auto-enrich with the context an analyst would otherwise look up. Suppose daily alerts fall to 600 and review time to four minutes. The load becomes 2,400 minutes, or 40 hours, about 5 reviewers. The lesson is not that fewer alerts is better; it is that unit review cost and alert volume multiply, and both can be engineered. Enrichment (showing the book snapshot, the participant’s 30-day alert history and linked chats on one screen) is often a larger lever than threshold changes.

The risk on the other side is under-alerting. Raising thresholds to cut volume trades false positives for false negatives, and a missed pattern is a regulatory failure, not an efficiency gain. The JPMorgan order found that the bank failed to identify, investigate and stop the conduct despite red flags that included internal surveillance alerts and inquiries from the exchange and the CFTC. That is a case where alerts existed and the process around them did not work. Tuning is therefore always documented: what changed, why, what the below-threshold sample looked like, and who approved.

Below-the-line testing

Responsible tuning samples alerts that did not fire. For each scenario, periodically lower the thresholds on a copy, run the lower setting over a recent period, and have analysts review a random sample of the extra alerts. If material suspicious patterns appear below the line, the threshold is too high. This below-the-line testing is a standard governance expectation and costs little because it runs offline on the replayable event log.

Case management

An alert is a hypothesis; a case is an investigation. Case management systems group related alerts by participant, instrument and time, attach the evidence (book snapshots, event sequences, communications), record the analyst’s reasoning, and enforce workflow: first-level triage, second-level investigation, compliance officer decision, and escalation. Four properties matter for audit:

• Immutability of the evidence. The case holds pointers to event-log ranges plus a rendered snapshot, so the evidence cannot change if a venue later corrects data.
• Complete disposition rationale. A dismissal must say why in terms that a reviewer can check, not “no further action”.
• Segregation of duties. The person who tunes a scenario should not be the person who dismisses its alerts.
• Time tracking. Time from alert to decision is a management metric and, where reporting deadlines apply, a compliance control.

Good case tools also expose a replay view: a time-scrubbable reconstruction of the book with the participant’s orders overlaid. Investigators who can watch Figure 2 happen in the actual market are faster and more consistent than those reading event tables.

Regulatory Reporting: STOR and SAR

When a case reaches reasonable suspicion, the output is a report. The EU and US regimes differ in structure, and a global firm often runs both.

STOR in the EU and UK. MAR Article 16 requires persons professionally arranging or executing transactions to report orders and transactions, including cancellations and modifications, where there is reasonable suspicion of market abuse. The ESMA peer review quotes the requirement that reports be sent “immediately” and notes that “without delay” may be more workable in less urgent cases; it also records the cross-border distribution problem, since the competent authority of the venue and of the firm may differ. STORs are submitted in a standard format set by delegated EU rules, and the report must contain identification of the persons, the instruments, a description of the suspicious activity and the reasons for suspicion. Because the threshold is suspicion, not proof, a STOR is not an accusation, and the ESMA review stresses that STORs should not automatically lead to investigations.

SAR in the US. US broker-dealers file suspicious activity reports with the Treasury’s FinCEN under Bank Secrecy Act rules, generally within 30 calendar days of initial detection of facts that may constitute a basis for filing (60 days if no suspect is identified). Futures and derivatives intermediaries have parallel obligations, and exchanges and self-regulatory organisations conduct their own surveillance and refer matters to the CFTC or SEC. I have not verified current form and deadline details in this run; confirm them against FinCEN guidance and counsel before encoding deadlines in workflow rules.

Engineering implications are practical. The case system should generate the narrative skeleton from structured evidence, enforce mandatory fields, record filing time and reference numbers, and maintain a tipping-off control: access to a case that leads to a report must be restricted and subjects must not be alerted. Our related piece on sanctions screening and watchlist filtering architecture covers the same need for restricted-access case handling and shared entity resolution on the financial-crime side, and the two systems increasingly share identity data.

Retention, Replay and Audit Requirements

The common thread in all of this is that you must be able to reconstruct what happened. Retention requirements vary by regime and record type, and I recommend treating the following as design requirements rather than quoting a single number.

First, order and transaction records must be kept long enough to answer a regulator’s inquiry, which in the EU generally means at least five years under MiFID II record-keeping rules, longer in some national regimes at a regulator’s request, and with different periods under US SEC and FINRA rules for broker-dealers (verify the applicable rule and period with counsel). Second, surveillance outputs have their own retention: alerts, dispositions, case files, the parameter versions in force and the reports filed. Third, communications records follow their own rules and are often the largest storage item.

Design consequences:

1. Write-once storage with legal hold. Object storage with immutability policies (WORM) for the event log and case evidence, with a hold mechanism that overrides lifecycle deletion when an inquiry arrives.
2. Versioned parameters. Every alert stores the scenario version and parameter set that produced it. Without that, you cannot explain why last March’s activity did or did not alert.
3. Point-in-time reproducibility. Given a date and a scenario version, the platform must regenerate the same alerts from the event log. This is the property that proves the system was working, and it is what regulators mean by being able to demonstrate effective arrangements.
4. Tiered storage. Hot storage for recent days to drive replay and analytics; warm columnar storage for the retention window; cold for the tail.
5. Access control and logging. Trade surveillance data contains client orders and personal data. Access is role-based, logged and reviewed, and subject to data-protection rules such as the GDPR where applicable.

Model Validation and Governance

Trade surveillance scenarios and models are models in the governance sense. Banking supervisors expect independent validation, and the same discipline helps with market abuse regulators. The process in Figure 4 describes a typical change lifecycle.

Figure 4: Change and validation lifecycle for surveillance rules and models, closing the loop with production monitoring.

A candidate change is backtested on labelled history, then replayed against the stored event log to measure alert volume and overlap with the incumbent. Targets are set on recall of known cases (seeded or historical) and on volume. An independent validation team challenges conceptual soundness, data quality, parameter choices and limitations, and signs off. A shadow run in production compares the new version’s alerts with the live one without reaching analysts. Only then is it released as a versioned artefact. Monitoring then watches alert yield (the proportion escalated), distribution shift in features and the volume per analyst, feeding the next change.

Testing with synthetic and seeded scenarios

Because confirmed cases are scarce, teams inject synthetic abusive patterns into replayed data and check that the scenario fires, a technique known as seeding. Seeding validates logic and data plumbing, but not real-world recall, since synthetic patterns reflect the designer’s imagination. Combine it with historical confirmed cases (such as the pattern in public enforcement orders), peer-benchmarking exercises, and independent red-team style exercises where a separate group designs evasive behaviour to test the model.

Documentation

A validator or regulator will ask for: the regulatory mapping (which MAR or CEA behaviour each scenario addresses), parameter rationale, data lineage, known limitations, test results and change history. Producing this from the version control history and the parameter registry, rather than writing it by hand each year, is a mark of a mature programme.

Trade-offs, Gotchas, and What Goes Wrong

Honest trade surveillance engineering means accepting limits.

Intent is not observable. The CEA test for spoofing is intent to cancel, and the CFTC guidance says recklessness is not enough. No data field encodes intent. Systems surface pattern evidence; people judge intent. Over-claiming in a case narrative that a model “detected spoofing” confuses these roles.

Legitimate behaviour looks like abuse. Market makers cancel constantly, and the CFTC guidance itself says orders cancelled in good faith as part of a legitimate attempt to complete a trade are not spoofing. A scenario without peer-group context will bury analysts in market-maker alerts.

Data gaps create blind spots. Cross-venue and cross-asset abuse (spoofing in a future to benefit a cash position, or layering on one venue and executing on another) requires joined data, and many platforms still surveil per venue. Missing client-to-owner mappings kill wash-trade coverage.

Clock and sequence errors manufacture or destroy patterns. A 5-millisecond timestamp error can reorder a cancel before a fill and flip a verdict. Document timestamp provenance for every source.

Alert fatigue is a control failure. The JPMorgan order is the cautionary example of alerts without effective follow-through. Volume without disposition quality is worse than a smaller, well-worked queue.

Model sprawl. Adding scenarios without retiring any inflates maintenance cost and review burden. Each scenario needs an owner, a purpose and a retirement criterion.

Regulatory divergence. Intent standards, thresholds for reporting and permitted data use differ across jurisdictions, so a global design must support jurisdiction-specific parameters rather than one universal rule set.

Practical Recommendations

Start from the evidence chain, not the algorithms. A firm that can reliably capture every order event with trustworthy timestamps, reconstruct the book, replay history and retain cases has the foundation to improve detection continuously. A firm with clever models on partial data has a fragile programme.

Build rules first for the behaviours your business can actually commit: layering and spoofing for anyone with algorithmic or high-volume order flow, wash trading wherever accounts can interact, front-running wherever a client-facing desk sits beside a proprietary one. Add ML as a ranking and anomaly layer once you have enough analyst dispositions to measure it against. Invest early in enrichment and case tooling, because review time is the largest operating cost.

Use this checklist as a pre-go-live review:

• Every source has a documented timestamp origin and an agreed clock-synchronisation target.
• The event log is append-only, hashed and replayable by date and instrument.
• Order-book reconstruction is deterministic and gap-handling is tested.
• Each scenario maps to a regulatory behaviour, has a named owner and a versioned parameter set.
• Peer-group normalisation exists for market makers and other legitimately high-cancel participants.
• Below-the-line testing and seeded-pattern tests run on a schedule.
• Cases hold immutable evidence, rationale and timings, with access restricted for reportable matters.
• STOR and SAR workflows are tested end to end, with deadline logic verified against current rules and counsel.
• Retention, legal hold and access logging are configured and tested by a restore drill.
• An independent validator has reviewed the scenarios and models, and drift monitoring is live.

Frequently Asked Questions

What is trade surveillance?

Trade surveillance is the monitoring of orders, trades and related communications to detect market abuse such as spoofing, layering, wash trading, insider dealing and front-running. Systems capture order events, rebuild market state, run rule-based scenarios and models, and route alerts to analysts. Confirmed suspicion goes to regulators as a STOR in the EU and UK or a SAR in the US. Venues and firms both run it.

What is the difference between spoofing and layering?

Spoofing is entering orders with intent to cancel before execution, creating a false impression of supply or demand. Layering is a form of spoofing that uses several orders at different price levels on one side to simulate depth, followed by a genuine trade on the other side and cancellation of the layers. Detection logic is similar, with layering requiring multi-level depth analysis. Regulators treat both as manipulative.

How does MAR define market manipulation?

Article 12 of Regulation (EU) 596/2014 defines market manipulation as transactions or orders that give false or misleading signals or secure an artificial price, transactions using fictitious devices or deception, dissemination of misleading information, and benchmark manipulation. It lists examples including dominant positions, trading at the open or close, and order placement, cancellation or modification, including algorithmic. Annex I provides non-exhaustive indicators that supervisors and firms weigh.

What is the US legal test for spoofing?

Under Section 4c(a)(5)(C) of the Commodity Exchange Act, added by Dodd-Frank, spoofing is bidding or offering with the intent to cancel the bid or offer before execution. CFTC interpretive guidance says recklessness is insufficient, a single instance can violate the provision, and good-faith cancellations while trying to complete a trade are not spoofing. Intent is usually inferred from patterns, fill characteristics and market context.

Can machine learning replace rule-based surveillance?

Not in regulated settings today. Models help with anomaly detection and alert prioritisation, but labels are scarce and biased towards past rule outputs, explainability is limited, abusers adapt and models drift. Most firms run rules and models in parallel: rules supply explainable, regulator-aligned coverage, and models rank the queue and add coverage for unanticipated patterns, all under independent validation.

How many alerts are false positives?

There is no universal figure, and I am not aware of a reliable published industry benchmark. Practitioners commonly report that the large majority of alerts are dismissed, which is why peer-group normalisation, enrichment and below-the-line testing matter. Measure your own alert yield (escalated divided by reviewed) per scenario, track it over time and use it to decide where tuning effort pays off most.

Further Reading

• Exchange matching engine architecture for low latency for how the venue generates the event stream surveillance consumes.
• Order book imbalance features in market microstructure for shared reconstruction and feature techniques.
• Sanctions screening and watchlist filtering architecture for adjacent financial-crime case handling.
• Regulation (EU) No 596/2014, Article 12 for the manipulation definition as adopted, and Annex I for the indicators.
• 7 U.S.C. 6c, disruptive practices and the CFTC release on the JPMorgan resolution for the US framework and enforcement.

By Riju — about