Payment Fraud Feature Engineering: Velocity Counters and Online Stores

A fraud model is only as good as the features it can read in the few milliseconds it has. The classifier itself, a gradient-boosted tree or a small neural network, is usually the easy part. The hard part is answering questions such as “how many distinct merchants has this card touched in the last ten minutes?” while the authorization is still waiting, and then answering the same question identically when you rebuild the training set six months later.

This matters now because instant payment rails and card-not-present volume have shortened the window in which a bad transaction can be stopped, while the cost of a wrong decline stays high. The features that carry most of the signal are aggregates over recent history, and aggregates are exactly where systems quietly go wrong.

You will leave with a practical design for payment fraud feature engineering: which velocity features earn their keep, how sliding and tumbling windows differ in state and accuracy, when approximate sketches beat exact counters, how to keep training and serving consistent, and a latency budget you can defend. Code is runnable; numbers I could not verify are labelled illustrative.

What this covers: feature families, window mechanics, exact versus approximate counting, the streaming and online-store topology, point-in-time correctness and label delay, backfill, a latency budget, failure modes and a checklist.

Disclaimer: this is a systems-architecture discussion, not legal, compliance or financial advice.

Context and Background

Card and account-based fraud systems score each event using a mix of three signal types: attributes of the event itself (amount, currency, merchant category, entry mode), attributes of the entities involved (card age, account tenure, device type), and behaviour over time (how those entities have acted recently). The third group is where velocity features live, and it is the group that requires state. Event attributes can be computed from the request; behaviour needs a memory.

The end-to-end shape of a scoring stack is covered in our overview of real-time fraud detection architecture. This post zooms into one box of that picture, the feature layer, and takes an angle the broader posts do not: the aggregate itself as a first-class engineered object with a defined window semantics, an error budget, a freshness guarantee and a replay story. If you want the generic online/offline parity discussion, see feature store architecture and online-offline parity; here the focus is what is special about payments.

Payments are special for four reasons. First, the decision sits in the authorization path, so a late answer is a default answer: the issuer or processor applies a fallback, usually approve or a static rule, when the model times out. Second, entity cardinality is huge and skewed: a handful of merchants and BIN ranges generate most traffic, while most cards appear a few times a month. Third, labels are late and noisy: a chargeback can arrive weeks or months after the transaction, and many frauds are never labelled. Fourth, adversaries adapt, so a feature that works today can be probed and evaded tomorrow.

Standards and vendor documentation give useful anchors rather than recipes. Redis documents HyperLogLog as using up to 12 KB per key with a 0.81 percent standard error, which is the figure most teams quote when sizing distinct-count features (Redis HyperLogLog docs). Apache Flink documents tumbling, sliding and session windows, and is explicit that a sliding window with a small slide multiplies state because each element belongs to size divided by slide windows (Flink windowing docs). The Feast project documents point-in-time joins as the mechanism that prevents future data leaking into training rows (Feast point-in-time joins). Those three ideas, bounded memory, bounded state and temporal correctness, organise everything below.

What counts as a fraud feature

It helps to name the families before engineering them. Velocity features count or sum events per entity over a window: transactions per card in 1 minute, 10 minutes, 1 hour and 24 hours; amount sum per account; declined attempts per device. Diversity features count distinct values: distinct merchants per card, distinct cards per device, distinct countries per account. Ratio and delta features compare windows: the last 10 minutes against the trailing 7-day average, or the current amount against the entity’s median. Recency features hold the time since a last event: seconds since the card’s previous transaction, days since a password reset. Graph features describe shared infrastructure: how many accounts share this device, address or phone number.

Each family has a different state shape. Counts are one integer per window. Sums are one number. Distinct counts need a set or a sketch. Recency needs one timestamp. Graph features need an adjacency structure. Choosing a feature is therefore also choosing a memory and update cost, and a good feature review asks both “does it carry signal?” and “what does it cost to keep fresh at p99?”

Reference Architecture: The Feature Path Around the Authorization

Direct answer: a real-time fraud feature layer has two paths. A read path fetches precomputed aggregates from a low-latency online store during authorization. A write path consumes the event log, updates windowed aggregates through a stream processor and also lands raw events in an offline store, so training can recompute identical features with point-in-time joins.

Figure 1: Reference architecture for payment fraud feature engineering. The decision service only reads; aggregates are written by the stream processor and mirrored to the offline store.

The diagram shows the separation that matters. The decision service reads counters from the online store and publishes the authorization event to the log. It does not compute aggregates inline, because inline computation couples the p99 of the decision to the cost of the most expensive feature. The stream job consumes the log, partitions by entity key, updates aggregates and upserts them to the online store. A second consumer appends immutable events to the lakehouse, which is the source of truth for training and backfill.

Read path versus write path

The read path is a key-value lookup, ideally one round trip. A single pipelined or multi-get request fetches all aggregates for the card, device, account and merchant involved. The write path is asynchronous and has its own freshness objective, for example “an event is reflected in the online aggregate within 500 ms at p99”. That freshness figure is a design choice you set, not a number to copy; it determines how much of a burst an attacker can push through before the counters catch up.

This split creates a subtle but important property: the feature value read at decision time never includes the current event, and may exclude the few most recent events still in flight. That is acceptable if, and only if, training reproduces exactly the same staleness. This is the root of training-serving skew, and we return to it in the point-in-time section.

Entity keys and the cost of fan-out

Every event touches several entities, so one authorization produces several counter updates. A card-not-present payment typically updates at least the card (or its network token), the account, the device or browser fingerprint, the IP address or subnet, the merchant and the shipping address. If each entity carries ten windowed features, one event triggers dozens of writes. This write amplification, not read volume, is what usually dictates the sizing of the online store.

Two design rules follow. Keep the entity set small and justified: each new key type multiplies writes and state. And prefer one hash or one structured value per entity and window family over many separate keys, so a read is one lookup per entity rather than one per feature. Where network tokens replace the primary account number, decide whether velocity is keyed on the token, the underlying card or both, because tokens can rotate while the funding account stays the same; our post on network tokenization architecture explains why this matters.

Where the online store sits

Redis, Valkey and Dragonfly are the common choices for sub-millisecond reads with server-side atomic operations; the trade-offs are discussed in Valkey vs Redis vs Dragonfly. Managed feature stores such as Feast, Tecton or cloud-native offerings wrap an online store and add registry, materialisation and point-in-time tooling. Whether you adopt one or write thin glue depends on team size; the invariants below hold either way. I am not claiming vendor-specific latency figures here, because published numbers depend heavily on payload, network and hardware.

Windows: Tumbling, Sliding and Bucketed

The window definition is the most consequential choice in a velocity feature, because it determines both the semantics the model learns and the state you hold.

Figure 2: Three ways to maintain a one-hour count. Tumbling windows are cheapest but blind at boundaries; true sliding windows are exact but costly; a bucketed ring approximates sliding at low cost.

A tumbling window partitions time into fixed, non-overlapping intervals; each event belongs to exactly one. State is one counter per interval per key, and the update is O(1). The flaw is the boundary. If a fraudster sends 20 attempts split evenly across 59:50 to 00:10, a “transactions in the current hour” counter sees 10 and 10 and never alarms. Boundary blindness is the classic reason pure tumbling windows are unsuitable for velocity rules.

A true sliding window reports, at any query time t, the count of events in (t minus W, t]. It is exact and has no boundary artefacts, but it requires per-event state: either a sorted collection of timestamps per key or a log you can trim. Memory grows with the event rate per key, not with the window count, so a hot merchant over 24 hours can be very large while a typical card is tiny.

A bucketed ring, sometimes called a smoothed or sub-windowed counter, divides the window into N small tumbling buckets (say six ten-minute buckets for one hour) and answers a query by summing the live buckets. State is N integers per key. The error is bounded by one bucket width: the oldest bucket is either wholly included or wholly excluded, so the effective window is between W minus one bucket and W. Making N larger shrinks the error and raises cost linearly.

Flink’s version of the trade-off

In Flink, a sliding window is declared with a size and a slide, and each element is assigned to every window whose range contains it. The documentation’s example uses a 10-minute size with a 5-minute slide, which places each element in two windows; a one-hour window sliding every second would place each element in 3,600, which is the state multiplication the docs warn about. In practice, teams either choose a coarse slide that matches the bucket width, or bypass windows and keep their own keyed state in a ring, which gives finer control over memory and time-to-live.

Session windows are the third assigner and suit behavioural features: a session closes after an inactivity gap and merges when late events bridge two sessions. They are useful for “events per browsing session” but are unsuitable for card velocity because their boundaries move with the attacker’s own pacing.

Event time, processing time and lateness

Velocity must be defined on event time, the time the authorization occurred, not the time the stream job processed it. Otherwise a consumer lag spike silently shortens every window and the same event can land in different buckets on replay. Flink’s default allowed lateness is zero, so events arriving after the watermark passes the window end are dropped; setting allowed lateness keeps state longer and can re-fire a window, so downstream upserts must be idempotent and tolerate updated values. A side output captures dropped late events so you can measure how much data you are discarding rather than guessing.

For payments, late events are routine: mobile apps batch uploads, acquirers retry, and reversals and advices arrive after the original. A pragmatic policy is to count late events in the buckets they belong to when they are within a bounded lateness, route the rest to a side channel, and track the late-event rate as a first-class metric. If the late-event rate climbs, your offline features will drift away from your online ones, because the offline job sees the full history.

Exact Versus Approximate Counting

Counts and sums are cheap to keep exactly. Distinct counts are not, and fraud leans on them heavily: distinct cards per device is among the most predictive card-testing signals. An exact set per device per window costs memory proportional to cardinality, and the cardinality is precisely what an attacker inflates.

HyperLogLog for distinct counts

HyperLogLog estimates cardinality in fixed memory. Redis documents up to 12 KB per key and a 0.81 percent standard error, with PFADD, PFCOUNT and PFMERGE as the commands (Redis docs). For fraud, 0.81 percent is irrelevant in the regime that matters: the difference between 3 and 4 distinct cards is a threshold you set on a model feature, and a decision at 40 distinct cards is not sensitive to a one percent wobble. Note, however, that the standard error describes the large-cardinality regime; at tiny cardinalities Redis uses a sparse representation that is exact or near-exact, which suits the common case of a device seen with one or two cards.

HyperLogLogs merge. This is what makes bucketed distinct counts possible: keep one HLL per bucket and PFMERGE the live buckets at read time to get a distinct count over the window. Merge cost grows with bucket count, so keep N modest and cache merged results for hot keys for a few hundred milliseconds if reads dominate. The mechanics of the structure and its relatives are covered in our guide to probabilistic data structures: Bloom filters, HyperLogLog and count-min sketch.

What HLL cannot do is tell you which members are present, and it cannot delete. If an investigator needs the list of cards on a device, you need the raw events or a bounded exact set kept alongside. A common compromise is a capped exact set (for example the first 50 distinct values) plus an HLL for the overflow count.

Count-min sketch for heavy hitters

A count-min sketch estimates frequencies for many keys in sub-linear memory. With width w = ceil(e / epsilon) and depth d = ceil(ln(1 / delta)), a point query overestimates by at most epsilon times the total count N with probability at least 1 minus delta (Cormode and Muthukrishnan). It never underestimates, which matters: it can raise false alarms on busy keys but will not hide a heavy hitter. Use it where the key space is unbounded and you want “is this BIN, IP block or merchant receiving unusual volume?” rather than per-card precision. Because the overestimate scales with total traffic N, sketches should be windowed too, typically one sketch per bucket and summed.

Decision guide

Feature	Exact structure	Approximate option	Recommendation
Txn count per card per window	Integer buckets	None needed	Exact ring
Amount sum per account	Number buckets	None needed	Exact ring
Distinct cards per device	Set	HyperLogLog	HLL plus capped set
Distinct merchants per card	Small set	HyperLogLog	Exact set (low cardinality)
Volume per IP block	Hash per block	Count-min	Sketch for the long tail
Time since last event	One timestamp	None	Exact

The rule I apply: use exact state wherever per-entity cardinality is naturally small and bounded, and an approximate structure only where an adversary can inflate it. Approximation should be a deliberate, documented property of the feature, because it must be reproduced identically offline, including the hash function and sketch parameters.

Implementation: A Bucketed Velocity Counter in Redis

The following is a minimal, runnable pattern: an atomic Lua script that increments the current bucket, expires old state and returns the live sum in one round trip. It uses the standard Redis commands HINCRBY, HDEL, HGETALL and EXPIRE inside EVAL, so the update and read are atomic with respect to other clients. Treat it as a teaching sketch; benchmark it against your own payloads before relying on it.

import time
import redis

BUCKET_SECONDS = 600          # 10 minutes
BUCKETS = 6                   # 1 hour window
WINDOW = BUCKET_SECONDS * BUCKETS

LUA = """
local key     = KEYS[1]
local bucket  = tonumber(ARGV[1])   -- current bucket id
local oldest  = tonumber(ARGV[2])   -- oldest live bucket id
local amount  = tonumber(ARGV[3])
local ttl     = tonumber(ARGV[4])
redis.call('HINCRBY', key, bucket .. ':n', 1)
redis.call('HINCRBYFLOAT', key, bucket .. ':s', amount)
redis.call('EXPIRE', key, ttl)
local all = redis.call('HGETALL', key)
local n, s = 0, 0.0
for i = 1, #all, 2 do
  local field = all[i]
  local b, kind = string.match(field, '(%d+):(%a)')
  b = tonumber(b)
  if b < oldest then
    redis.call('HDEL', key, field)
  elseif kind == 'n' then
    n = n + tonumber(all[i + 1])
  else
    s = s + tonumber(all[i + 1])
  end
end
return {n, tostring(s)}
"""

r = redis.Redis(decode_responses=True)
velocity = r.register_script(LUA)

def record_and_read(entity: str, event_ts: float, amount: float):
    bucket = int(event_ts // BUCKET_SECONDS)
    oldest = bucket - BUCKETS + 1
    n, s = velocity(
        keys=[f"vel:{entity}"],
        args=[bucket, oldest, amount, WINDOW + BUCKET_SECONDS],
    )
    return int(n), float(s)

if __name__ == "__main__":
    now = time.time()
    for amt in (12.5, 40.0, 9.99):
        print(record_and_read("card:4111xxxx", now, amt))

Two caveats. First, this reads-after-write, so the returned count includes the current event, which contradicts the earlier rule that decision-time features exclude it. In production, split the roles: the decision service does a read-only call (the same script without the HINCRBY lines, or a plain HGETALL), and the stream job performs the write. The version above is the write-side updater. Second, the bucket id is derived from event time passed in, not from the Redis server clock, which keeps replay deterministic.

HyperLogLog distinct count with merge

HLL_BUCKET = 600
def add_card_to_device(device: str, card: str, event_ts: float):
    b = int(event_ts // HLL_BUCKET)
    key = f"hll:dev:{device}:{b}"
    pipe = r.pipeline()
    pipe.pfadd(key, card)
    pipe.expire(key, HLL_BUCKET * 8)
    pipe.execute()

def distinct_cards(device: str, now: float, buckets: int = 6) -> int:
    b = int(now // HLL_BUCKET)
    keys = [f"hll:dev:{device}:{b - i}" for i in range(buckets)]
    return r.pfcount(*keys)   # PFCOUNT over several keys merges on the fly

PFCOUNT accepts multiple keys and returns the cardinality of their union, so the merge is done server side. Hash the card identifier (with a keyed hash and a salt held outside the store) before adding, so the online store never holds raw primary account numbers; this narrows PCI DSS scope but is a design aim, not a compliance determination.

The Streaming Write Path with Flink

The write path has to be ordered per entity, fault tolerant and replayable. Kafka partitioned by entity key gives ordering within a key; Flink keyed state gives exactly-once state updates under checkpointing; the sink to the online store should be an idempotent upsert, because a restart replays events from the last checkpoint. The broader engine comparison is in Flink vs Spark Streaming vs Kafka Streams, and if the source events originate in a database, change data capture with Debezium is the usual on-ramp.

Figure 3: Sequence for one authorization. The decision service reads, publishes the event, and the stream job updates aggregates asynchronously while a second consumer lands the event in the offline store.

A keyed-state bucket ring in PyFlink

The following sketch keeps the ring in Flink keyed state rather than using window assigners, which avoids the per-element window multiplication. It is illustrative; adapt the sink and serialization to your platform.

from pyflink.datastream import KeyedProcessFunction, RuntimeContext
from pyflink.datastream.state import MapStateDescriptor
from pyflink.common.typeinfo import Types

BUCKET_MS = 600_000
BUCKETS = 6

class VelocityRing(KeyedProcessFunction):
    def open(self, ctx: RuntimeContext):
        self.ring = ctx.get_map_state(
            MapStateDescriptor("ring", Types.LONG(), Types.TUPLE([Types.LONG(), Types.DOUBLE()]))
        )

    def process_element(self, ev, ctx):
        # ev = (entity, event_ts_ms, amount)
        b = ev[1] // BUCKET_MS
        n, s = self.ring.get(b) or (0, 0.0)
        self.ring.put(b, (n + 1, s + ev[2]))
        horizon = b - BUCKETS + 1
        total_n, total_s = 0, 0.0
        for k in list(self.ring.keys()):
            if k < horizon:
                self.ring.remove(k)
            else:
                bn, bs = self.ring.get(k)
                total_n += bn
                total_s += bs
        yield (ev[0], b, total_n, total_s)   # sink: idempotent upsert keyed by (entity, b)

The output is keyed by entity and bucket so the sink can overwrite rather than append, which makes replays harmless. Note that it emits a value derived from the event’s bucket, not from wall-clock time. A production job would also set state time-to-live so entities that go quiet release memory, and would emit to a side output when an event’s bucket is older than the horizon.

Idempotency and ordering

Two guarantees keep the counters honest. Deduplicate on a stable event identifier before counting, because acquirers and gateways retry; the same logic that prevents double charges, described in payment API idempotency keys, applies to feature updates. And keep per-key ordering so that “seconds since last event” is computed against the true previous event. Partitioning by the card key preserves that ordering, but a feature keyed on device is then fed by a different partitioning; this is why multi-entity features usually need a repartition (a keyBy per entity) rather than one pass.

Point-in-Time Correctness, Label Delay and Skew

This is the section where most fraud systems that look fine in notebooks fail in production. A model trained on features computed with knowledge it would not have had at decision time will post excellent offline metrics and degrade live.

Figure 4: Training rows are built with an as-of join on event time. Labels arrive late, so immature rows must be excluded or handled explicitly. A replay check compares offline features with what the online store served.

As-of joins

A point-in-time (as-of) join attaches to each labelled transaction the feature values that were valid at that transaction’s timestamp, never later. Feast documents this behaviour as a core mechanism of historical feature retrieval (Feast docs); the same logic can be written in SQL with a window function or an ASOF join where your engine supports one. The subtle part is what “valid at” means. If the online path serves values up to 500 ms stale, the offline join should use a timestamp offset by the same freshness lag, or the model will learn from features that are fresher than anything it sees live.

The more reliable alternative to joining precomputed feature tables is to recompute features from raw events using the same code that runs online. If the stream job’s logic is expressed once, for instance in a shared library or as streaming SQL run in batch mode, the offline and online paths cannot diverge by implementation. When two implementations exist, test them against each other.

Label delay and maturity

A card-not-present chargeback can arrive weeks or months after the transaction; card networks set dispute time limits that are commonly measured in tens to a hundred or more days, depending on scheme and reason code (verify the figures for the scheme you operate in). Consequently the most recent weeks of training data are systematically under-labelled: transactions that will eventually be fraud are still marked clean. Training on them teaches the model that fraud is rarer than it is and that recent fraud patterns look benign.

Three mitigations are standard. Use a maturity cutoff and train only on transactions older than the label horizon you trust. Add earlier, weaker labels as separate targets or features: confirmed account takeover reports, customer disputes opened, manual-review outcomes and issuer fraud alerts arrive faster than chargebacks. And evaluate on mature windows only, with a clear note that recent-period performance is estimated, not measured. Selection effects also apply: declined transactions never get a fraud outcome, so a model trained only on approved traffic cannot see what it blocked. Teams address this with a small randomised exploration slice, a policy choice that needs risk and legal sign-off.

Detecting skew

Skew shows up in three ways: a feature’s distribution differs between training and serving, a feature is missing or defaulted more often online, or the same event yields different values on replay. Log the exact feature vector the decision service used for each scored event, keyed by event id. Then run a scheduled job that recomputes features offline for a sample and diffs them against the logged vector, reporting per-feature mismatch rates. Expect small differences for time-sensitive features; investigate any feature whose mismatch rate exceeds a threshold you set from baseline, and track null rates and distribution shift (for example population stability measures) per feature per day.

Latency Budgets in the Authorization Path

Fraud scoring shares a total authorization budget with routing, issuer processing and network hops, and the fraud slice is small. The right way to reason about it is to allocate a p99 budget per stage and then measure against it, rather than to assume the average. The numbers in this table are an illustrative allocation for a hypothetical 50 ms fraud slice, not a benchmark and not a standard; your own total will differ.

Stage	Illustrative p99 budget	Notes
Request parse and entity resolution	3 ms	Keys derived from the message, no I/O
Online store multi-get (card, device, account, merchant)	8 ms	One pipelined round trip, same availability zone
Request-time features (ratios, deltas)	2 ms	Pure computation on fetched values
Model inference	15 ms	Tree ensembles are typically cheaper than deep networks
Rules and policy	5 ms	In-process, no network
Logging the feature vector and decision	2 ms	Async, off the critical path
Headroom for jitter and GC	15 ms	Do not allocate this away

Three engineering moves protect the budget. Fetch everything in one batched call per decision; sequential lookups multiply network round trips. Co-locate the decision service and the store in the same zone, because cross-zone latency alone can consume a large share of a single-digit-millisecond allocation. And set a hard timeout on the feature read with a defined degraded mode: when a feature is missing, the model should receive an explicit missing indicator it was trained to handle, not a silent zero.

Tail latency usually comes from the long tail of keys. A hot merchant key, a very large HLL merge across many buckets, or a Lua script iterating a big hash will blow p99 while p50 looks healthy. Cap per-key state (bounded buckets, capped sets), avoid O(n) commands on large structures in the read path, and alert on p99 of the feature read separately from p99 of the whole decision so you can tell which layer regressed. Connection handling matters too; if the decision service autoscales, new instances open cold connection pools and warm slowly, an effect similar in spirit to the start-up costs discussed in container image lazy pulling and cold starts: keep a warm minimum of instances rather than scaling to zero on the scoring path.

Backfill, Entity Graphs and Rolling Out New Features

Backfilling a new feature

A new velocity feature is useless until it has history. The online store only knows what the stream job has seen since deployment, so a feature shipped on Monday has no seven-day window until next Monday. The standard approach is dual: run the stream job’s logic in batch mode over the event lake to compute historical values for training, and replay recent events through the live job (or bootstrap its state from a snapshot) so online values are warm at launch.

Backfill must honour event time and the same lateness policy. A common bug is computing historical aggregates over final, deduplicated, fully arrived data while the live job sees partial, duplicated, late data; the model then trains on cleaner features than it serves. Where possible, backfill from the same raw event log the stream job consumed, including its late arrivals, and apply the same dedup keys. Version every feature definition, and never change the window or bucket width of an existing feature in place: publish a new version, train on it, run it in shadow and cut over.

Entity graph features

Graph features answer “how connected is this entity to known bad ones?” A cheap and effective version needs no graph database: maintain bipartite counters such as distinct accounts per device and distinct devices per account, using HLLs, and expose their ratios and maxima as features. A richer version computes connected components or neighbourhood statistics in batch, say hourly, and publishes them as slowly changing features with an explicit staleness. Do not put multi-hop traversals in the authorization path; precompute, and read the result as another key lookup. The related detection patterns for scams that originate from authorised customers, where the signal is more about payee and session behaviour than card velocity, are in our post on authorized push payment fraud detection architecture, and instant-rail specifics are in instant payments fraud architecture across SEPA Instant, FedNow and RTP.

Rolling out safely

Run new features in shadow first: compute and log them without feeding the model, then compare their distributions and their offline lift. Gate promotion on three checks: no skew beyond threshold, no p99 regression in the feature read, and a measurable improvement on mature labels. Keep a kill switch per feature that substitutes the missing indicator, so a misbehaving counter can be removed in minutes without a model redeploy.

Trade-offs, Gotchas, and What Goes Wrong

Boundary blindness. Tumbling counters let bursts split across a boundary evade thresholds. Fix: bucketed rings or true sliding windows for any alerting feature.

Hot keys. A large merchant, a shared corporate IP or a payment gateway’s egress address concentrates writes on one key. That key’s state grows, its Lua script slows, and the shard becomes the bottleneck. Fix: cap and shard hot keys, treat shared infrastructure keys as lower-signal, and exclude known gateway addresses from IP velocity.

Cardinality attacks. An adversary who knows you track distinct values per entity can inflate state. HLL caps memory per key, but the number of keys is unbounded. Fix: TTLs on every key, key-count alerts and rate limits on key creation; the techniques overlap with distributed rate limiting with token buckets and GCRA.

Feedback loops. Declines change behaviour: a blocked fraudster rotates cards, which shifts velocity distributions, which retrains the model on a population shaped by its own policy. Fix: monitor drift, keep a holdout, and retrain on a schedule with mature labels.

Store failure semantics. If the online store fails over and loses recent writes, counters under-count and fraud passes at exactly the moment you are least able to see it. Decide whether the stream job can rebuild state by replaying a bounded window after failover, and whether the decision service should tighten rules while counters are known stale.

Clock and timezone bugs. Mixing event time, processing time and server clocks yields irreproducible features. Fix: one authoritative event timestamp in the schema, UTC throughout, and bucket ids derived only from it.

Privacy and retention. Behavioural aggregates keyed on personal identifiers are personal data in many jurisdictions. Fix: hash and salt keys, set retention to the shortest window the model needs, and document the lawful basis with your privacy team. This is an engineering note, not legal advice.

Over-engineering. Not every feature needs a stream job. If the model is a simple rule set over three counters, a Redis script and a nightly batch may be enough. Add the full topology when feature count, entity count or training-serving skew justify it.

Practical Recommendations

Start from the decision, not the technology. List the ten features your current model or rules rely on, classify each by state shape (count, sum, distinct, recency, graph), and choose the cheapest structure that meets its accuracy need. Exact rings for counts and sums, HLL plus a capped set for distinct counts adversaries can inflate, count-min only for the long tail.

Define windows on event time with bucketed rings, publish bucket width and lateness policy as part of each feature’s contract, and track the late-event rate. Separate the read path from the write path so the authorization never pays for aggregation, and give the read path a p99 budget, a timeout and an explicit missing-value behaviour.

Treat training-serving consistency as a test, not a hope. Log the served vector, replay a sample offline daily, and block releases on mismatch thresholds. Handle label delay with a maturity cutoff and earlier proxy labels, and be explicit about which evaluation windows are measured versus estimated.

[ ] Every feature has an owner, a state-shape classification and a version
[ ] Windows defined on event time; bucket width and lateness documented
[ ] Decision service does one batched, read-only feature fetch with a timeout
[ ] Stream sink is an idempotent upsert; events deduplicated by id
[ ] Served feature vector logged per decision; daily replay diff in place
[ ] Training uses as-of joins and a label maturity cutoff
[ ] New features run in shadow with a per-feature kill switch
[ ] Key TTLs, hot-key caps and key-count alerts configured
[ ] Retention and hashing reviewed with privacy and security teams

Frequently Asked Questions

What are velocity features in fraud detection?

Velocity features measure how often an entity does something within a time window: transactions per card in ten minutes, declined attempts per device in an hour, or the sum of amounts per account in a day. They are powerful because fraud, especially card testing and account takeover, tends to compress many actions into a short period. They need state, so they are computed by a stream processor or counter store, not from the request alone.

Should I use sliding windows or tumbling windows for fraud counters?

Use sliding behaviour, either true sliding windows or a bucketed ring that approximates them. Tumbling windows are cheap but blind at boundaries, so an attacker can split a burst across two intervals and stay under a threshold. A bucketed ring with six to twelve sub-buckets gives an error bounded by one bucket width at modest memory cost, which is usually the best balance for payments.

Is HyperLogLog accurate enough for distinct-count fraud features?

Generally yes. Redis documents a 0.81 percent standard error with up to 12 KB per key, and fraud thresholds are rarely sensitive to a one percent difference. The real limits are different: HLL cannot list members or delete them, and you must reproduce the same hashing offline. Keep a capped exact set where investigators need the actual values, and use HLL for the overflow count.

How do I avoid training-serving skew in fraud models?

Express feature logic once and run it in both paths, or test the two implementations against each other. Log the exact feature vector served for each decision, recompute a sample offline using point-in-time joins, and alert on per-feature mismatch and null rates. Match staleness too: if online values lag by a few hundred milliseconds, offline training should reproduce that lag.

How should label delay be handled when training fraud models?

Chargebacks can arrive weeks or months after the transaction, so recent data is under-labelled. Apply a maturity cutoff and train only on transactions old enough for labels to settle, supplement with faster proxy labels such as customer disputes, confirmed account takeovers and review outcomes, and report recent-period performance as an estimate. Also account for selection bias, since declined transactions never reveal their true outcome.

Do I need a managed feature store for real-time fraud features?

Not necessarily. A feature store adds a registry, materialisation and point-in-time retrieval, which pay off as feature and team counts grow. A small team with a handful of counters can run Redis plus a Flink job and a shared feature library. Whichever you choose, the invariants stay the same: event-time windows, an idempotent write path, a read-only serving path and a replayable offline source of truth.