Value-at-Risk and Expected Shortfall Engines: Historical Simulation, Monte Carlo and FRTB Architecture (2026)

A value at risk engine is deceptively simple on a whiteboard: take a portfolio, shock the market a few thousand ways, sort the profit-and-loss outcomes, and read off a quantile. In production it is one of the heaviest batch systems a bank runs. It must price millions of positions under thousands of scenarios, reproduce yesterday’s number to the cent when an auditor asks, and explain every dollar of movement between two business dates.

The pressure has grown because the metric itself changed. The Basel Committee’s Fundamental Review of the Trading Book (FRTB) moved the internal-models capital measure from 99% Value-at-Risk (VaR) to 97.5% Expected Shortfall (ES), and added desk-level approval tests, liquidity horizons and a separate treatment for risk factors that lack data. Each change is an engineering requirement, not just a regulatory one.

This post walks through the full design: the three classic VaR methods with a runnable worked example, why ES replaced VaR, the FRTB machinery around it, and the architecture of a market risk engine that can survive an audit. It is a systems analysis, not investment or regulatory advice.

What this covers: VaR and ES definitions, historical versus Monte Carlo methods, backtesting, FRTB’s engineering demands, scenario storage and aggregation, GPU acceleration, lineage, failure modes, and runnable Python.

Context and Background

VaR answers one question: what loss will not be exceeded over a given horizon with a given confidence? A one-day 99% VaR of 1.0 million means that, if the model is right, losses should exceed 1.0 million on about one trading day in a hundred. JPMorgan popularised the idea in the 1990s through RiskMetrics, and the 1996 Basel Market Risk Amendment let banks use internal VaR models for regulatory capital. The metric survived because it is a single number that can be compared across desks, products and asset classes.

It also has well-known limits. VaR says nothing about how bad losses are beyond the quantile, and it is not subadditive in general, so the VaR of a combined portfolio can exceed the sum of its parts. The 2007-2009 crisis exposed both problems in practice: models calibrated to calm periods understated tail losses, and the numbers could be gamed by positions that sat just beyond the cut-off.

Regulators answered with the Fundamental Review of the Trading Book. The Basel Committee published the revised market risk standard in January 2019 (document d457, with a corrected version in February 2019), and the Bank for International Settlements (BIS) describes its internal models approach as relying on expected shortfall models, with non-modellable risk factors receiving separate capital treatment. You can read the standard directly at the Basel Committee’s market risk framework page.

Implementation has been staggered. As of the sources I checked for this post, the European Union had postponed FRTB’s application to 1 January 2027 via a 2025 delegated act, with further adjustment under consideration; the United Kingdom had pushed Basel 3.1 to 1 January 2027 with market risk provisions anticipated from 1 January 2028; and the United States had not set an FRTB date, with revised Basel endgame proposals still pending. Treat these dates as a snapshot to re-verify against current regulator publications before you plan a programme around them.

For engineers, the practical consequence is that a market risk engine now serves two masters. The legacy VaR process still feeds limits, management reporting and, in many jurisdictions, current capital. The FRTB process demands ES, desk-level approval and richer data. Both must read the same positions and market data, and both must be reproducible. That is a data architecture problem before it is a mathematics problem, and it connects directly to BCBS 239 risk data aggregation and lineage, which sets the principles for how risk numbers are sourced and reported.

Definitions that the rest of the post depends on

Let the random profit and loss over the horizon be P, and the loss be L = -P. For a confidence level alpha, VaR is the alpha-quantile of the loss distribution: the smallest value v such that the probability of L being at most v is at least alpha. Expected Shortfall at the same level is the average loss in the tail beyond that quantile, the expectation of L given that L is at least VaR.

Two properties matter for engineering. First, both are computed from the same sorted vector of scenario losses, so an engine that can produce VaR can produce ES almost for free. Second, ES is a coherent risk measure: it is subadditive, which means diversification never makes the measure worse. That property is what makes it attractive for capital allocation, and it also makes aggregation behave sensibly when you roll up from desk to bank.

The Core Architecture of a Value at Risk Engine

A production market risk engine is a pipeline that turns market data and positions into a vector of scenario P&L values per portfolio node, then reduces that vector to risk measures. The four stages are scenario generation, revaluation, storage and aggregation. Every method discussed later, from historical simulation to Monte Carlo, differs only in how it produces the scenarios; the rest of the pipeline is shared.

Figure 1: Reference architecture of a value at risk engine. Market data snapshots feed a risk factor store, a scenario generator drives the pricing grid, and the resulting scenario P&L vectors are stored, aggregated and sent to reports, limits and backtesting.

The diagram shows the data flow left to right. Market data snapshots are frozen at a defined time and versioned. The risk factor store turns them into a consistent set of curves, surfaces and spot levels. The scenario generator produces shocked versions of those factors. The pricing grid revalues every trade under every scenario. Scenario P&L vectors land in a store that the aggregator reads to compute VaR, ES and contributions, while backtesting and P&L attribution read from the same lineage.

Stage 1: Market data snapshots and the risk factor store

The first discipline is that every run starts from an immutable, versioned snapshot. If a trader asks why yesterday’s VaR was 4% higher than the day before, the engine must be able to rebuild both runs from stored inputs. That means snapshotting prices, yield curves, volatility surfaces and FX rates with an identifier, a timestamp and the source of each point, and refusing to overwrite them.

The risk factor store sits on top. A risk factor is a market variable the model shocks: an equity price, an interest rate at a tenor, an implied volatility at a strike and expiry, a credit spread, an FX rate. A mid-sized bank can have tens of thousands of factors once curves and surfaces are decomposed into points. The store holds the full history of each factor, the shock convention (absolute or relative, log or arithmetic), and the mapping from products to the factors that drive them.

Shock conventions are a quiet source of bugs. Applying a relative shock to an interest rate that is near zero or negative produces nonsense, which is why rates and spreads are usually shocked in absolute terms while equities and FX use returns. Writing the convention into the store, not into pricing code, keeps every consumer consistent.

Stage 2: Scenario generation

A scenario is one coherent set of shocks to every risk factor. Historical simulation takes the observed daily changes from past dates and applies them to today’s levels. Monte Carlo draws shocks from a fitted statistical model. Stress scenarios use hand-built or regulatory shocks. All produce the same output type, a matrix with scenarios as rows and factors as columns, so the downstream pricing grid does not care which produced it.

Scenario sets should be treated as first-class data. Storing the scenario matrix, not only the final P&L, lets you re-run a modified portfolio without regenerating random numbers and makes Monte Carlo runs reproducible when you fix and record the random seed and generator version. A 500-day, 20,000-factor historical matrix is about 10 million values, small by modern standards; a 100,000-path Monte Carlo over the same factors is two billion, which needs deliberate engineering.

Stage 3: Revaluation, full versus sensitivities-based

Revaluation is where the cost lives. Full revaluation calls the actual pricing library for each trade under each scenario. It is exact for the scenarios used, including nonlinearity, barriers and path dependence, but expensive: one million trades under 500 scenarios is 500 million pricing calls per run.

The cheaper alternative is sensitivities-based (or Greeks-based) revaluation. The engine computes delta, gamma and vega once per trade and approximates scenario P&L with a Taylor expansion. This is fast and is the foundation of the FRTB Standardised Approach, but it fails for large moves, for options near barriers, and for products whose payoff has kinks. Most banks use a hybrid: full revaluation for non-linear books and sensitivities for vanilla linear ones, with a controlled reconciliation between the two.

Stage 4: Storage and aggregation

The aggregator needs the P&L vector for every node in the hierarchy: trade, book, desk, business, legal entity, bank. Because VaR and ES are not additive, you cannot sum node-level risk numbers. You can, however, sum node-level scenario P&L vectors, because P&L is additive scenario by scenario. That is the central trick: store scenario P&L per trade or per small bucket, roll up by vector addition, and compute the quantile at each level only after summing.

This design makes aggregation cheap and consistent. A bank-level number and a desk-level number come from the same underlying vectors, and a what-if analysis (add a hedge, remove a trade) is a vector add or subtract followed by a re-sort. The trade-off is storage: 500 scenarios times 8 bytes times 5 million trades is about 20 GB per run if kept at trade level, which is why many platforms store at an intermediate bucket granularity and retain trade-level vectors only for a rolling window.

Why the pipeline shape matters more than the quantile

It is tempting to focus on the statistics, but in real programmes the failures cluster in the pipeline. Late market data, a changed shock convention, a pricing library upgrade that shifts a model result, or a position feed that drops a booking system will each move the headline number far more than the choice between historical and Monte Carlo. The engine that wins audits is the one where every stage is versioned and every output carries a lineage identifier, a point developed in event sourcing and bitemporal data for financial audit trails.

VaR Methods Compared: Parametric, Historical, Filtered and Monte Carlo

There are four standard families of method. They share the downstream pipeline but differ in assumptions, cost and failure modes, so the choice is as much an engineering decision as a statistical one.

Figure 2: Method selection for a value at risk engine. Linear portfolios can use parametric VaR; nonlinear portfolios need historical, filtered historical or Monte Carlo full revaluation, depending on tail behaviour.

Parametric (delta-normal) VaR

Parametric VaR assumes factor returns are jointly normal and the portfolio is linear in the factors. Portfolio standard deviation is computed from the covariance matrix and the exposure vector, and VaR is the standard deviation multiplied by the normal quantile: about 2.326 at 99% and 1.960 at 97.5%. It needs no scenarios and runs in milliseconds, which is why it persists for intraday limit monitoring.

Its weaknesses are the ones its assumptions predict. Equity and credit returns have fat tails and skew, so a normal model understates extreme losses. Options and other nonlinear instruments violate the linearity assumption. A delta-gamma extension helps modestly, but at that point most teams move to simulation.

Historical simulation

Historical simulation (HS) applies the actual factor changes from the past N days, commonly 250 to 500, to today’s market state and revalues the portfolio under each. It makes no distributional assumption, captures fat tails and correlation changes that appeared in the window, and is easy to explain to a regulator: here are the real days, here is how today’s book would have fared.

Its limits are equally clear. The estimate can only reflect events inside the window, so a calm 500 days produces a low number just before a crisis. The 99th percentile of 500 observations is determined by the fifth worst loss, which is statistically noisy. And equal weighting means a shock that happened 400 days ago counts the same as one last week, so the measure responds slowly when volatility regimes change.

Filtered and weighted historical simulation

Two refinements address the regime problem. Age-weighted HS gives recent observations more weight with an exponential decay. Filtered historical simulation (FHS) fits a volatility model such as GARCH or EWMA to each factor, standardises historical returns by the volatility at the time, and then rescales them by today’s volatility estimate. The shape of the tail still comes from history, but its scale reflects current conditions.

FHS typically responds faster to volatility changes and reduces procyclical lag, at the cost of extra model risk: you now depend on a volatility model per factor and on the stability of its parameters. It also needs more compute and data engineering, because standardised residuals must be stored alongside raw returns.

Monte Carlo VaR

Monte Carlo VaR fits a model to the risk factors, usually a multivariate distribution with a covariance matrix estimated from history, draws many random scenarios from it, and revalues the portfolio under each. Because scenarios are synthetic, you can generate 100,000 or a million of them, which makes the tail estimate far smoother than a 500-day history gives.

The price is model dependence. If you draw from a multivariate normal, you inherit the thin tails of the normal even though you used a huge sample; more paths reduce sampling noise but not model error. Better engines use Student-t or copula-based dependence, regime mixtures, or bootstrap residuals, and they control variance using antithetic variates or quasi-random (Sobol) sequences. Monte Carlo is also the natural method for path-dependent and multi-horizon products.

A worked numeric example

To make the differences concrete, the code below builds a three-factor portfolio with Student-t distributed returns (4 degrees of freedom, so genuinely fat-tailed), then computes VaR and ES with historical simulation, a Monte Carlo that assumes normality, and a parametric formula. All numbers are produced by the script on synthetic data with a fixed seed; they illustrate relative behaviour, not any real portfolio.

import numpy as np

rng = np.random.default_rng(42)

def hist_var_es(pnl, alpha=0.99):
    losses = np.sort(-pnl)
    k = int(np.ceil(alpha * len(losses)))
    var = losses[k - 1]
    es = losses[k - 1:].mean()
    return var, es

# Synthetic history: 3 risk factors, fat tails, 500 days
T, nu = 500, 4
corr = np.array([[1.0, 0.6, 0.3], [0.6, 1.0, 0.4], [0.3, 0.4, 1.0]])
vol = np.array([0.010, 0.012, 0.008])
L = np.linalg.cholesky(corr)
z = rng.standard_t(nu, size=(T, 3)) / np.sqrt(nu / (nu - 2))
hist = (z @ L.T) * vol
notionals = np.array([40e6, -15e6, 25e6])

pnl_hist = hist @ notionals
print("HS 99%% VaR/ES: %.0f / %.0f" % hist_var_es(pnl_hist, 0.99))
print("HS 97.5%% VaR/ES: %.0f / %.0f" % hist_var_es(pnl_hist, 0.975))

cov = np.cov(hist.T)
sims = rng.multivariate_normal(np.zeros(3), cov, size=100_000)
pnl_mc = sims @ notionals
print("MC 99%% VaR/ES: %.0f / %.0f" % hist_var_es(pnl_mc, 0.99))
print("MC 97.5%% VaR/ES: %.0f / %.0f" % hist_var_es(pnl_mc, 0.975))

sigma = np.sqrt(notionals @ cov @ notionals)
print("Parametric 99%% VaR: %.0f" % (2.3263 * sigma))

Running it produced the following (one-day horizon, currency units):

Method	99% VaR	99% ES	97.5% VaR	97.5% ES
Historical simulation, 500 days	1,081,821	1,413,990	925,229	1,176,018
Monte Carlo, normal, 100,000 paths	923,308	1,062,987	776,308	930,057
Parametric delta-normal	917,939	n/a	n/a	n/a

The pattern is the instructive part. Monte Carlo and parametric agree with each other, as they should, because both assume normality; they differ from historical simulation by about 17% at 99% VaR, because the historical sample has fat tails the normal cannot produce. The gap widens in the tail: 99% ES under historical simulation is about 33% above the normal-based Monte Carlo figure. Part of that gap is sampling noise in a 500-point sample, and a different seed would move it, so do not read these numbers as a universal ratio. The direction, however, is robust: normal-based methods understate tail severity when the underlying data is fat-tailed.

It also shows why more Monte Carlo paths do not fix the problem. The 100,000-path run is smooth, but smooth around the wrong distribution. Model choice, not sample size, drives that error.

Expected Shortfall and the FRTB Machinery

The shift from VaR to ES is the central change in the Basel market risk framework. Understanding it requires separating three ideas that are often blurred together: the statistical choice (97.5% ES instead of 99% VaR), the scaling choice (liquidity horizons), and the control choice (desk-level approval and risk factor eligibility).

Why 97.5% ES replaced 99% VaR

For a normal distribution, 97.5% ES is close to 99% VaR; the two measures are calibrated to coincide, so the change was not intended to raise capital for well-behaved tails. The difference appears when the tail is heavy. ES averages over every loss beyond the quantile, so a book with a few catastrophic outcomes shows a higher ES than a book with the same VaR but milder tail losses. A strategy that sells deep out-of-the-money options and keeps VaR flat while stacking up tail risk is penalised by ES and invisible to VaR.

There is also a statistical argument. Estimating a 97.5% quantile from a finite sample is more stable than estimating a 99% one, because more observations fall beyond it. With 250 observations, 2.5% leaves about six tail points, versus about two or three at 99%. Averaging tail points in ES uses all of them rather than a single order statistic. The committee cited both the tail sensitivity and the better estimation properties in moving to ES.

ES has a known weakness of its own: it is not elicitable on its own, which complicates direct backtesting. Research by Gneiting and later by Fissler and Ziegel showed that VaR and ES are jointly elicitable, which gives a theoretical basis for comparing forecasts, and that is why FRTB backtesting continues to test VaR exceptions rather than ES values directly. The practical upshot is that ES is harder to validate than VaR, so supervisors compensate with other tests.

Liquidity horizons

A one-day loss is unrealistic for positions that cannot be sold in a day. FRTB assigns each risk factor a liquidity horizon, the time needed to exit or hedge without moving the market. The standard defines five horizon buckets of 10, 20, 30, 60 and 120 days, with categories ranging from large-cap equity price and major FX rates at the short end to some credit spreads, other-type volatilities and exotic risk factors at the long end. The exact mapping of each factor class to a bucket is set out in the standard, and I recommend reading the table directly in the Basel text before encoding it.

The engineering implication is large. The regulatory ES is not one number but a cascade: ES is computed with all risk factors shocked over a base 10-day horizon, then repeatedly with only the factors whose liquidity horizon is at least 20, 30, 60 and 120 days respectively, and the results are scaled by the square root of the horizon differences and combined. One run of ES becomes five partial runs over nested factor subsets, each of which must be stored and aggregated. A scenario store that tags every risk factor with its liquidity horizon makes this a filtering operation; one that does not makes it a redesign.

Stressed calibration and reduced factor sets

FRTB computes the internal-model ES on a reduced set of risk factors calibrated to the most stressful 12-month period in the history, then scales it to the full set by the ratio of full-set ES to reduced-set ES measured over the current window. The aim is to apply stress calibration without requiring a 20-year history for every factor, but it adds a second, parallel scenario set and a ratio calculation to the pipeline. Stress period selection must itself be documented, reproducible and re-evaluated, which is another auditable artefact.

Desk-level approval

Under the old approach, a bank either had model approval for a broad scope or did not. FRTB moves approval down to the trading desk. Each desk must pass tests, in particular a profit and loss attribution (PLA) test and backtesting. A desk that fails the tests falls back to the Standardised Approach, which is typically more capital intensive. The consequence is that the risk engine must produce desk-level results with desk-level quality metrics, not just bank-level totals.

Figure 3: FRTB desk-level approval. Each desk is tested using P&L attribution and backtesting; passing desks use the internal model ES with liquidity horizon scaling and a non-modellable risk factor add-on, while failing desks fall back to the Standardised Approach.

P&L attribution and risk-theoretical P&L

The PLA test compares two daily P&L series for a desk. Hypothetical P&L (HPL) is the front-office-style revaluation of the desk’s positions held static under the day’s market moves. Risk-theoretical P&L (RTPL) is the P&L the risk engine’s own model would produce for the same moves, using only risk factors eligible for the model. If RTPL does not track HPL closely, the risk model is not capturing what front office sees, and the desk fails.

The standard uses two statistical measures over a rolling window: a Spearman rank correlation and a Kolmogorov-Smirnov distance between the two distributions, with thresholds that place a desk in green, amber or red zones. The specific thresholds and window length should be taken from the Basel text, and I have not reproduced them here. What matters for design is the data dependency: the risk engine and front-office pricing must use identical trade data, market data and, as far as possible, pricing models, or the test will measure plumbing differences instead of model quality.

This is why many banks are converging front-office and risk pricing libraries. Running two separate valuation stacks was tolerable when the only check was a periodic reconciliation; under PLA it is a recurring cause of desk failures. Treat library convergence as a prerequisite, not an optimisation.

Backtesting under FRTB

Backtesting compares the model’s one-day VaR with actual and hypothetical P&L and counts exceptions, days on which the loss exceeded VaR. The standard tests both 99% and 97.5% one-day VaR over a 250-day window, using both actual and hypothetical P&L. As commonly described, a desk with more than 12 exceptions at 99%, or more than 30 at 97.5%, over 250 days fails; I am citing this from the widely documented framework description and recommend confirming the exact counts in the primary text.

This links to the older Basel traffic-light approach to VaR backtesting. Under that approach, a bank with zero to four exceptions in 250 days at 99% sits in the green zone, five to nine in the yellow zone with an increasing capital multiplier, and ten or more in the red zone. Statistical tests such as the Kupiec proportion-of-failures (POF) test ask whether the observed exception rate is consistent with the nominal one.

The Kupiec likelihood ratio compares the observed exception frequency x/n against the nominal p. For n = 250 and p = 1%, four exceptions give a statistic of about 0.77 and eight give about 7.73 (computed with the code below). The 95% critical value of a chi-square distribution with one degree of freedom is 3.841, so four exceptions are statistically consistent with a correct 99% model while eight reject it. That illustrates why backtesting has low power in small samples: you need a lot of days to distinguish a good model from a mediocre one.

import numpy as np

def kupiec_lr(n, x, p=0.01):
    ph = x / n
    ll = lambda q: (n - x) * np.log(1 - q) + (x * np.log(q) if x else 0.0)
    return -2 * (ll(p) - ll(ph))

for x in (0, 2, 4, 6, 8, 10):
    lr = kupiec_lr(250, x)
    print(x, round(lr, 3), "reject" if lr > 3.841 else "ok")

Non-modellable risk factors

A risk factor is modellable only if there is enough real price observation data to support its inclusion. The standard sets criteria based on the number of “real price” observations over the prior 12 months, such as a minimum number of observations per period and a minimum spacing, and I recommend checking the exact thresholds in the text. Factors that fail are classified as non-modellable (NMRF) and receive a separate stress-scenario capital charge, computed outside the ES.

NMRFs are a data engineering problem as much as a risk problem. To classify a factor you need an auditable record of every real price observation, with its source, timestamp and whether it is a committed quote or a trade. Banks now build observation stores that ingest trades from their own books, vendor feeds and pooled data such as consortium sources, and then run the eligibility test daily. In my experience, and this is an opinion rather than a sourced statistic, capital from non-modellable factors tends to concentrate in a few illiquid factors, so improving observation coverage for those few can matter disproportionately.

Implementation status as a design constraint

Because jurisdictions are on different timelines, a global bank often runs the current regime and FRTB-style calculations in parallel. For design purposes, assume you will be running both for several years. That argues for a single scenario store and pricing grid with two aggregation and reporting layers on top, rather than two independent engines. The shared infrastructure is the expensive part; the aggregation logic is relatively cheap to duplicate.

Building the Engine: Compute, Storage and Lineage

With the methods and the regulatory demands clear, the next question is how to build something that runs overnight, reruns on demand, and explains itself. The dominant costs are pricing compute, scenario storage and the engineering effort of reproducibility.

Figure 4: Daily batch sequence in a value at risk engine. A scheduler freezes the market data snapshot, dispatches trades to pricing shards, the aggregator sorts the tail to compute ES, and the report carries lineage ids back to the scheduler.

The pricing grid

The pricing grid is a distributed compute layer that revalues trades under scenarios. Work is sharded by trade (or trade group sharing the same market data) and scenario block. The key optimisation is to cache what does not change across scenarios: trade setup, curve construction for unshocked factors, and calibration results where valid. Without caching, the same swap schedule is regenerated thousands of times per run.

Throughput planning starts from simple arithmetic. If a vanilla swap prices in 20 microseconds and an exotic in 5 milliseconds (illustrative figures, not benchmarks), a book of one million swaps and ten thousand exotics under 500 scenarios needs about 10,000 core-seconds for the swaps (1e6 x 500 x 20e-6) and 25,000 core-seconds for the exotics (1e4 x 500 x 5e-3). Fewer than 1% of trades account for most of the compute, so the optimisation effort belongs on the exotics.

Grid failure handling matters as much as speed. A run touching hundreds of millions of pricing calls will see failed tasks, so shards must be idempotent and retryable, and the system should distinguish pricing failures (which should be reported and replaced with a defined fallback, not silently zeroed) from infrastructure failures (which should be retried).

GPU and vectorised Monte Carlo

Monte Carlo is embarrassingly parallel across paths, which maps well to GPUs. Random number generation, correlation through a Cholesky factor, and the payoff evaluation for simple instruments are all dense, regular operations. NVIDIA’s cuRAND library offers generators such as Philox and Sobol for this purpose, and libraries such as CuPy or JAX let teams express the same vectorised code on CPU and GPU. Reported speed-ups for regular Monte Carlo workloads are large, but they depend heavily on instrument complexity and data movement, so benchmark on your own book rather than quoting a vendor ratio.

Even without GPUs, vectorisation helps. Replacing a Python loop over scenarios with a matrix operation over all scenarios at once routinely cuts time by an order of magnitude. The code below sketches a vectorised full revaluation for a European call, using Black-Scholes as a stand-in for a real pricer, and shows the structure: scenario arrays in, P&L array out.

import numpy as np
from math import log, sqrt
from scipy.stats import norm

def bs_call(S, K, r, vol, T):
    d1 = (np.log(S / K) + (r + 0.5 * vol**2) * T) / (vol * np.sqrt(T))
    d2 = d1 - vol * np.sqrt(T)
    return S * norm.cdf(d1) - K * np.exp(-r * T) * norm.cdf(d2)

rng = np.random.default_rng(7)
S0, K, r, vol, T = 100.0, 105.0, 0.03, 0.25, 0.5
n_scen = 100_000
dS = rng.standard_t(4, n_scen) / np.sqrt(2) * 0.012  # 1-day return, fat tails
dvol = rng.normal(0, 0.01, n_scen)                   # vol shock in absolute terms

base = bs_call(S0, K, r, vol, T)
shocked = bs_call(S0 * np.exp(dS), K, r, np.maximum(vol + dvol, 0.01), T - 1/252)
pnl = (shocked - base) * 10_000                      # 10,000 options long

losses = np.sort(-pnl)
k = int(np.ceil(0.975 * n_scen))
print("97.5%% VaR %.0f  ES %.0f" % (losses[k-1], losses[k-1:].mean()))

The structure matters more than the numbers: scenarios are arrays, pricing is a vectorised function, and the P&L vector is the engine’s output. Swapping NumPy for CuPy often requires changing only the import for code written this way. Non-vectorisable pricers, such as those with Monte Carlo inside Monte Carlo or iterative calibration, are the ones that need real grid engineering.

Variance reduction and convergence

More paths reduce sampling error at a rate proportional to one over the square root of N, so a tenfold accuracy gain needs a hundredfold more paths. Variance reduction is therefore worth the complexity. Antithetic variates pair each random draw with its negation and cut noise for roughly symmetric payoffs. Sobol low-discrepancy sequences fill the space more evenly and converge faster in moderate dimensions. For tail estimation specifically, importance sampling shifts the distribution toward the tail and reweights, at the cost of more delicate implementation.

Tail quantities converge more slowly than means. Estimating ES at 97.5% from 100,000 paths uses 2,500 tail points, which is enough for stable results on one portfolio but can be noisy at small desk level. A practical control is to run the same portfolio with two independent seeds and treat the difference as a built-in estimate of Monte Carlo noise; if it exceeds a tolerance, increase paths for that node.

Scenario storage and aggregation at scale

Store scenario P&L vectors in a columnar format (Parquet or an in-memory columnar store) partitioned by business date and portfolio node. The aggregation query is then a sum over rows grouped by hierarchy level, followed by a sort, which columnar engines handle well. Keep the scenario identifiers aligned across all portfolios: scenario 17 must mean the same date or random path for every trade, or vector addition becomes meaningless.

Because ES at the root requires the sorted full vector, the aggregator holds one vector per hierarchy node. For a hierarchy of 50,000 nodes and 10,000 scenarios at 8 bytes, that is about 4 GB, which fits in memory on one machine; the heavier cost is reading trade-level data. Pre-aggregating to the smallest meaningful bucket at pricing time, rather than writing trade-level vectors, cuts storage by orders of magnitude, at the cost of limiting later drill-down.

Incremental and component VaR

Risk managers need not just a total but where it comes from. Incremental VaR measures how the total changes when a position is added or removed; it is computed by recomputing the quantile on the modified vector. Component (or marginal) VaR allocates the total across positions so that components sum to the total, under smooth assumptions via the Euler allocation: each position’s contribution is its exposure times the partial derivative of the risk measure with respect to that exposure.

For ES, Euler allocation has a clean empirical form. The contribution of a position is the average of that position’s scenario losses over the scenarios that make up the tail. Because it is just a conditional mean over the already-sorted tail set, it comes almost free from the same vector the aggregator already holds. By contrast, contributions to a historical-simulation VaR, which depends on a single scenario, are unstable: tiny changes in the portfolio can change which scenario sits at the quantile and shuffle the contributions. This is a practical reason risk teams prefer ES contributions for reporting.

Data lineage and P&L explain

Every number should be traceable. A lineage record for a run includes the market data snapshot id, the position snapshot id, the model and pricing library versions, the scenario set id and random seed, the parameter set (window, confidence, horizon), and the code version of the aggregation. Store these with the output as immutable metadata, so any figure in a report can be reproduced.

P&L explain is the day-to-day application of lineage. When VaR moves, the risk team decomposes the change into the effect of new trades, the effect of market data changes, the effect of the window rolling forward (an old extreme day dropping out of the sample), and model or methodology changes. Building the engine so it can rerun with one input changed at a time makes this decomposition mechanical, not forensic. The expectations in BCBS 239 on accuracy, completeness and timeliness of risk data aggregation make this more than convenience, as covered in the BCBS 239 architecture guide.

Where the risk engine meets the rest of the trading stack

The overnight VaR engine sits downstream of trading and upstream of limits, but it is only one of three latency tiers. Pre-trade checks must complete in microseconds and typically use simple exposure and limit logic, as described in the pre-trade risk engine architecture post. Intraday risk uses sensitivities and parametric or partial-revaluation methods updated every few minutes. The end-of-day engine described here runs the full scenario process and is the one that feeds capital and official reporting.

Model development has a related dependency. The same historical data and replay infrastructure that powers a backtesting engine for strategies, as outlined in the event-driven backtesting engine architecture, can be reused for risk model validation. Sharing the data layer avoids two parallel versions of history that disagree.

Trade-offs, Gotchas, and What Goes Wrong

Every design choice above has a failure mode, and the most damaging ones are often quiet. The following are the recurring problems in practice.

Procyclicality

Models calibrated on recent data produce low risk numbers in calm markets and high ones in turbulence, so capital requirements tighten exactly when markets are stressed. Equal-weighted historical simulation lags changes in both directions: after a spike it takes the full window length to forget the event. Stressed calibration in FRTB partly addresses this by anchoring part of the measure to a stress period, but the base ES still moves with the market. Engines that expose the effect (reporting VaR both on the current window and on the stress window) make the cyclical component visible to management.

Stale or poor-quality data

A single stale price, a mis-scaled quote or a missing curve point can produce a spurious extreme scenario that dominates the tail. In a 500-day window, one bad data point becomes one of the worst five scenarios and directly moves the 99% VaR. Data quality controls (outlier detection with human review, completeness checks, cross-source comparison) should run before scenarios are generated, with a clear policy for how filled or proxied values are flagged. Proxy risk factors, used where no direct history exists, introduce basis risk the engine does not capture unless modelled explicitly.

Model risk and the illusion of precision

A VaR number reported to seven digits implies an accuracy it does not have. Sampling error on a 99% quantile from 500 points is large, and model choice (normal versus fat-tailed, window length, volatility scaling) can shift results by tens of percent, as the worked example showed. Mature programmes run challenger models, report confidence intervals or bootstrap ranges alongside point estimates, and treat large divergences between methods as a signal for investigation rather than noise.

Backtesting limits and gaming

Backtesting has low statistical power, so a model can pass while being wrong. It can also be sensitive to the P&L definition: actual P&L includes fees, intraday trading and reserves, which contaminate the comparison, and hypothetical P&L requires a clean revaluation of static positions. Desks with a high proportion of intraday activity will show differences between the two series that have nothing to do with the model.

PLA failures from plumbing, not models

Desks fail P&L attribution frequently for mundane reasons: different market data timestamps in the front-office and risk systems, different holiday calendars, a missing risk factor mapping, or a risk model that omits a factor front office prices. These are fixable engineering defects, but only if the engine exposes diagnostics that decompose the RTPL-HPL gap by factor and by trade. Without that, teams spend months guessing.

Liquidity-horizon cascade cost

The five-way ES cascade multiplies the aggregation workload and, if the pricing grid is not designed to emit factor-bucketed P&L vectors, the pricing workload as well. The efficient approach is to price once per scenario on the full factor set while tracking P&L attribution to factor groups, but that requires pricing code that supports it, which legacy libraries often do not.

Non-determinism

Parallel floating-point summation, library upgrades and unpinned random generators make reruns differ in the last digits, which breaks reconciliation. Fix seeds, pin library versions, use deterministic reduction orders where possible, and define a tolerance for acceptable rerun differences. Silent differences are worse than loud failures.

Practical Recommendations

Start by deciding what the engine must answer before choosing a method. If the goal is regulatory capital, the method is largely dictated by the rule, and your effort goes into data, desk-level controls and reproducibility. If the goal is management reporting, a filtered historical simulation backed by a parametric intraday cross-check is a defensible default for most books. Use Monte Carlo where products are path dependent or where history is too short to support a tail estimate.

Treat the scenario P&L vector as the platform’s central data product. Build around it: store it, version it, make it queryable, and compute everything else (VaR, ES, contributions, what-ifs, PLA inputs) from it. This one decision removes most of the duplication that makes risk systems expensive to change.

Invest early in the unglamorous parts. Snapshot and version market data. Tag every risk factor with its liquidity horizon, modellability status and shock convention. Align front-office and risk valuation before the PLA test forces the issue. Record lineage by default. And since regulatory dates keep moving, design for running two regimes at once.

A short checklist for a build or review:

Immutable, versioned market data and position snapshots for every run.
Scenario sets stored with seed, generator version and window definition.
Scenario P&L stored at additive granularity so any node can be aggregated by vector sum.
Liquidity horizon and eligibility metadata on every risk factor.
Reduced and full factor-set ES, plus stress-period ES, produced from the same grid.
Desk-level backtesting and PLA dashboards with factor-level diagnostics.
Daily data quality gates before scenario generation.
Method-comparison reports (HS versus MC versus parametric) as a model risk control.
Rerun determinism tests and lineage on every published figure.

This post is a technical systems analysis for educational purposes. It is not investment, risk management or regulatory advice.

Frequently Asked Questions

What is a value at risk engine?

A value at risk engine is the software system that calculates VaR, and usually Expected Shortfall, for a portfolio. It ingests market data and positions, generates scenarios by historical simulation, Monte Carlo or parametric methods, revalues the portfolio under each scenario, and aggregates the resulting profit and loss into risk measures at every level of the hierarchy. Production engines also handle backtesting, lineage, attribution and regulatory reporting, so most of the engineering effort is in data and reproducibility, not the quantile calculation.

Why did FRTB replace VaR with Expected Shortfall?

VaR ignores the size of losses beyond its cut-off and is not subadditive, so it can discourage diversification and reward tail risk that sits just past the quantile. Expected Shortfall averages the tail losses, is coherent, and is more sensitive to heavy tails. The Basel Committee chose 97.5% ES to be roughly equivalent to 99% VaR for normal distributions while capturing tail severity better for non-normal ones, and estimating a 97.5% quantile from finite data is also somewhat more stable.

Is historical simulation or Monte Carlo better for VaR?

Neither dominates. Historical simulation makes no distributional assumption and is easy to explain, but it is limited to events in the window and is noisy at the 99% level. Monte Carlo allows far more scenarios and handles path-dependent products, but it is only as good as the model that generates the paths. In the worked example above, a normal-based Monte Carlo understated 99% ES versus fat-tailed history by about a quarter, despite 100,000 paths. Most banks use a mix.

How is Expected Shortfall backtested?

Directly backtesting ES is harder than backtesting VaR because ES alone is not elicitable. FRTB therefore backtests one-day VaR at 99% and 97.5% against actual and hypothetical P&L over a 250-day window, counts exceptions, and uses thresholds on those counts to decide whether a desk may keep using the internal model. Supplementary tests such as the Kupiec proportion-of-failures test and the older Basel traffic-light zones are used for validation, and research tests for ES itself exist but are not the regulatory standard.

What are liquidity horizons and non-modellable risk factors in FRTB?

Liquidity horizons recognise that positions cannot be exited in one day. Each risk factor is assigned a horizon bucket of 10, 20, 30, 60 or 120 days, and ES is computed in nested stages and scaled accordingly. Non-modellable risk factors are those lacking enough verifiable real price observations to support modelling; they are excluded from ES and charged separately using stress scenarios. Both require extra metadata on each risk factor and a store of price observations.

Do I need GPUs for a market risk engine?

Not necessarily. GPUs pay off for Monte Carlo with regular payoffs and large path counts, where random number generation and arithmetic dominate. For historical simulation with full revaluation of complex trades, the bottleneck is usually the pricing library and data movement, so CPU grids with good caching can be more cost effective. Vectorise first, profile on your actual book, and move to GPUs only where benchmarks on your workload justify the added operational complexity.