Self-Driving Lab Data Provenance and Reproducibility: The Replayability Problem

Running one autonomous experiment is a solved engineering problem. Trusting a six-month closed-loop campaign well enough to replay it, audit it, or defend it to a reviewer is not. Self-driving lab data provenance is the discipline that closes that gap: capturing enough of the who, what, when, and which-version at every hop so a whole autonomous discovery run can be re-derived from evidence rather than remembered from lore. In a self-driving lab an AI plans, robots synthesize, instruments characterize, a model learns, and the loop repeats — thousands of times, unattended. Each turn silently mutates the state that produced the next decision: model weights, calibration offsets, reagent lots, scheduler timing. Miss the capture and the campaign becomes a black box that produced numbers nobody can stand behind.

This is a systems-architecture analysis of how provenance and reproducibility are engineered in autonomous labs, not lab-bench how-to and not scientific or investment advice.

What this covers: why provenance is uniquely hard in self-driving labs, the provenance DAG and its capture points, PROV-O and FAIR mapping, ELN/LIMS/SiLA/AnIML integration, versioning and content-addressed logs, the replayability layer, and the failure modes that quietly destroy reproducibility.

Context and Background

A self-driving lab (SDL) differs from ordinary lab automation in one decisive way: the machine, not a human, chooses the next experiment. That choice is a function of everything the system has observed so far, so the provenance burden is not linear — it compounds. Reproducing experiment number 900 requires reconstructing the exact model state that experiments 1 through 899 produced, plus the physical drift the hardware accumulated along the way. A pipetting robot loses microliters as tips wear; a spectrometer’s baseline creeps between recalibrations; a reagent bottle is swapped mid-run for a new lot with a subtly different impurity profile. None of these are logged by default, and each one changes results.

It helps to be precise about what “reproducible” even means, because the word is overloaded. The widely used distinction separates three levels. Repeat: the same team re-runs the same setup and gets the same result — a test of precision. Replicate: a different team runs an independent implementation of the same protocol and reaches a compatible result — a test of robustness. Reproduce: the original data and code are re-analyzed to confirm the reported findings without collecting new data — a test of the analysis itself. SDLs make all three harder because the “setup” is a moving target and the “protocol” was authored by an optimizer, not written down by a person.

This is why self-driving lab data provenance is a systems problem rather than a documentation problem. A human-run study can lean on a lab notebook and a diligent postdoc’s memory to bridge gaps; an autonomous campaign running unattended overnight has no such fallback. If the who-what-when-which-version was not captured at the instant an event occurred, it is gone. The autonomy that makes SDLs fast is the same autonomy that removes the human observer who used to be the provenance system of last resort, so the machinery has to record its own history with a completeness no manual process ever demanded.

The stakes are not hypothetical. Berkeley’s A-Lab reported synthesizing 41 novel inorganic compounds from 58 targets in 17 days of autonomous operation (Nature, 2023). Within weeks, independent crystallographers publicly questioned how many of those phases were correctly identified, arguing that some assignments were ambiguous from the raw diffraction data alone. Whatever the merits on either side, the episode is the whole argument for provenance in miniature: the fastest way to resolve a dispute about an autonomous result is to replay the exact decision and reduction chain that produced it. If you designed for replay, that is a query; if you did not, it is an archaeology project. For how the loop itself is wired, see our self-driving lab closed-loop architecture breakdown.

The Provenance Architecture

A self-driving lab data provenance system is, at its core, a directed acyclic graph (DAG) of typed events that records, for every artifact, which activity produced it, which agent ran that activity, and which prior artifacts it consumed. The unit of capture is the edge, not the file: each transition from hypothesis to plan to protocol to robot action to raw data to reduced result to model update is stamped with a who, a what, a when, and a which-version. Replayability is the property that falls out when that graph is complete and its inputs are pinned.

Figure 1: The experiment lineage DAG — hypothesis, plan, protocol version, robot actions, raw instrument data, reduced results, and model update, closing the loop back to the next hypothesis.

Long description: A vertical flow shows seven nodes connected top to bottom — Hypothesis (campaign goal), Plan (model version), Protocol (version hash), Robot actions (execution log), Raw instrument data, Reduced results, and Model update (new weights). A final edge returns from Model update back to Hypothesis, forming the closed learning loop where each cycle’s model state seeds the next decision.

The mistake most teams make is treating provenance as logging. Logging is append-only text a human reads after something breaks; provenance is a queryable graph a machine traverses to answer “what produced this?” and “what did this affect?” The difference shows up the moment you ask a lineage question — a downstream query like “every result that depended on calibration file C-2291” is trivial on a graph and nearly impossible on a pile of log lines. The design target is that any node in the DAG can be expanded backward to its complete causal history and forward to everything it influenced.

Granularity is the first real design decision. Capture too coarsely — one event per experiment — and you lose the ability to attribute a bad result to a specific pipetting step or a mid-run reagent swap. Capture too finely — every servo movement — and the graph becomes a storage and query burden with little added insight. The workable middle is to make each semantically meaningful transition a node: a plan decision, a compiled protocol, a discrete robot task, a raw acquisition, a reduction, a model update. That granularity is coarse enough to query cheaply and fine enough that any dispute about a result lands on a specific edge you can inspect.

A second decision is whether provenance is recorded prospectively or retrospectively. Retrospective provenance — reconstructed after the fact by parsing outputs — is always incomplete because it can only see what happened to be written down. Prospective provenance, where the plan itself declares the intended lineage and the executing systems confirm or amend it as they run, is far stronger: you know not only what happened but what was supposed to happen, which is exactly the comparison you need when a run goes sideways. Mature self-driving lab data provenance combines both, treating the planned DAG as the expectation and the executed DAG as the evidence.

Capture points and what to record at each

Every edge in the lineage graph is a capture point, and the cost of a missing field is paid months later when nobody can reconstruct it. At the plan edge, record the planner’s model identifier and weights hash, the acquisition function, the candidate set it scored, and the random seed. At the protocol edge, record a content hash of the exact protocol version the plan compiled to, not a mutable pointer to “the current protocol.” At the robot-action edge, record the executed action log with real timestamps and the firmware and deck-configuration versions of the cell. At the instrument edge, record the raw data, the instrument serial, and the active calibration state. At the reduction edge, record the analysis code commit and parameters. At the model-update edge, record the training data snapshot and the resulting weights hash.

The rule of thumb is that anything a human would need to ask a colleague about — which lot, which firmware, which seed, which threshold — must instead be an attribute on an edge. Provenance done well means no such colleague is ever required, because the graph already answers.

The seed and the acquisition function deserve special emphasis because they are what make an autonomous decision reconstructable. When the planner scores a candidate set and picks the next experiment, that choice is deterministic given the model weights, the observed data, the acquisition function, and the random seed. Record all four and you can re-run the decision and get the identical experiment; drop the seed and you have a plausible story about why the machine chose what it did, but not a reproduction of it. In a self-driving lab data provenance model, the decision edge is the most information-dense edge in the graph and the one most often under-captured.

PROV-O as the interoperability spine

You do not have to invent this data model; the W3C PROV-O ontology already provides the vocabulary, and adopting it makes your provenance portable across tools instead of trapped in one vendor’s schema. PROV-O has three core classes: Entity (a thing with provenance — a raw spectrum, a reduced result, a model checkpoint), Activity (something that acts on entities over time — a synthesis run, a reduction job), and Agent (who or what bears responsibility — the planner, the robot cell, the operator). These map cleanly onto SDL artifacts.

The relations do the real work. prov:wasGeneratedBy links an entity to the activity that produced it; prov:used links an activity to the entities it consumed; prov:wasDerivedFrom chains entities directly (reduced result derived from raw spectrum); prov:wasAssociatedWith attributes an activity to an agent; and prov:wasInformedBy links activities that communicated. A synthesis run wasAssociatedWith a robot cell, used a protocol version and a reagent lot, and generated a raw dataset that a reduction activity then used to generate a result. Because these terms are a shared standard, a provenance store built on them can be queried, merged, and validated by off-the-shelf tooling rather than bespoke code.

The event bus and content-addressed store

Capture has to happen at the moment of the event, from the system that owns the truth, or it degrades into reconstruction. The robust pattern is an append-only provenance event bus that every component emits to — planner, orchestrator, robot cell, instruments, ELN, and LIMS — with each event written to a content-addressed store (CAS) where the address is the hash of the content itself.

Figure 2: The provenance capture layer — planner, ELN, orchestrator, robot cell, instruments, and LIMS all emit typed events to a shared provenance event bus that persists to an immutable content-addressed store.

Long description: On the left, six sources — Planner (AI agent), ELN (hypothesis notes), Orchestrator (scheduler), Robot cell (action log), Instruments (SiLA 2), and LIMS (samples and lots) — each feed arrows into a central Provenance event bus. The bus feeds a single arrow into a CAS store described as an immutable log, showing that all provenance from heterogeneous systems converges into one append-only, hash-addressed record.

Content addressing gives you two properties provenance cannot live without. First, immutability: because the address is derived from the bytes, you cannot silently alter a record without changing its address, which makes tampering and accidental overwrites detectable. Second, natural deduplication and integrity: identical protocols or datasets resolve to the same hash, and a later fetch either returns bit-identical content or fails loudly. The orchestration layer that emits these events is worth designing deliberately — our note on SiLA 2 lab-as-code orchestration covers how instrument control and event emission fit together.

An immutable event log also gives self-driving lab data provenance an audit posture that a mutable database cannot. Because events are append-only and hash-linked, the record functions like a ledger: you can prove a result existed at a point in time, demonstrate that no upstream input was altered after the fact, and reconstruct exactly what the system knew when it made a given decision. That property matters well beyond academic reproducibility — it is what a regulated lab, a patent dispute, or an internal safety review needs, and it is nearly impossible to retrofit onto a system that overwrote its own history as it went.

FAIR Data, Standards, and the Replayability Layer

Provenance answers “how was this produced?”; FAIR answers “can anyone, including future-you, actually find and reuse it?” The FAIR principles — Findable, Accessible, Interoperable, Reusable — were formalized by Wilkinson et al. in 2016 and are the reuse contract every SDL artifact should satisfy. Provenance without FAIR is a perfect record nobody can locate; FAIR without provenance is a findable file you cannot trust. A complete self-driving lab data provenance strategy therefore treats the two as one system: the lineage graph is the content, and FAIR is how that content is addressed, retrieved, and licensed for reuse.

The mapping onto SDL practice is concrete. Findable means every entity carries a persistent identifier (PID) — a DOI or handle — and rich, indexed metadata. Accessible means the artifact is retrievable by that identifier over an authenticated, documented protocol, and that metadata survives even if the raw bytes are later restricted. Interoperable means data and metadata use shared vocabularies — PROV-O for lineage, domain ontologies for materials and instruments. Reusable means a clear license plus the provenance and protocol context needed to legitimately reuse the result. FAIR is the addressing and trust layer that sits on top of the lineage graph.

Figure 3: The FAIR-plus-versioning stack — persistent identifiers and FAIR metadata over content-hashed versioning of code, protocol, model weights, and physical material lots.

Long description: The upper block shows a FAIR layer (PID and metadata) branching into its four principles — Findable (DOI or handle), Accessible (API with auth), Interoperable (PROV-O and AnIML), and Reusable (license and lineage). The lower block shows a separate Version layer branching into four versioned artifact types — Code (git commit), Protocol (content hash), Model (weights hash), and Material (lot id) — establishing that reproducibility requires versioning physical inputs, not only digital ones.

Standards turn these principles into wire formats. On the instrument side, SiLA 2 standardizes device control and data interfaces so a spectrometer or liquid handler exposes a discoverable, typed API instead of a vendor blob. For analytical data, AnIML (Analytical Information Markup Language) and the Allotrope Foundation’s data models give raw and processed instrument results a self-describing, metadata-rich container. Upstream, the ELN holds hypotheses and human context and the LIMS tracks samples, lots, and locations; wiring the planner and orchestrator into both is what keeps digital decisions and physical inventory on the same provenance graph.

ELN/LIMS integration is where most self-driving lab data provenance efforts either succeed or quietly rot. The failure pattern is a lab that has an excellent LIMS and an excellent orchestrator that never talk: the LIMS knows which lot of precursor is in bottle 14, and the orchestrator knows which experiment used bottle 14, but no event ever joins those two facts. When the join is missing, a supplier changing an impurity profile looks like a genuine discovery. The integration that matters is not a nightly export; it is a live reference — the moment a robot draws from bottle 14, the emitted provenance event carries the LIMS lot ID by reference, so the material’s full history is one hop away from the result forever.

A related discipline is that instrument metadata must travel with the data, not beside it. A raw diffraction file stored next to a separate spreadsheet of acquisition settings is one careless move away from an orphan. Standards like AnIML and Allotrope exist precisely so the settings, units, calibration reference, and instrument identity live inside the same self-describing container as the measurement, which is why adopting them is a provenance decision as much as a data-format one.

The subtle, expensive part is versioning everything a result depends on — and physical inputs are the ones teams forget. Reproducibility fails if any dependency drifts unrecorded.

Artifact	Versioning mechanism	Failure if unversioned
Analysis and control code	Git commit hash, pinned deps	Silent reduction changes alter results
Protocol	Content hash of exact version	“Same” protocol quietly edited between runs
Model	Weights hash + training snapshot	Cannot reconstruct the decision that was made
Environment	Container image digest	Library update changes numeric output
Physical material	Lot / batch ID in LIMS	Impurity or supplier change misread as signal
Instrument state	Calibration file ID + timestamp	Drift attributed to chemistry, not hardware

With every dependency pinned, replay becomes mechanical rather than heroic. The replayability layer takes a campaign or single-result identifier, resolves its subgraph from the immutable event log, pins each referenced version, rebuilds the computational environment from its container digest, checks that the referenced calibration state and material lots are available, re-derives the reduction, and diffs the output against the original. Digital steps should reproduce bit-for-bit; physical steps reproduce within documented tolerance, and the provenance tells you which is which.

It is worth being honest about the limit of replay. The computational half of a campaign — the planning decisions and the data reduction — can be reproduced exactly, because it is deterministic given pinned inputs. The physical half cannot: you cannot rewind a consumed reagent or restore an instrument to a calibration state it has since drifted past. What the provenance record buys you is the ability to distinguish these two failure modes cleanly. When a replayed reduction matches, you have proven the analysis; when it matches but a fresh physical run does not, you have localized the discrepancy to the hardware or materials rather than the math. That separation is the single most valuable thing a self-driving lab data provenance system produces, because it turns “the result did not reproduce” from an unfalsifiable worry into a specific, addressable question.

Figure 4: The replay flow — from a campaign query, resolve the DAG, pin versions, rebuild the environment, check calibration and lots, re-derive the reduction, and diff against the original result.

Long description: A left-to-right sequence of six steps — Replay query (campaign id), Resolve DAG (from event log), Pin versions (code, model, protocol), Rebuild env (container image), Check calibration and lots, Re-derive reduction, and Compare to original — showing the deterministic path by which a stored provenance graph is turned back into a re-executed, verifiable result.

Trade-offs, Gotchas, and What Goes Wrong

Most reproducibility failures in self-driving labs are not dramatic; they are quiet omissions that surface only when someone tries to replay a campaign and cannot. Knowing the common failure modes is how you design the capture to prevent them.

Silent calibration drift is the classic. An instrument’s baseline creeps between recalibrations, and because the calibration state was never bound to each measurement, a real hardware artifact gets learned by the model as chemistry. The fix is to stamp every raw dataset with the active calibration file ID and timestamp, and to emit a provenance event on every recalibration so drift is visible as a step change in the record.

Unversioned protocol edits are the second. An operator tweaks a temperature ramp or a mixing time and saves over the existing protocol; every prior run now points at a definition that no longer describes what happened. Only a content hash of the exact executed version — captured at execution time, never a mutable “current protocol” pointer — prevents this, which is why versioned experiment protocols are non-negotiable.

Lost negative results are the most under-appreciated. Failed, aborted, and out-of-spec runs carry information the optimizer used to make its next choice, so discarding them breaks lineage and biases any later meta-analysis. Aborted runs must be first-class provenance entities with a recorded reason, not deleted rows.

Metadata rot and orphaned raw data close the list. Raw files land in object storage with no back-link to the run that produced them, PIDs go unminted, and free-text fields decay until the dataset is findable by luck only. And a non-deterministic scheduler — one that interleaves parallel runs differently each time without recording the actual execution order and timing — quietly destroys replayability even when every other field is captured, because the order of physical operations is itself an input. Record the realized schedule, not just the intended one.

There is a real trade-off underneath all of this: completeness versus throughput. Every provenance event is a synchronous cost on the critical path of a fast loop, and a naive implementation that blocks each robot action on a durable write will throttle the very speed that justifies the lab. The resolution is architectural — emit events asynchronously to a durable buffer, acknowledge fast, and persist to the content-addressed store off the hot path — but it has to be designed in, because a self-driving lab data provenance layer that is bolted on late tends to get sampled or disabled the first time it slows a campaign. Provenance that is optional under load is provenance you do not have.

The other standing tension is storage. Raw instrument data at full fidelity, retained for every run including failures, grows fast, and the temptation to prune “uninteresting” runs is strong. Resist it selectively: reduced results and full metadata are cheap and must be kept indefinitely, while bulky raw streams can move to cold storage under a documented retention policy — as long as the provenance record keeps the PID and hash so a future replay knows precisely what was archived and can prove it was not altered.

Practical Recommendations

Treat provenance as a build-time requirement of the SDL, not a reporting feature bolted on before publication. The cheapest provenance is the kind captured automatically at the moment of the event; everything reconstructed later is partial and disputable.

Start with the event bus and content-addressed store, because they are the substrate everything else writes to. Then instrument each component — planner, orchestrator, robot cell, instruments, ELN, LIMS — to emit PROV-O-typed events. Adopt SiLA 2 for device interfaces and AnIML or Allotrope for analytical data so instrument output arrives self-describing rather than as a blob you annotate by hand. Mint a persistent identifier for every dataset and model checkpoint, and make negative results first-class.

Governance is what keeps the system honest once it is running. Assign an owner for the provenance schema so that adding a new instrument or a new planner does not quietly introduce an un-captured edge, and gate schema changes through review the same way you would a database migration. Run the pre-publish and pre-archive gate as an automated check, not a human habit: a campaign should not be publishable until every referenced diagram, dataset, calibration reference, and version pin resolves. Periodic audits — pick a random past result and attempt a full replay — are the only reliable way to discover that a capture point silently stopped emitting weeks ago. In a mature self-driving lab data provenance program, the audit is scheduled, its results are themselves logged, and a failed replay is treated as a defect with the same seriousness as a failed test in the codebase.

Use this checklist as a minimum bar:

• Pin every dependency: code commit, container digest, protocol hash, model weights hash, material lot, calibration file — on every run.
• Capture at the source, in real time: each system emits its own provenance; never reconstruct from memory.
• Make the log immutable: content-addressed storage so records cannot be silently altered.
• Record the realized schedule, not just the plan, including actual order and timestamps of physical operations.
• Keep negative and aborted runs as first-class entities with a reason code.
• Assign PIDs and FAIR metadata so every artifact is findable, accessible, interoperable, and reusable.
• Test replay routinely: periodically re-derive an old result and diff it; a replay you never exercise is a replay that does not work.

Frequently Asked Questions

What is self-driving lab data provenance?
It is the complete, queryable record of how every result in an autonomous lab was produced — which hypothesis, plan, model version, protocol, robot actions, raw data, and reduction led to it — captured as a lineage graph so the result can be traced, trusted, and replayed rather than merely reported.

How is provenance different from ordinary logging?
Logs are append-only text a human reads after a failure; provenance is a structured, queryable graph a machine traverses to answer causal questions like “what produced this result?” and “what did this calibration file affect?” You can build provenance on top of logs, but unstructured logs alone cannot answer lineage queries.

Why is reproducibility harder in self-driving labs than in normal automation?
Because the machine chooses the next experiment based on everything it has observed, reproducing a late experiment requires reconstructing the exact model state that all prior experiments produced, plus the physical drift the hardware accumulated. The setup is a moving target, not a fixed protocol.

What do PROV-O and FAIR each contribute?
PROV-O provides the standard vocabulary for lineage — entities, activities, agents, and the relations among them — so provenance is portable across tools. FAIR (Findable, Accessible, Interoperable, Reusable) is the reuse contract that makes those records locatable and trustworthy via persistent identifiers and rich metadata. You need both.

What is the difference between repeat, replicate, and reproduce?
Repeat is the same team getting the same result with the same setup (precision). Replicate is a different team getting a compatible result with an independent implementation (robustness). Reproduce is re-analyzing the original data and code to confirm the findings without new data collection (analysis integrity).

Which failure mode causes the most trouble?
Silent calibration drift and lost negative results are the two most damaging, because both corrupt what the model learns while leaving no obvious trace. Binding calibration state to every measurement and treating aborted runs as first-class provenance entities are the direct countermeasures.

Further Reading

• W3C PROV-O: The PROV Ontology — the standard provenance vocabulary.
• The FAIR Guiding Principles (Wilkinson et al., 2016) and GO-FAIR.
• SiLA 2 standard, AnIML, and the Allotrope Foundation for instrument and analytical data.
• A-Lab, Nature 2023 and the reproducibility critique.
• Related: autonomous characterization closed-loop architecture.

By Riju — about