MLflow 3: Experiment Tracking, Model Registry and GenAI Tracing

Most machine learning teams do not fail because their models are bad. They fail because nobody can say which code, data and parameters produced the model that is currently in production, or why last Tuesday’s agent started answering differently. MLflow is the open source project that most teams reach for to close that gap, and with the 3.x line it has stretched from classic experiment tracking into the messy territory of generative AI applications.

That stretch matters now because the unit of deployment has changed. A modern system is rarely one pickled estimator. It is a model, a prompt, a retrieval index, a tool graph and a set of evaluators, all of which change independently. MLflow 3 responds with a new first-class entity, the LoggedModel, plus tracing and evaluation built around it.

This guide gives you a working mental model of the whole platform, the data model underneath it, a production deployment shape, and the failure modes that bite teams at scale. You will leave able to decide what to adopt, how to host it, and what to watch.

What this covers: architecture, the run and LoggedModel data model, the registry with aliases, MLflow Tracing and OpenTelemetry, production hosting, alternatives, code, and failure modes.

Context and Background

MLflow began in 2018 as a way to stop teams reinventing the same three things: logging experiments, packaging models, and promoting them. For years it was described through four components, namely Tracking, Projects, Models and the Model Registry. Tracking recorded parameters, metrics and artifacts per run. Models defined a packaging format with “flavors” so one artifact could be loaded by sklearn, PyTorch, XGBoost or a generic Python function. The Registry added named, versioned models with lineage.

Two things changed the picture. First, deep learning and large language models made single-run, single-model thinking too narrow: checkpoints multiply, and evaluation happens continuously rather than once. Second, GenAI applications introduced behaviour that metrics alone cannot explain. When an agent calls a retriever, three tools and a model before answering, you need the full execution tree, not a final accuracy number.

MLflow 3 is the project’s answer. According to the official MLflow 3 documentation, the headline additions are the LoggedModel entity that acts as a metadata hub linking an application version to its code, configuration, traces and evaluation runs, GenAI evaluation, a Prompt Registry, and linking of traces to a model through a model_id. The same page lists removals such as MLflow Recipes, the fastai and mleap flavors, and the AI gateway client APIs, so a migration from 2.x deserves a read of the release notes rather than a blind upgrade. At the time of writing, PyPI listed mlflow 3.17.1 (released 13 October 2026) as the latest version, requiring Python 3.10 or later; check PyPI for the current number before pinning.

It also helps to place MLflow within the wider operations landscape. If your models run at the edge, the packaging and promotion concerns here connect directly to our guide on edge MLOps pipelines for industrial IoT. If your features must match between training and serving, pair this with feature store architecture and online/offline parity. The authoritative reference for everything below is the official MLflow documentation, which this article cross-checks against throughout.

One caution on scope. MLflow is a system of record and a set of conventions, not a scheduler, a feature store or a training orchestrator. It records what happened and gives you stable names to refer to it. Teams that expect it to run pipelines are disappointed; teams that treat it as the ledger of the ML lifecycle are not. That framing drives most of the design choices later in this article.

MLflow Reference Architecture: Server, Stores and Registry

At its simplest, MLflow is a client SDK that writes to two stores, fronted by an optional server. The backend store holds metadata such as experiments, runs, parameters, metrics, tags and traces. The artifact store holds large binary payloads such as model weights, plots and datasets. The tracking server is a web application that exposes a REST API and the UI, brokers access to both stores, and hosts the Model Registry. Everything else is a client of these three things.

Figure 1: MLflow reference architecture. Training jobs, GenAI apps and OpenTelemetry-instrumented services all converge on the tracking server, which splits metadata and artifacts across two stores.

Figure 1 shows the shape. Training code and GenAI applications talk through the client SDK, while services instrumented with OpenTelemetry can send traces directly to the server’s OTLP endpoint. The server persists metadata to a relational database and artifacts to object storage, and the registry sits on the same metadata layer so that versions and aliases are transactional.

Backend store: where metadata lives

The official architecture documentation says the backend store is a database, or a file system that emulates one, and that it works through SQLAlchemy with PostgreSQL, MySQL, SQLite and MSSQL supported. Treat SQLite as a laptop option and PostgreSQL as the sensible production default. The reasons are concurrency and operability: run logging produces many small writes, and the registry needs transactions that behave under contention.

The default experience requires no server at all. Without configuration, the client writes metadata and artifacts to a local mlruns directory. That is perfect for a notebook and a trap for a team, because file-based storage has no access control, poor concurrency and no clean path to the registry features that need a SQL database. If more than one person or job writes, move to a database-backed store on day one.

Artifact store: where bytes live

The artifact store is ordinary object storage in practice, with Amazon S3 the example the documentation gives and Google Cloud Storage and Azure Blob Storage the common siblings. Weights for a large model can run to many gigabytes, so you never want those streaming through a database or through a single small server process.

There are two access modes. In direct mode, each client talks to object storage itself, which needs credentials on every client. In proxied mode, the tracking server mediates artifact upload and download, which centralises credentials but makes the server a data path and a bottleneck. The documentation describes the server as a FastAPI application that acts as a proxy for remote access to metadata and artifacts. Choose proxied access when you value central control, and plan its capacity accordingly.

Registry, flavors and the model format

A model in MLflow is created by a flavor’s log_model() method or, since MLflow 3, by mlflow.create_external_model() for models hosted elsewhere. The flavor mechanism wraps native frameworks and also offers the generic pyfunc interface, which gives any model a uniform predict contract, an environment specification and an input and output signature. That uniformity is what lets the same registered model be loaded by a batch job, a container or a serverless endpoint without bespoke glue.

The registered model is the stable identity; model versions are immutable snapshots under it; and aliases are movable pointers. We return to this in the registry section because aliases replace the older stage mechanism and the difference shapes how you build promotion workflows.

Why separating metadata from bytes matters

The split is not a trivia point. It determines how the system fails. Metadata is small, relational and queried constantly: search for runs with accuracy above a threshold in this experiment. Artifacts are large, write-once and read rarely. Putting them on the same store would force you to scale the expensive thing for the cheap workload.

The practical consequence is that the two stores can drift apart. A run row can exist whose artifacts were lifecycle-expired, or artifacts can exist for runs deleted from the database. Many of the real operational problems we cover later, from broken model loads to runaway storage cost, trace back to this separation, so keep it in mind when you design retention.

The Data Model: Experiments, Runs, LoggedModels and Lineage

A reference architecture tells you where things are stored. The data model tells you what questions you can ask. MLflow’s model is deliberately small, and understanding its few nouns precisely is the fastest way to avoid a messy tracking history.

An experiment is a named container, typically one per problem or product surface. A run is one execution: it has parameters (inputs, written once), metrics (numeric series that can have a step and timestamp), tags (free-form labels) and artifacts (files). Runs can nest. A parent run represents a hyperparameter sweep or a multi-stage pipeline, and child runs represent the individual trials or stages, linked by a parent tag.

Figure 2: The MLflow 3 data model. Runs produce LoggedModels, LoggedModels are registered as versions, and aliases point at versions. Traces and evaluation runs attach to the LoggedModel.

Figure 2 adds the MLflow 3 piece. A LoggedModel carries its own model_id, and that identifier becomes the join key between the thing you built and everything that observed it. The official documentation describes it as a metadata hub linking an application version to external code such as a Git commit, to configuration, to traces and to evaluation runs, and states that it also works for traditional ML models and deep learning checkpoints.

Why a LoggedModel and not just a run

In MLflow 2.x the model was effectively an artifact inside a run. That worked when one run produced one model and nobody asked questions afterwards. It broke down in two situations.

The first is deep learning, where one training run emits many checkpoints, and you want to compare them as first-class objects with their own metrics rather than as folders inside a run. The second is GenAI, where an “application version” may involve no training run at all: a new prompt, a changed retriever, a different model endpoint. There is nothing to run, yet there is clearly a version whose quality you want to track.

A LoggedModel resolves both by making the model a top-level entity. Metrics can be logged against it, traces can be grouped under it, and evaluation runs can reference it. The documentation shows mlflow.set_active_model(name=...) to declare the application version being exercised, mlflow.get_active_model_id() to retrieve its identifier, and passing model_id to mlflow.search_traces() and mlflow.log_metrics() to scope queries and measurements. The MLflow 3 notes also say that mlflow.pyfunc.log_model() now takes name rather than the deprecated artifact_path, and that logging a model no longer requires wrapping the call in mlflow.start_run().

Parameters, metrics and tags: conventions that save you later

The schema is permissive, which is both a strength and a hazard. Three conventions pay for themselves quickly.

Use parameters for everything that defines the run and is known up front: learning rate, dataset version, prompt hash, base model name. Parameters are write-once, which protects you from accidental overwrite and makes runs comparable. Use metrics for anything that changes over time or is measured after the fact, and always pass a step, because a metric without a step collapses into a single point and loses the training curve. Use tags for organisational and lineage facts: the Git commit, the pipeline run identifier, the owning team, the data snapshot.

Lineage is the real prize. If every run carries its code commit, its data version and its upstream pipeline identifier, you can answer an audit question in one query instead of a week of archaeology. If you use a feature store, log its feature view version too; the feature store parity article explains why a mismatch there is among the quietest sources of training-serving skew.

A minimal tracking example

The following snippet shows the core loop: set the tracking URI, choose an experiment, log parameters and stepped metrics, and log a model with a signature. It is deliberately plain so that you can see what autologging would otherwise do for you.

import mlflow
from mlflow.models import infer_signature
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

mlflow.set_tracking_uri("http://mlflow.internal:5000")
mlflow.set_experiment("fraud-screening-baseline")

X, y = make_classification(n_samples=5000, n_features=20, random_state=7)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=7)

with mlflow.start_run(run_name="rf-depth-sweep") as parent:
    mlflow.set_tag("git_commit", "abc1234")
    for depth in (4, 8, 12):
        with mlflow.start_run(run_name=f"depth-{depth}", nested=True):
            model = RandomForestClassifier(max_depth=depth, n_estimators=200)
            model.fit(X_tr, y_tr)
            acc = accuracy_score(y_te, model.predict(X_te))
            mlflow.log_param("max_depth", depth)
            mlflow.log_metric("accuracy", acc)
            sig = infer_signature(X_te, model.predict(X_te))
            mlflow.sklearn.log_model(model, name="model", signature=sig)

Note the nested=True flag: it is what builds the parent and child structure in Figure 2. The model is logged with name="model", matching the MLflow 3 convention described above. The dataset and metric here are synthetic and chosen only to keep the example self-contained.

Autologging: convenience with a cost

Autologging removes boilerplate. The documentation says that calling mlflow.autolog() before training enables automatic logging for each supported library you have installed, as soon as you import it, and that it records metrics chosen for the model and library, parameters including library defaults you did not set, a model signature, artifacts such as checkpoints, and training datasets where applicable. It also describes run handling: if no run is active, MLflow creates one and ends it after training; if a run is active, it logs into that run and leaves it open.

The documented list of supported libraries includes Keras and TensorFlow, LightGBM, Paddle, PySpark, PyTorch (via PyTorch Lightning only), scikit-learn, Spark, Statsmodels and XGBoost. You can disable one library with its flavor and disable=True, for example mlflow.sklearn.autolog(disable=True), then keep the rest on.

Autologging is a patching mechanism: it wraps training entry points such as fit so that the library’s own callbacks and return values feed the tracker. That has two consequences worth planning for. Library upgrades can change what is captured or break the patch, so pin library and MLflow versions together. And autologging records what the library exposes, not what matters to your business. You will still want explicit calls for your own evaluation metric, your data version and your lineage tags.

Model Registry: Versions, Aliases and Promotion Workflows

The registry converts “a model file in a run” into “a named thing the organisation can refer to”. It provides a central store, API set and UI for managing a model’s lifecycle, and it tracks lineage, versions, aliases, tags and annotations.

The mechanics are simple. The first time you register a model under a name, it becomes version 1; each later registration under the same name increments the version. You can register in one step by passing registered_model_name to a flavor’s log_model(), or afterwards by calling mlflow.register_model(model_uri=..., name=...) on an already logged model. You load a specific version with a models:/<name>/<version> URI.

Aliases replace stages

Older MLflow versions moved model versions through fixed stages: None, Staging, Production, Archived. The official workflow documentation states that “Model Stages are deprecated and will be removed in a future major release”, and that they have been deprecated since MLflow 2.9.0. Aliases are the replacement.

An alias is a mutable, named reference to one version of a registered model. You choose the names. Reassigning champion from version 1 to version 2 changes what every consumer of models:/example-model@champion receives, without touching consumer code. The documented client calls include set_registered_model_alias, get_model_version_by_alias and delete_registered_model_alias, alongside tag management through set_model_version_tag and set_registered_model_tag.

from mlflow import MlflowClient
import mlflow

client = MlflowClient()
name = "fraud-screening"

# Register the best child run from the sweep
mv = mlflow.register_model(model_uri="runs:/<RUN_ID>/model", name=name)

# Tag the version with validation evidence, then move a candidate alias
client.set_model_version_tag(name, mv.version, "validation_status", "passed")
client.set_registered_model_alias(name, "challenger", mv.version)

# Consumers load by alias; promotion is an alias move, not a redeploy
model = mlflow.pyfunc.load_model(f"models:/{name}@champion")

Why is this better than stages? Fixed stages hard-code a promotion philosophy. Real organisations have more states: a shadow deployment, a regional canary, a regulated-market variant, a rollback target. Aliases let you model each as a name, and several aliases can point at the same version at once. Stages also allowed only one version per stage in practice, which made side-by-side comparison awkward.

A promotion workflow that survives audits

A robust pattern uses tags for evidence and aliases for intent. A training pipeline registers a new version and tags it with the run, the data snapshot and the evaluation results. A separate validation job reads those tags, runs additional checks, and writes a validation_status tag. Only when the tag says passed does a promotion job move the challenger alias, and only after a shadow or canary period does it move champion. The previous champion version is recorded in a tag before the move, so rollback is another alias move.

Notice what this design does not do: it does not let humans edit the production pointer by hand in the UI as the normal path. Reserve manual alias edits for emergencies, and put alias changes behind an automated job with a service identity so the audit trail has meaning. For monitoring after promotion, connect the champion’s predictions to drift checks as described in our guide to ML model monitoring with data drift, concept drift, PSI and KS tests.

Registry race conditions and idempotency

Two jobs promoting concurrently can leave the alias pointing at a version neither intended. Because alias assignment is last-writer-wins, make promotion a single serialized job, or compare the current alias target with an expected value before writing. Likewise make registration idempotent: derive a deterministic key from the run identifier and check for an existing version tagged with it before creating another, otherwise a retried pipeline task will create duplicate versions that confuse downstream consumers.

MLflow Tracing and GenAI Evaluation

Classic experiment tracking answers “which hyperparameters gave the best score”. GenAI applications need a different question answered: “what exactly happened inside this one request”. MLflow Tracing is the observability layer built for that. The documentation describes it as recording the inputs, outputs and metadata of each intermediate step in a request, so you can pinpoint where a bug or unexpected behaviour originated.

How a trace is built

A trace is a tree of spans. The root span covers the whole request, and child spans cover retrieval calls, tool invocations, model calls and any function you chose to instrument. Each span stores its inputs, outputs, timing and attributes. You can create spans in three ways: one-line automatic tracing for supported libraries, the @mlflow.trace decorator on your own functions, and manual tracing of arbitrary code blocks.

The documentation names OpenAI, LangChain, LlamaIndex, DSPy, Pydantic AI and Vercel AI among the auto-traced integrations, with mlflow.openai.autolog() as the example, and notes that integrations can be combined. Combining matters in practice because a real agent rarely uses one library: a LangChain graph may call an OpenAI client inside a custom tool, and you want one coherent tree rather than two disconnected ones.

Figure 3: Lifecycle of a trace. The app calls an instrumented function, spans are recorded through an OpenTelemetry layer, traces export asynchronously to the server, and evaluation jobs later query traces by model_id and write feedback back.

Figure 3 shows the flow. The key design choice is the asynchronous export. The documentation says async logging writes traces in the background so application performance is less affected, which is what makes tracing viable on a latency-sensitive path. It also lists production controls: grouping and filtering traces by session, redacting PII, setting a sampling ratio, disabling tracing globally, and propagating trace context across services.

A traced agent in code

import mlflow
from openai import OpenAI

mlflow.set_tracking_uri("http://mlflow.internal:5000")
mlflow.set_experiment("support-agent")
mlflow.set_active_model(name="support-agent-v7")  # LoggedModel for this version
mlflow.openai.autolog()                           # auto-trace model calls

client = OpenAI()

@mlflow.trace
def retrieve(query: str) -> list[str]:
    # your vector search here
    return ["Refunds are processed within 5 business days."]

@mlflow.trace
def answer(query: str) -> str:
    context = "\n".join(retrieve(query))
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "system", "content": f"Use only this context:\n{context}"},
                  {"role": "user", "content": query}],
    )
    return resp.choices[0].message.content

print(answer("How long do refunds take?"))

Each call to answer now produces a trace with a root span, a retrieve child, and an automatically captured model-call child. Because set_active_model was called, the trace is linked to the support-agent-v7 LoggedModel, and mlflow.search_traces(model_id=...) can later retrieve exactly the traffic that one application version served. The model name in the snippet is illustrative; substitute whatever you deploy.

Evaluation: scoring traces, not just datasets

The MLflow 3 documentation states that the same quality scorers apply from development through production, and lists built-in GenAI metrics in mlflow.metrics.genai such as answer_similarity, answer_correctness and faithfulness. The pattern is to run these scorers over a curated dataset before release, and over sampled production traces afterwards, writing results back as feedback attached to the trace or as metrics on the LoggedModel.

Be disciplined about what these scores mean. Faithfulness and correctness scorers are typically implemented with a judge model, which introduces its own variance and bias. Treat them as a regression signal, a cheap detector of “something changed”, and not as ground truth. Keep a small human-labelled set to calibrate the judge. The MLflow 3 page also notes that human feedback tracking was, at the time of that page, available only in Databricks Managed MLflow with an open source release planned, so verify current status before designing around it.

Prompt Registry

The documentation describes mlflow.genai.register_prompt() for creating versioned prompt templates with a commit message, and lists prompt optimization using evaluation feedback and labeled datasets. The idea is to give prompts the same discipline as models: immutable versions, a name consumers refer to, and a diff. If your prompts are strings inside code, a prompt change is invisible to your tracking history. If they live in a registry and the version is logged on each trace, “answers got worse after Thursday” becomes a query.

OpenTelemetry: tracing without lock-in

Tracing in MLflow is built to coexist with the wider observability ecosystem. The documentation states that it is compatible with OpenTelemetry and supports the GenAI semantic conventions for both export and ingestion. It says the server exposes an OTLP endpoint at /v1/traces that accepts traces from applications in any language with OpenTelemetry support, that traces and metrics can be exported to compatible backends such as Datadog, Grafana and Prometheus, and that dual export to MLflow and another backend at once is supported. I could not verify the exact environment variables or header names from the overview page, so consult the ingest and export subpages for those details before configuring collectors.

This matters for architecture. A polyglot company can instrument Java and Go services with standard OpenTelemetry SDKs and still land LLM-related spans in MLflow, while Python agents use the richer MLflow SDK. For the span attribute names that make this interoperable, see our walkthrough of OpenTelemetry GenAI semantic conventions for LLM and agent observability. The deployment guidance also mentions a lightweight mlflow-tracing package, recommended for production, with a footprint about 95 percent smaller than the full mlflow package. That is a meaningful point for serverless functions and slim containers where install size and cold start count.

Production Deployment: Hosting, Auth, Scaling and Retention

A laptop MLflow and a team MLflow differ in almost every operational respect. The officially documented team setup is a remote tracking server with a database-backed metadata store and an object-store artifact store, and Docker Compose is the suggested self-hosted route, with managed MLflow offerings as an alternative.

Figure 4: A production MLflow shape. A TLS and auth proxy fronts stateless tracking servers, which share a Postgres database and a versioned object store governed by backups and lifecycle rules.

Stateless servers behind a proxy

The tracking server holds no durable state of its own when pointed at a SQL database and object storage, so you can run several replicas behind a load balancer. Terminate TLS and enforce authentication at the proxy, and keep the servers on a private network. The architecture documentation says the server provides access control features and that workspaces add workspace-level permissions for grouping resources on a shared server, requiring a SQL database backend. It does not detail authentication methods on the overview page, so treat the Security documentation as the source of truth for your version and test what your deployment actually enforces rather than assuming.

A pragmatic pattern is to authenticate people with single sign-on at the proxy and give CI jobs and training pipelines service credentials, so that every registry change in the audit log is attributable. If you also use proxied artifact access, remember that your proxy and servers now carry model-weight traffic; size bandwidth and timeouts for multi-gigabyte uploads.

Capacity: where the load actually comes from

Capacity planning is driven by three workloads that behave differently. Metric logging is high-frequency and small: a deep learning job logging every step from many workers can generate a large number of rows per minute, so batch metric calls (log_metrics takes a dictionary) and avoid logging per-iteration values you will never chart. Artifact transfer is low-frequency and large and belongs on object storage bandwidth. Trace ingestion scales with request volume times spans per request, and unlike runs it is driven by production traffic, so it can dwarf your training metadata within weeks.

For traces, use the documented sampling ratio and PII redaction controls from day one rather than after an incident. A plausible policy is to keep all traces for errors and low-scoring requests and a small sample of the rest. That policy is an illustration, not an MLflow default, and you should tune it to your risk and cost profile.

Retention and cost

Metadata grows without bound unless you act. Define retention per experiment class: exploratory work can be purged after a short window, while experiments linked to a production registered model must be kept as long as the model is. Soft-deleted runs still occupy the database until a garbage collection step removes them, and deleting a run does not necessarily delete its artifacts from the bucket, which leaves orphaned objects. Schedule a periodic job to reconcile the two stores. Use object-store versioning for the artifact bucket so an accidental overwrite is recoverable, and lifecycle rules to move old checkpoints to cheaper tiers, but exempt any artifact referenced by a registered model version.

Deployment targets

A registered model is a pointer, and what you deploy is up to you. Common targets include loading the pyfunc model inside a batch job, packaging it as a container image for a Kubernetes service, or pushing it to a managed endpoint on a cloud platform. Whatever the target, resolve the alias at deploy time and record the resolved version number in the deployment metadata. An alias is a convenient input and a poor audit record, because it moves. For edge devices where you cannot reach the registry at inference time, export the model and its environment as a bundle and carry the version in the bundle manifest, as discussed in the edge MLOps pipelines guide.

How MLflow Compares with Other Experiment Trackers

MLflow is not the only choice, and the honest comparison turns on what you value. The following reflects commonly understood positioning; I did not re-verify vendor feature sets against current documentation for this article, so confirm specifics with each vendor before deciding.

Concern	MLflow	Weights & Biases	Neptune	Comet	ClearML
Licensing model	Open source, self-host or managed	Primarily hosted product with self-host options	Hosted product	Hosted product with enterprise options	Open source core plus hosted
Strength	Open format, registry, broad ecosystem, GenAI tracing	Collaboration and visualisation polish	Metadata store for large experiment volumes	Experiment comparison and reporting	Orchestration and pipelines bundled
Model packaging	Built-in flavors and pyfunc	Artifacts-based	Artifacts-based	Registry features	Built-in model repository
Lock-in risk	Low, standard APIs and OTel	Moderate	Moderate	Moderate	Lower for open core

Mark this table as qualitative and unverified at the feature level. The structural point stands: MLflow’s differentiator is that it is an open, self-hostable standard with a packaging format and a registry, rather than only a dashboard. That also means you carry operational responsibility. If your team does not want to run a database, an object store and a proxy, a managed offering of any of these tools is the rational trade.

When to choose what

Choose MLflow when you need a vendor-neutral system of record, want to host it in your own network for data governance, or need the registry and packaging format as a common contract across teams. Choose a hosted tracker when your priority is rich collaborative dashboards and you accept a vendor dependency. Choose a bundled orchestration platform when you want pipeline execution and tracking from one tool and can accept its opinions. Many organisations run MLflow as the registry of record while individual researchers use another visualisation tool; the cost of that split is duplicated logging, which is manageable if you wrap logging behind one internal helper.

Trade-offs, Gotchas, and What Goes Wrong

Every real MLflow deployment hits a subset of the same failure modes. Knowing them in advance is worth more than any feature list.

Artifact store drift. The metadata store says a model exists, but the artifact was expired by a lifecycle rule, moved to another bucket, or written with credentials that other clients cannot read. Loading fails far from the cause, usually at deploy time. Mitigate with lifecycle exemptions for registered versions, a periodic integrity job that attempts to resolve every champion and challenger URI, and a single documented artifact root per environment.

Metric explosion. Logging per-step metrics from many workers, or logging high-cardinality metric names (one name per layer per step), bloats the database and slows the UI. Log aggregates, batch calls, and keep metric names bounded. Treat metric names as a schema, not a free-for-all.

Registry race conditions. Concurrent promotion jobs, retried pipelines creating duplicate versions, and humans editing aliases during automated runs all produce confusing states. Serialize promotion and make registration idempotent, as described earlier.

Alias-as-audit-record. Because aliases are mutable, logging “deployed champion” tells you nothing a month later. Always record the resolved version.

Autologging surprises. Patched libraries change behaviour across upgrades, and autolog can log large artifacts you did not intend, such as full checkpoints at every epoch. Review what it writes and set options deliberately.

Trace volume and privacy. Traces capture prompts and outputs, which may contain personal or confidential data. Without redaction and sampling, you create a sensitive data lake by accident. Apply redaction, restrict access by experiment or workspace, and set retention shorter than for training metadata.

Judge-model drift in evaluation. If your faithfulness scorer’s underlying judge model changes, your scores shift without any change in your application. Pin the judge, log its identity with each evaluation, and recalibrate against human labels periodically.

Migration from 2.x. The documented removals in MLflow 3, including MLflow Recipes, the fastai and mleap flavors and the AI gateway client APIs, and the artifact_path to name change in model logging, mean that a version bump can break pipelines. Upgrade in a staging environment, run your model-load tests against existing registered versions, and only then move production. Also check feature availability in the open source build against Databricks-managed documentation, since some capabilities are described as managed-only at certain points in time.

Practical Recommendations

Start by deciding what is the system of record. If it is MLflow, make that true in practice: every training job and every GenAI release must create a run or LoggedModel, with code commit, data version and configuration attached. Anything that bypasses the tracker will be the thing you cannot explain later.

Host a real server early. Use PostgreSQL for metadata, object storage for artifacts, a reverse proxy for authentication, and at least two server replicas if the service becomes critical. Decide between proxied and direct artifact access deliberately, and document the choice.

Adopt aliases and abandon stages. Keep two or three aliases at most at first, such as challenger and champion, add evidence tags, and automate promotion. For GenAI work, call set_active_model per application version, trace everything in staging, sample in production, redact sensitive fields, and run scorers over both curated datasets and sampled traces.

A short checklist to close the loop:

Pin MLflow and ML library versions together; test upgrades in staging.
Use PostgreSQL and object storage; never rely on local mlruns for a team.
Log code commit, data version and feature view version as tags on every run.
Promote with aliases from an automated, serialized job; record the resolved version at deploy.
Enable trace sampling, PII redaction and a retention policy before launch.
Version prompts in the registry and log the prompt version on each trace.
Reconcile database and artifact store on a schedule; exempt registered versions from lifecycle expiry.
Calibrate judge-based scorers against a small human-labelled set.

Frequently Asked Questions

What is MLflow used for?

MLflow is an open source platform for managing the machine learning and GenAI lifecycle. Teams use it to record experiments with parameters, metrics and artifacts, to package models in a portable format, to version and promote models through a registry, and, since MLflow 3, to trace and evaluate LLM applications and agents. It is a system of record and not a scheduler or training orchestrator.

What is new in MLflow 3?

According to the official documentation, MLflow 3 introduces the LoggedModel entity that links an application version to code, configuration, traces and evaluations, GenAI evaluation metrics, a Prompt Registry, and model_id based linking of traces. It also removes MLflow Recipes, the fastai and mleap flavors and the AI gateway client APIs, and changes log_model to use name instead of artifact_path.

Are model registry stages still supported?

Stages still exist but are deprecated. The official workflow documentation says Model Stages have been deprecated since MLflow 2.9.0 and will be removed in a future major release. Use aliases instead: mutable, named pointers to a specific model version, loaded with a models:/<name>@<alias> URI. Aliases are more flexible because you define the names and several can point at one version.

Does MLflow Tracing work with OpenTelemetry?

Yes. The documentation states MLflow Tracing is compatible with OpenTelemetry and supports the GenAI semantic conventions for export and ingestion. The server exposes an OTLP endpoint at /v1/traces for applications in any OpenTelemetry-supported language, and traces can be exported to backends such as Datadog, Grafana and Prometheus, including dual export. Check the ingest and export pages for configuration details.

What do I need to run MLflow in production?

You need a tracking server, a SQL backend store such as PostgreSQL, an artifact store on object storage such as S3, and authentication at a reverse proxy. Add multiple stateless server replicas for availability, backups for the database, versioning and lifecycle rules for the bucket, and a retention policy for runs and traces. Docker Compose is the suggested self-hosted starting point in the docs.

How is MLflow different from Weights & Biases or ClearML?

MLflow’s distinguishing traits are that it is an open, self-hostable standard with a model packaging format, a registry and OpenTelemetry-compatible tracing. Hosted trackers often emphasise collaboration and visualisation polish, while ClearML bundles orchestration. Feature sets change quickly, so verify current capabilities with each vendor. Many teams use MLflow as the registry of record regardless of which dashboard researchers prefer.