Llama 4 vs DeepSeek V3 vs Claude Sonnet: Industrial-Reasoning Benchmark (2026)

Llama 4 vs DeepSeek V3 vs Claude Sonnet: Industrial-Reasoning Benchmark (2026)

Llama 4, DeepSeek V3 and Claude Sonnet Reasoning for Industrial Work: A Benchmark You Can Rerun (2026)

Most model comparisons answer a question nobody on a plant floor is asking. They tell you which model recalls the most trivia or solves the most competition mathematics. They do not tell you which model can read a serialized OPC UA node tree, align an alarm flood with a maintenance ticket, or notice that a bill of materials mixes millimetres and inches. Choosing between Llama 4, DeepSeek V3 and Claude Sonnet for industrial reasoning is a different problem, and a public leaderboard cannot settle it.

It matters more now than when the first version of this post appeared, because the model landscape moved underneath it. Claude Sonnet 4 is retired, DeepSeek has shipped a V4 generation, and Meta’s lead line is no longer Llama. A comparison that names the old versions is not only stale, it is misleading.

This rewrite keeps the part that does not expire, the benchmark design, and replaces what does: the model facts, the pricing arithmetic and the cost framing. You leave with a five-family task suite, a two-layer judging stack, a cost-per-correct-answer method, and a clear rule for which model class suits which job.

What this covers: why generic benchmarks fail on plant data, the November 2026 state of the three model families, the harness architecture, the five task families, judge calibration, the cost frontier, a routing policy, failure modes and a rollout checklist.

Context and Background

Industrial reasoning is the work an engineer does between a symptom and a decision. A pump trips, three alarms fire in nine seconds, a work order from last month mentions a seal, and a P&ID shows two valves nobody labelled consistently. The reasoning is multi-hop, the inputs are semi-structured, and the cost of a confident wrong answer is a lost shift rather than a failed unit test.

Public benchmarks barely touch this. MMLU-style suites reward breadth of recall. Coding suites reward fixing issues in popular open-source repositories. Math suites reward symbolic stamina. None of them contain a Structured Text routine written in 2004 by someone who has since retired, or a tag name like FIC_2201.PV whose meaning depends on the plant.

Three structural differences explain the gap:

  • Alien token distribution. Instrument lists, ladder-logic fragments, ASCII drawings and plant-specific acronyms are rare in pre-training corpora. The abbreviation CV means control variable upstream and calorific value downstream.
  • Multi-index reasoning. Root-cause work joins an alarm log, a trend and a ticket. A model must hold three partial pictures and reconcile them.
  • Asymmetric failure cost. A hallucinated unit conversion is far more expensive than an awkward sentence, yet most scorers weigh them equally.

The serving layer also shapes results. Throughput and tail latency swing widely between inference engines, which is why our Q2 2026 inference engine comparison of vLLM, TGI, SGLang and Triton is a companion to this piece. Evaluation tooling matters too; the LLM evaluation harness architecture guide covers contamination and harness design in general terms, while this post specialises it for plant data.

A note on what this post does not do. The earlier version published a table of scores from a single 200-item run. Those numbers belonged to models that are now one or more generations old, so they are retired here. Publishing fresh numbers that we did not measure would be fabrication, so you will find a method, a scoring template and a decision policy, not a leaderboard. Where a figure below is illustrative it is labelled as such.

The Model Landscape in November 2026

Answer first: three vendor lines matter for this comparison. Anthropic’s current mid-tier model is Claude Sonnet 5.5, DeepSeek’s open-weight flagship generation is V4 with V3 as its predecessor, and Meta’s open Llama 4 family (Scout, Maverick, Behemoth) remains available but is no longer Meta’s lead product line. Facts below were checked against vendor pages in early November 2026 and should be re-confirmed before you commit budget.

Claude Sonnet 5.5 and the retirement of Sonnet 4

Anthropic’s deprecation page lists the Sonnet 4 snapshot claude-sonnet-4-20250514 as retired, with claude-sonnet-5-5 as the recommended replacement. Requests to the retired model fail, so any harness pinned to it is broken rather than merely old. The older Sonnet 4.5 snapshot is itself on a deprecation schedule, so avoid pinning to it for new work.

The current model table lists Sonnet 5.5 at roughly 2 US dollars per million input tokens and 10 dollars per million output tokens, with a one-million-token context window and 128K maximum output. That is cheaper than the 3 and 15 dollars the Sonnet 4 generation charged, which changes the cost frontier materially. Treat the prices as a snapshot: vendors revise them, and batch or cached-input discounts alter real bills.

For harness authors the practical consequence is the judge. If your judge was Sonnet 4, you must choose a new one and redo calibration before comparing anything with historical numbers. That is the single most common silent failure in evaluation pipelines after a model retirement.

DeepSeek: from V3 to V4

DeepSeek-V3 is a mixture-of-experts (MoE) model with 671 billion total parameters and about 37 billion active per token. Its code is MIT licensed and its weights ship under a separate model licence that permits commercial use. Our MoE routing deep-dive covering DeepSeek V3 and Mixtral explains why only a small slice of those parameters fires per token, which is what makes it cheap to serve relative to its size.

Since V3, DeepSeek released V3.2 in December 2025 and then the V4 generation. As reported by DeepSeek and secondary coverage, V4-Pro is a roughly 1.6 trillion total, 49 billion active parameter model and V4-Flash is about 284 billion total, 13 billion active, both with a one-million-token context and MIT-licensed weights on Hugging Face. DeepSeek’s API pricing page now lists a flash tier and a pro tier with different off-peak and peak rates; the flash tier’s output price is well under two dollars per million tokens even at peak. Check the live pricing page before computing any budget, because the peak and off-peak split makes a single number misleading.

One hardware claim from older write-ups fails a basic check: 671 billion parameters at 8-bit precision is roughly 671 GB of weights, which cannot fit on a single 80 GB H100. Self-hosting any of these frontier MoE models needs a multi-GPU node, and the V4 generation needs more.

Llama 4: Scout, Maverick, Behemoth

Meta’s Llama 4 announcement describes three mixture-of-experts models. Scout has 17 billion active and 109 billion total parameters across 16 experts and advertises a 10-million-token context. Maverick has 17 billion active and about 400 billion total across 128 experts, with a one-million-token context on its model card. Behemoth, at 288 billion active and nearly two trillion total, was described as still training at announcement.

The earlier version of this post benchmarked “Llama 4 70B” and “Llama 4 405B” as dense decoders. We could not find Llama 4 models of those sizes in Meta’s materials, so those labels should be read as stand-ins for a small and a large open-weight Llama, not as Llama 4 releases. The Llama 4 Community License also carries conditions worth reading: a separate licence is required above 700 million monthly active users at release, distribution requires “Built with Llama” attribution, and derived model names must begin with “Llama”.

Strategically, Meta launched a proprietary model called Muse Spark in April 2026 and said current Llama models will remain open source, with future Muse versions planned to be open-sourced. That is stated intent, not a shipped fact for larger models, so do not plan a roadmap on it.

What this means for the comparison

The honest framing for 2026 is not three named models but three archetypes: a frontier closed API model, a very large open MoE with cheap per-token economics, and a mid-sized open MoE you can realistically host yourself. The benchmark below is designed so you can drop any current model into each slot.

Benchmark Architecture: Seven Stages That Isolate Failure

Answer first: a trustworthy industrial benchmark separates structural failures from reasoning failures. Run each prompt several times at low temperature, check the output deterministically first, then pass only what survives to a calibrated LLM judge, and report medians with bootstrapped confidence intervals rather than single runs.

Industrial LLM benchmark pipeline from task source to cost frontier report for Llama 4, DeepSeek and Claude Sonnet

Figure 1: The seven-stage evaluation pipeline. Deterministic checks sit before the judge so that structural errors are never mistaken for reasoning gaps.

The pipeline in Figure 1 starts with a task source, which is a curated item plus a gold answer. A versioned prompt template turns it into a request. The model is called three times at a temperature of about 0.2, a normaliser strips wrappers such as markdown fences or conversational preambles, and then the deterministic layer runs. Only afterwards does the judge model score what remains, and an aggregator computes the statistics and renders the report.

The ordering is the design. If the deterministic checker fails, you know the model produced malformed JSON, wrong units or a broken tag path. If the checker passes and the judge scores low, you know it is a knowledge or reasoning gap. Collapsing the two into one judge score hides which fix you need: a schema-constrained decoder repairs the first, while only a better model or better retrieval repairs the second.

Why three repetitions and a low temperature

Language models are stochastic decoders, and a single run can land on a lucky or unlucky sample. Three repetitions at temperature 0.2 keep variance modest while still surfacing instability. Report the median per item, then bootstrap a 95 percent confidence interval across items using about a thousand resamples. If two models’ intervals overlap heavily, you have no evidence of a difference, and any ranking you publish is noise dressed as a result.

Temperature settings also interact with vendor behaviour. Some reasoning-oriented endpoints restrict or ignore sampling parameters, so record the effective settings the API reports rather than assuming your request was honoured. A benchmark that silently compared one model at 0.2 and another at its fixed default is not a comparison.

Symmetric infrastructure

Fairness at the infrastructure layer is easy to neglect. Figure 2 shows the topology: one harness, a router that selects a backend, open-weight endpoints behind an OpenAI-compatible interface, and Claude through the Anthropic Messages API. All three feed a single log that records tokens, latency and cost.

Harness topology routing a benchmark to vLLM endpoints and the Anthropic API for industrial reasoning tests

Figure 2: One harness, three backends, one log. Using the same serving stack and settings for the open models keeps latency comparable.

For open weights, pin the same vLLM version, the same KV cache dtype and the same maximum context for every model, and warm each model with a few dozen throwaway prompts before timing starts. Our vLLM prefix caching analysis explains why a warm cache can flatter a model whose prompts share a long system prefix, which is exactly the shape of an industrial prompt with a fixed instrument glossary.

Claude is a black box behind an API, so measure end-to-end wall time including network overhead and state that clearly. Do not compare its latency to a local endpoint without a caveat, and do not compare the provider’s list price to your amortised GPU cost without showing the utilisation assumption.

Context length: a confound that changed

The original benchmark capped every model at 32K tokens to remove context length as a variable. That was a reasonable fairness choice when flagship APIs topped out near 200K. In late 2026, Sonnet 5.5, DeepSeek V4 and Llama 4 Maverick all advertise context windows around a million tokens, and Scout advertises ten million.

The question has shifted from “can it fit” to “does it still reason well when the whole plant historian extract is in the prompt”. Run the suite twice: once with a tight, retrieved context and once with a deliberately bloated one. The gap between the two scores is a more useful number than either score alone, and it tells you whether to invest in retrieval or just buy a bigger window.

The Five Task Families

Answer first: five families cover most AI-assist requests that reach a PLM or digital-twin team. Each is chosen so that a failure has a distinct signature, a deterministic check exists for part of the answer, and a subject-matter expert can grade the rest in a few minutes.

Forty hand-reviewed items per family gives 200 items. That is small by NLP standards, but expert review of every item is what makes it valuable. Figure 4 later turns the outcome into a routing policy; first, the families themselves.

OPC UA question answering

Given a JSON serialization of an OPC UA address space and a natural-language question, the model returns a browse path and a value. Deterministic checks are strong here: the path either resolves in the tree or it does not, and the value either matches or it does not. This family punishes hallucinated node names, a common failure when models pattern-match on tag-like strings. The OPC Foundation’s OPC UA overview is the reference for the information model, and companion specifications for devices and machinery supply realistic node structures you can serialize without any customer data.

Root-cause analysis

The input is an alarm sequence, a trend snippet and a free-text maintenance note. The output is a ranked list of hypotheses with the evidence that supports each. Scoring uses the judge for correctness, completeness and calibration, and a deterministic check that every cited piece of evidence actually appears in the input. Citation grounding catches the most dangerous behaviour: plausible causes supported by evidence the model invented.

Bill of materials cleansing

Given a 200-line BOM, the model flags duplicates, unit mismatches and revision drift. Gold labels come from items you seeded with known defects, so precision and recall are computed mechanically. This family connects directly to product-lifecycle work; see how engineering data feeds language models in our piece on AI-native PLM and LLMs over engineering data.

Control-loop tuning hints

The model reads a PID step-response trace and suggests changes to proportional, integral and derivative gains. A deterministic layer simulates the suggested change on a simple first-order-plus-dead-time plant model and checks whether overshoot and settling improve. This is the one family where you can verify the advice by running it, and it exposes models that give generic “reduce the gain” advice regardless of the trace.

Industrial code reading

The input is 300 lines of ladder logic converted to text, or Structured Text per IEC 61131-3. The model explains behaviour in plain English and lists risks such as missing interlocks or unguarded latches. Seeded defects again give a mechanical recall measure, and the judge scores clarity and safety awareness.

Dataset sourcing

Use two sources. Synthetic items are generated from public schemas, ISA-95 examples and open BOMs, which lets you publish that subset. Scrubbed real items come from anonymised projects, with plant codes, tag names and identifying numerics replaced and a second reviewer confirming anonymity, and you need written consent to use them. A 60 percent synthetic and 40 percent real mix is a sensible starting point: pure synthetic is too clean, pure real cannot be shared. Never let customer data reach a hosted API unless the contract allows it, which is itself an argument for including a self-hosted model in the comparison.

Judging: Deterministic First, Calibrated Judge Second

Answer first: an LLM judge is only as good as its agreement with human experts. Calibrate it on a set of expert-rated items, tune the rubric until Cohen’s kappa clears a bar you set in advance for every axis, and re-run that calibration whenever the judge model changes.

Sequence of an industrial benchmark item through model, deterministic checker and calibrated LLM judge

Figure 3: The life of one benchmark item, from submission through deterministic checks and judging to cost per correct answer.

Figure 3 traces one item. The engineer submits an item with a gold answer, the harness prompts the candidate three times, the deterministic checker validates schema, units and graph structure, and only then does the judge model score the answer on a zero to four rubric across correctness, completeness, safety and clarity. The harness returns the median, the confidence interval and the cost per correct answer.

The deterministic layer

Deterministic checks are cheap, reproducible and immune to the judge’s biases, so push as much scoring there as possible. Units can be checked with a quantity library such as Pint. JSON output can be validated against a schema. Tag trees can be compared as graphs. Seeded BOM defects give exact precision and recall. A rule of thumb: if you can write the check in under fifty lines, do not ask a language model to do it.

# Sketch of a deterministic unit-aware numeric check (illustrative, not a full harness)
import pint
ureg = pint.UnitRegistry()

def numeric_ok(answer: str, gold: str, rel_tol: float = 0.01) -> bool:
    a = ureg.Quantity(answer)
    g = ureg.Quantity(gold)
    if a.dimensionality != g.dimensionality:
        return False                      # wrong physical quantity
    return abs(a.to(g.units).magnitude - g.magnitude) <= rel_tol * abs(g.magnitude)

A check like this fails an answer of “12 psi” against a gold of “0.83 bar” only if the conversion is wrong, which is precisely the failure that costs a shift in a real plant.

Calibrating the judge

Rate 80 or so items by hand with two experts, resolve disagreements, and compare the judge against the consensus on each axis. The earlier version of this benchmark required kappa above 0.78 on every axis, a bar that is a judgement call rather than a standard. Pick your own threshold, write it down before you look at results, and treat a miss as a reason to refine the rubric rather than to lower the bar.

Three biases deserve explicit tests. Verbosity bias makes judges prefer longer answers, so include pairs where the shorter answer is better. Self-preference bias makes a judge favour text in its own style, so if Claude is both a candidate and the judge, validate that its scores for Claude’s answers are not systematically higher than the human scores. Position bias matters in pairwise formats, so randomise order.

The safest design for a head-to-head including a Claude candidate is to use a judge from a different family, or to use two judges and report both. If the two disagree on the ranking, that disagreement is itself a finding.

Contamination and leakage

Public industrial schemas and open BOMs may appear in training data, so keep a private held-out subset built from scrubbed real items and never publish it. Compare the model’s performance on public versus private items. A large gap suggests memorisation rather than reasoning. Our evaluation harness architecture article covers contamination checks in more depth.

Reading the Results: Model Personality, Not a Leaderboard

Answer first: expect each archetype to show a recognisable personality across the five families. The patterns below are hypotheses to test on your own data, drawn from how these model classes generally behave and from the earlier run’s qualitative observations. They are not new measurements, and no new scores are claimed here.

The earlier single run suggested that the closed frontier model led on free-form reasoning and code reading, the DeepSeek line was strongest on strict structured output, and the smaller open model was the economical workhorse. Those observations came from superseded models, so use them as priors to confirm, not conclusions to repeat.

Frontier closed model: free-form reasoning and calibrated doubt

The value of a frontier model like Sonnet 5.5 on industrial work is usually not raw accuracy but how it handles uncertainty. In root-cause work, a model that says “the evidence supports two hypotheses and cannot separate them” is more useful than one that confidently picks one. Score this explicitly with a calibration axis: reward correct hedging, penalise confident errors more than abstentions.

Typical friction is strict output formats. Closed models sometimes wrap JSON in prose or add keys. Most vendors now offer structured-output or tool-use modes that constrain the response to a schema, and you should enable them in the benchmark when the production system would. Otherwise you measure a deployment mistake instead of the model.

Large open MoE: schema discipline and economics

Large open mixture-of-experts models such as the DeepSeek line tend to be strong at following explicit output formats and cheap per token. Their risk on root-cause tasks is premature commitment to one hypothesis when evidence is ambiguous. If your application is a tag-lookup assistant or an extraction pipeline, this class is often the right default. If it is diagnostic, test the abstention behaviour carefully.

Self-hosting is a real option but not a casual one. A model with hundreds of billions of parameters requires a multi-GPU node, careful expert parallelism and an operations team. Our tensor, expert and data parallelism guide lays out why serving MoE models differs from serving dense ones.

Mid-sized open MoE: the owned-hardware workhorse

A model in the Llama 4 Scout or Maverick class is the realistic self-hosting target for most plants. It handles retrieve, summarise and extract work well, and its weakness is deep multi-hop deduction, particularly root-cause items needing three or more evidence joins. If a task is mostly lookup plus formatting, it is the economical choice; if it is genuinely deductive, step up a class. Remember that a mixture-of-experts model’s memory footprint follows its total parameters even though compute follows the active ones, so a 400-billion-total model still needs hundreds of gigabytes of GPU memory.

Cost per Correct Answer

Answer first: the cost that matters is not price per million tokens but spend per answer that an engineer accepts without rework. Compute raw cost per attempt, divide by the acceptance rate, and add the human cost of the wrong ones. A cheap model with a low acceptance rate can be the most expensive option.

The formula has three parts. Raw cost per attempt is input tokens times input price plus output tokens times output price. Cost per correct answer is raw cost divided by the fraction of answers that pass your acceptance test. Total cost of ownership adds the review time spent on rejected answers, which usually dwarfs token spend.

Here is a worked example with deliberately illustrative numbers, not measurements. Suppose a task averages 800 input and 600 output tokens. At 2 dollars per million input and 10 per million output, a call costs about 0.0016 plus 0.0060, so roughly 0.0076 dollars. At an 80 percent acceptance rate, that is about 0.0095 dollars per correct answer. Now assume a cheaper model at 0.30 and 1.20 per million costs about 0.00096 per call; at a 60 percent acceptance rate it is about 0.0016 per correct answer. The cheap model wins on tokens by six times.

Now add review. If an engineer spends three minutes at a loaded cost of 60 dollars an hour on each rejected answer, that is 3 dollars per rejection. The 80 percent model leaves 20 rejections per hundred, or 60 dollars; the 60 percent model leaves 40, or 120 dollars. Token spend per hundred answers was under a dollar for either. The human term decides the outcome, which is why acceptance rate, not list price, should drive model selection for anything an engineer must review.

The owned-hardware term

For self-hosted models, amortise capital cost over realistic utilisation. A GPU node idle half the time costs twice as much per token as one busy all day, so state your utilisation assumption beside every self-hosted figure. Add power, cooling, engineering time for upgrades and the cost of evaluating each new release. Our offline batch inference comparison of provider batch APIs and self-hosted vLLM works through when batch pricing beats owning hardware for non-interactive jobs such as nightly BOM cleansing.

Price volatility

Both Anthropic and DeepSeek changed prices and model lineups several times within a year. A cost frontier computed in spring may be wrong by autumn. Store prices as dated configuration, not constants in code, and have the report print the date. DeepSeek’s peak and off-peak split adds another wrinkle: a nightly batch job can pay a fraction of the daytime rate, so schedule non-urgent evaluation runs accordingly.

A Routing Policy Instead of a Winner

Answer first: do not pick one model for the plant. Route by task shape, validate every answer deterministically, and keep an engineer in the loop for anything that changes a setpoint, a recipe or a safety function.

Routing policy that sends industrial questions to open MoE, frontier or small open models and a validator

Figure 4: A routing policy by task shape. Every path ends in a deterministic validator and an engineer review gate.

Figure 4 encodes four rules. Strict-schema work, such as tag lookups and structured extraction, goes to a cost-efficient open MoE model with schema-constrained decoding. Free-form diagnostic work goes to a frontier model, accepting higher token cost for better calibration. Bulk summarisation and extraction goes to a small open model on owned GPUs. Anything whose data cannot leave the site goes to self-hosted open weights regardless of benchmark rank.

All four paths converge on the same validator and review gate. This is the part teams skip under schedule pressure and regret later. A language model is a proposer of answers, not an authority, and in an environment governed by functional-safety thinking, the validator and the human are the controls that count. Never route a model output directly to a controller write.

Implementing the router

A router can be as simple as a lookup keyed on task family, with a confidence-based escalation: if the cheap model’s answer fails the deterministic check or its self-reported uncertainty is high, retry on the frontier model. Log the escalation rate. If more than a third of traffic escalates, your cheap tier is not carrying its weight and the savings on paper are not savings in practice.

# Illustrative routing config; model identifiers must be checked against vendor docs
routes:
  opcua_qa:        { primary: open_moe_large,  escalate_to: frontier, schema: browse_path_v2 }
  root_cause:      { primary: frontier,        escalate_to: null,     require_citations: true }
  bom_cleanse:     { primary: open_mid_moe,    escalate_to: frontier, batch: true }
  loop_tuning:     { primary: frontier,        simulate: fopdt }
  code_reading:    { primary: frontier,        escalate_to: null }
review_gate: always

Running the harness

A minimal model configuration for a current run looks like the following. Model identifiers change; always copy them from the vendor’s model listing, as the earlier post’s identifier for Sonnet 4 did not match the documented one.

models:
  - id: llama4-maverick
    backend: openai_compatible
    endpoint: http://vllm-cluster:8000/v1
    temperature: 0.2
  - id: deepseek-v4-flash
    backend: openai_compatible
    endpoint: https://api.deepseek.com
    temperature: 0.2
  - id: claude-sonnet-5-5
    backend: anthropic
    model_name: claude-sonnet-5-5
    temperature: 0.2
judge:
  id: a-different-family-from-the-candidates
  calibration_set: calibration/expert_rated_80.jsonl
  min_kappa: 0.75   # set before seeing results

Check each vendor’s documentation for whether temperature is honoured on the specific endpoint, and record what the API actually applied. The Anthropic models overview lists current identifiers and pricing, and DeepSeek’s pricing page lists its current tiers.

Trade-offs, Gotchas, and What Goes Wrong

Answer first: the common failures are not model failures but evaluation failures. Retired model IDs, uncalibrated judges, tiny samples and unlabelled assumptions produce confident, wrong conclusions.

Retired or renamed models. A pinned identifier can vanish. Add a start-of-run check that calls each model with a trivial prompt and fails loudly on error, and store the exact snapshot string in every results file so old numbers remain interpretable.

Judge drift. Replacing the judge silently changes every score. Version the judge, keep its calibration report with the results, and never compare runs across judge versions without re-scoring a common subset.

Small samples. Forty items per family cannot resolve a difference of a few percentage points. Use confidence intervals, and describe overlaps honestly. If you need finer resolution, grow the suite where the models are closest, not uniformly.

Prompt overfitting. Tuning prompts on the benchmark items and then reporting on them inflates results. Split items into a development set for prompt work and a locked test set for reporting.

Format versus reasoning. A model that wins on schema compliance may lose on reasoning, and vice versa. Report the two separately, as the pipeline in Figure 1 allows, and decide which matters for each route.

Retrieval confounds. In production, retrieval quality often dominates model choice for CAD, BOM and document tasks. If your benchmark feeds gold context, you measure the model in a best case. Run a second condition with your real retriever and compare.

Safety and confidentiality. Plant data can be sensitive and, in some sectors, regulated. Sending scrubbed items to a hosted API is a decision for your security team, not an evaluation convenience. Record where each item was processed.

Over-trusting synthetic data. Synthetic items generated by a language model can be easier for language models. Prefer items derived from real schemas and hand-written defects.

Licence surprises. Open weights are not all equally open. The Llama Community License has user-count and naming conditions, and DeepSeek’s weights licence differs from its MIT-licensed code. Have legal read the actual text before productising.

Practical Recommendations

Start small and make the loop repeatable. A benchmark you can rerun in an afternoon will be rerun; one that takes a month will be done once and forgotten. Because models are replaced every few months, rerunnability is the real asset, not any single score table.

Begin with 20 items per family from your own plant, scrub them, and have an expert write the gold answers. Build the deterministic layer first because it gives immediate value and anchors the judge. Calibrate a judge from a different family than your candidates. Run the suite against one model per archetype, read the confidence intervals, and only then decide whether you need more items.

Treat the cost model as a living spreadsheet with dated prices and an explicit utilisation figure. When a vendor retires a model or changes a price, rerun and diff. When a new model arrives, add it as a fourth column rather than starting over.

Checklist:

  • Pin exact model snapshot identifiers and record them per run.
  • Verify each identifier still resolves before a run starts.
  • Split items into development and locked test sets.
  • Put deterministic checks before the judge.
  • Calibrate the judge and record kappa per axis.
  • Use a judge from a different family than the candidates.
  • Report medians with bootstrap confidence intervals.
  • Show cost per correct answer with the review-time term.
  • State GPU utilisation next to every self-hosted cost.
  • Keep an engineer review gate on every route.
  • Print the run date and price snapshot in every report.

Frequently Asked Questions

Which is best for industrial reasoning, Llama 4, DeepSeek or Claude Sonnet?

No single model wins every task. Frontier closed models like Claude Sonnet 5.5 are usually strongest on free-form diagnosis and calibrated uncertainty, large open MoE models like DeepSeek’s often win on schema-heavy extraction and price, and mid-sized Llama 4 models suit self-hosted retrieval and summarisation. The only reliable answer comes from running a suite built from your own plant data and comparing medians with confidence intervals.

Is Claude Sonnet 4 still available for benchmarking?

No. Anthropic lists the Sonnet 4 snapshot claude-sonnet-4-20250514 as retired, with Claude Sonnet 5.5 recommended as its replacement, and requests to the retired model fail. If your harness used Sonnet 4 as a judge, you need a new judge and a fresh calibration against expert ratings before comparing any new results with old ones.

Can I self-host DeepSeek or Llama 4 on a single GPU?

Not the large ones. A 671-billion-parameter model at 8-bit precision needs roughly 671 GB for weights alone, far beyond one 80 GB GPU, and the V4 generation is larger. Llama 4 Scout, at 109 billion total parameters, can be run on a single high-memory GPU only with aggressive quantization, and you should verify memory needs for your context length and batch size.

What is cost per correct answer and why use it?

It is the raw cost of an attempt divided by the share of answers an engineer accepts, plus the review cost of rejected ones. It beats price per million tokens because a cheap model with a low acceptance rate can cost more once human rework is counted. It also forces you to define acceptance, which is the hardest and most valuable step.

How many test items do I need?

Forty expert-reviewed items per family is a workable start and gives 200 across five families, but it only resolves large differences. Use bootstrapped confidence intervals to see whether rankings are real, and add items where two models are closest. Quality of gold answers matters more than count, and a locked held-out set prevents you from overfitting prompts.

Should an LLM ever write directly to a control system?

No. Treat model output as a proposal. Pass it through deterministic validation, then an engineer review gate, and keep any write path to controllers under your existing change-management and functional-safety processes. Benchmarks measure reasoning quality, not suitability for closed-loop control, and nothing in this method certifies a model for safety-related use.

Further Reading

By Riju — about

Comments

No comments yet. Why don’t you start the discussion?

Leave a Reply

Your email address will not be published. Required fields are marked *