Model Merging for LLMs: mergekit, TIES, DARE, SLERP and Evolutionary Merging

Most teams that need a model with two skills assume they must train for both: more data, more GPU hours, another round of evaluation. Model merging offers a different route. You take checkpoints that were already fine-tuned from the same base model and combine their weights directly, on a laptop CPU, in minutes, with no gradient step and no training data. The result is a single dense model with the same size and inference cost as each parent.

This matters now because open-weight ecosystems are crowded with specialised derivatives of the same few bases, and because the tooling (mergekit above all) has made the operation routine. It also matters because merging fails in ways that are easy to miss, from leaderboard contamination to licence conflicts.

You will leave with a working mental model of weight-space geometry, a clear picture of what linear averaging, SLERP, task arithmetic, TIES, DARE, DELLA and evolutionary search each do, and runnable mergekit configurations with an evaluation loop you can trust.

What this covers: why merging works at all, the main algorithms and their knobs, a hands-on mergekit tutorial, LoRA and mixture-of-experts variants, evolutionary recipes, and the traps that waste the most time.

Context and Background

Weight averaging is older than large language models. Averaging checkpoints along a training run, as in stochastic weight averaging, was known to improve generalisation. The modern line of work starts with “Model soups” (Wortsman et al., arXiv 2203.05482), which averaged the weights of several models fine-tuned from one pretrained network with different hyperparameters. The authors report that the averaged model often beats the best individual model and, unlike an ensemble, costs nothing extra at inference. Their headline result was a ViT-G model at 90.94% top-1 on ImageNet.

The next step was to treat the difference between fine-tuned and pretrained weights as an object in its own right. Ilharco et al. introduced task vectors in “Editing Models with Task Arithmetic” (arXiv 2212.04089): subtract the pretrained weights from the fine-tuned weights and you get a direction in weight space. Negating it removes a behaviour, adding several combines behaviours. That framing is the foundation for nearly every method in this article.

Language models arrived in the story through the open-source community. Fine-tuners on Hugging Face discovered that merged checkpoints could climb the Open LLM Leaderboard, and the mergekit library (Goddard et al., arXiv 2403.13257) packaged the methods behind a YAML interface. The paper notes that thousands of models have been merged by the community and that some resulting checkpoints ranked among the strongest open models on that leaderboard at the time. Sakana AI then showed that the recipe itself could be searched automatically (Akiba et al., arXiv 2403.13187).

Merging sits alongside other adaptation techniques rather than replacing them. If you are choosing between merging, parameter-efficient training, and distillation, our comparison of LoRA, QLoRA, full fine-tuning and distillation covers the training side. If your real problem is knowledge freshness rather than skill, read fine-tuning versus RAG versus long context first, because no merge will teach a model facts that none of its parents know.

Two constraints frame everything that follows. First, merging in the sense used here requires models with an identical architecture, and in practice a shared base checkpoint and tokenizer. Second, merging is an optimisation of a cheap proxy: you are betting that interpolating weights interpolates capabilities. Sometimes that bet pays off spectacularly. Sometimes it produces a model that is fluent and subtly broken.

Why Weight-Space Merging Works: The Core Mechanism

Direct answer: Model merging works because fine-tuning from a shared pretrained checkpoint moves weights only a short distance, and models that start in the same basin of the loss landscape stay linearly connected. Averaging or editing their deltas therefore lands in a region that still performs well, while models trained from different random initialisations generally do not merge this way.

Figure 1: The general model merging pipeline. Each fine-tune is reduced to a task vector, a merge operator combines the vectors, and the result is added back to the shared base. No training step appears anywhere.

Figure 1 shows the pipeline that almost every method instantiates. Let the base weights be theta_0 and a fine-tuned model be theta_i. The task vector is tau_i = theta_i – theta_0. A merge produces theta_0 + f(tau_1, …, tau_n), and methods differ only in the function f. Plain averaging uses the mean, task arithmetic uses a scaled sum, TIES and DARE first sparsify and reconcile the vectors, and SLERP works on the weights directly rather than on deltas.

Linear mode connectivity and the shared basin

The theoretical support is linear mode connectivity (LMC). Frankle et al. (arXiv 1912.05671) examined whether networks trained with different SGD noise end up in a linearly connected minimum, meaning the loss stays low along the straight line between their weights. For standard vision models they found this stability emerges early in training, after which the solution is largely fixed within one connected region. Fine-tuning from a strong pretrained model starts well past that point, so all descendants share a basin.

This explains the constraint that merges need a common ancestor. Ainsworth et al. (“Git Re-Basin”, arXiv 2209.04836) argue that independently trained networks live in nearly one basin only after you account for permutation symmetries of hidden units: swap two neurons and their incoming and outgoing weights, and the function is unchanged but the weight vector is not. They present algorithms that permute one network to align with another and report zero-barrier LMC between independently trained ResNets on CIFAR-10. For billion-parameter LLMs, permutation-aligning independent models is not what practitioners do. They merge siblings of a common parent, where alignment is free.

Why task vectors are small and mostly redundant

A second observation justifies the sparsification methods. Supervised fine-tuning changes weights by tiny amounts. The DARE paper (Yu et al., arXiv 2311.03099) reports that SFT delta parameters are usually very small, mostly within 0.002, and that 90% or even 99% of them can be dropped, with the survivors rescaled, while the model keeps its abilities. In other words, a task vector carries a thin sliver of essential signal inside a large mass of noise-like redundancy.

If most delta entries are removable, then two experts’ deltas rarely need the same parameters. That is the opening that sparsify-then-merge methods exploit: remove the redundancy first and the experts barely collide.

Interference: the real enemy

When you add task vectors naively, they interfere. TIES-Merging (Yadav et al., arXiv 2306.01708) names two sources: redundant parameter values that dilute the signal, and sign disagreement, where one expert wants a weight up while another wants it down. Averaging a +0.01 and a -0.01 gives zero, erasing both experts’ signals on that parameter. The authors conclude that resolving sign interference is especially important, which is why sign election is the signature step of the method.

A rough worked example (illustrative numbers, not from a paper): suppose one weight has deltas of +0.010, +0.008 and -0.009 across three experts. Plain averaging gives +0.003, a third of the dominant magnitude and sitting between two behaviours. TIES elects the positive sign because its summed mass is larger (+0.018 against -0.009), discards the dissenting -0.009, and averages the two agreeing values to +0.009. The result keeps the strong consensus intact.

Interpolation on the sphere

SLERP, spherical linear interpolation, takes a different view. High-dimensional weight tensors have a norm, and a straight-line average of two vectors with a large angle between them has a smaller norm than either endpoint. SLERP interpolates along the arc instead, preserving magnitude. For parameter t between 0 and 1, the formula is slerp(a, b, t) = sin((1-t)W)/sin(W) * a + sin(tW)/sin(W) * b, where W is the angle between a and b. In mergekit, t = 0 returns the base model and t = 1 the other model, and the method accepts exactly two models. Its strength is simplicity; its limit is that it cannot merge more than two parents in one step.

Where the shared-ancestor assumption bends

Real-world merges are looser than the theory. Community checkpoints often descend from the same base but went through different chat templates, continued pretraining, or vocabulary changes. Mergekit exposes tokenizer and chat-template choices precisely because these mismatches are common. The shared-basin argument is a good prior, not a guarantee, and it is the reason every merge needs an evaluation pass rather than an assumption of success.

The Algorithm Toolbox: From Averaging to DELLA

Having a mechanism, we can compare the methods. They sit on a spectrum from “no hyperparameters” to “many knobs that interact”.

Figure 2: The TIES pipeline (trim, elect sign, disjoint merge) and the DARE variant (random drop and rescale) that can feed the same sign-election stage. This is how mergekit’s dare_ties composes the two ideas.

Linear averaging and model soups

The simplest merge is a weighted average of weights. It needs no base model, since you average the full tensors. It is robust when parents are close, such as checkpoints from the same fine-tuning run with different seeds or learning rates, which is exactly the model-soups setting. When parents are specialised for different tasks, the average blurs both and typically underperforms more careful methods, but it is a cheap baseline you should always run.

Task arithmetic

Task arithmetic computes theta_0 + lambda * sum(w_i * tau_i). In mergekit this is task_arithmetic, with a per-model weight and a global lambda that defaults to 1.0, and it requires a base_model. The strength is interpretability: you can scale a skill up or down, or subtract one by using a negative weight. The weakness is that nothing reconciles conflicts, so quality degrades as you stack more experts.

TIES-Merging

TIES adds three steps, as the paper describes: trim parameters that changed little, elect a sign per parameter, and merge only the values agreeing with the elected sign. In mergekit ties takes per-model weight and density plus a global lambda. density is the fraction of each task vector’s largest-magnitude entries you keep, so density: 0.5 retains the top half. The practical lesson: lowering density sharply often does little harm, because of the redundancy discussed earlier, and it reduces overlap between experts.

DARE and dare_ties

DARE drops a random fraction p of delta parameters and rescales survivors by 1/(1-p), keeping the expected value of each delta unchanged. Because it is a random mask, the selection is unbiased with respect to magnitude, which differs from TIES’ magnitude trimming. The authors report that gains grow with model size and that they built a merged 7B model that ranked first among 7B models on the Open LLM Leaderboard at that time. Mergekit offers dare_linear and dare_ties; the latter applies DARE pruning and then TIES-style sign consensus. DARE merging has an important caveat: the “free lunch” depends on small deltas. Models fine-tuned heavily, such as with large-scale continued pretraining, have larger deltas, and aggressive dropping can break them.

DELLA and Model Breadcrumbs

DELLA (Deep, Bhardwaj and Poria, arXiv 2406.11617) replaces DARE’s uniform random drop with MAGPRUNE: it ranks parameters by magnitude and drops lower-magnitude ones with higher probability, then rescales. Across language, math and code experts they report averages 3.6 points above TIES and 1.2 above DARE on their benchmark set (AlpacaEval, GSM8K, MBPP); treat that as the authors’ own result on one setup. In mergekit the della method adds epsilon and requires density - epsilon > 0 and density + epsilon < 1; keep probabilities then range from density minus epsilon for the smallest-magnitude entries up to density plus epsilon for the largest.

Model Breadcrumbs (Davari and Belilovsky, arXiv 2312.06795, ECCV 2024) sparsifies by removing both outliers and negligible weights, and claims it needs no per-task hyperparameter retuning as tasks are added. Mergekit’s breadcrumbs uses density and gamma: with density 0.9 and gamma 0.01 it prunes the top 1% and bottom 9% by magnitude.

Other methods you will see in configs

The mergekit documentation also lists NuSLERP, Multi-SLERP, Karcher mean, SCE, Model Stock, Nearswap and Arcee Fusion. I have not benchmarked these for this article, so treat them as options to test, not recommendations. passthrough deserves separate mention: it takes exactly one model and copies layers unchanged, which is the building block for layer-stacking “frankenmerges” covered later.

A decision matrix

Situation	First method to try	Why
Two models, same task family	SLERP	One parameter, preserves norm
Same-run checkpoints or seeds	Linear soup	Parents already in same neighbourhood
3+ specialised experts	TIES or dare_ties	Handles sign and overlap interference
Need to remove a behaviour	Task arithmetic, negative weight	Directly subtracts a task vector
Many experts, noisy deltas	DELLA	Magnitude-aware dropping
Want separate experts retained	MoE merge	Keeps specialisation, costs memory

The table is a starting heuristic, not a verdict from a controlled study. Published comparisons disagree across tasks and model sizes, so the honest procedure is to run two or three candidates and measure.

Hands-On Tutorial: Merging with mergekit

Mergekit’s repository (github.com/arcee-ai/mergekit) is licensed LGPL-3.0, and its main entry point is mergekit-yaml config.yml output-dir, with optional flags such as --cuda and --lazy-unpickle. The tool streams tensors lazily, which is why merges of 7B models run in modest RAM.

Figure 3: The practical merge workflow. The evaluation gate is the step that most hobby merges skip, and it is where most of the value is decided.

Step 0: Install and choose parents

python -m venv .venv && source .venv/bin/activate
pip install mergekit            # or: pip install git+https://github.com/arcee-ai/mergekit.git
huggingface-cli login           # needed for gated bases

Pick parents deliberately. They must share an architecture and, ideally, the same base checkpoint and tokenizer. Read each model card for its base model, licence and chat template. In the examples below the names are placeholders; substitute real repositories you have verified.

Step 1: A SLERP merge of two models

SLERP requires a base_model and exactly two models in the merge. The t parameter can vary by layer and by module type, a trick the community popularised to bias attention and MLP blocks differently.

# slerp.yml
slices:
  - sources:
      - model: your-org/model-instruct-a
        layer_range: [0, 32]
      - model: your-org/model-code-b
        layer_range: [0, 32]
merge_method: slerp
base_model: your-org/model-instruct-a
parameters:
  t:
    - filter: self_attn
      value: [0, 0.5, 0.3, 0.7, 1]
    - filter: mlp
      value: [1, 0.5, 0.7, 0.3, 0]
    - value: 0.5
dtype: bfloat16

A list of values is interpolated across the layer range, so the first entry applies near the bottom of the stack and the last near the top. Here attention drifts from model A toward model B with depth while the MLPs do the opposite. Run it with:

mergekit-yaml slerp.yml ./merged-slerp --cuda --lazy-unpickle

Drop --cuda to merge on CPU; it is slower but works.

Step 2: A TIES or DARE-TIES merge of three experts

For three or more specialists, use a base model and per-model weight and density.

# dare_ties.yml
merge_method: dare_ties
base_model: your-org/base-7b
models:
  - model: your-org/base-7b-math
    parameters:
      weight: 0.4
      density: 0.5
  - model: your-org/base-7b-code
    parameters:
      weight: 0.35
      density: 0.5
  - model: your-org/base-7b-chat
    parameters:
      weight: 0.25
      density: 0.6
parameters:
  lambda: 1.0
dtype: bfloat16
tokenizer:
  source: base

Parameters can be set at four levels, with precedence running from the most specific (slices.*.sources.parameters) down to the global parameters catch-all. Swap merge_method to ties for deterministic magnitude trimming, or to della and add epsilon: 0.1 per model (with density chosen so that density minus epsilon stays positive and density plus epsilon stays below one).

Two practical notes. First, tokenizer: source: base pins the output tokenizer to the base model’s, avoiding mismatched vocabularies. Second, the exact semantics of options such as how weights are normalised differ between methods and versions; read the merge-methods page of the version you installed rather than trusting a blog post, including this one.

Step 3: Extract and merge LoRA adapters

Merging also applies to low-rank adapters. Two routes exist. The conventional route is to fold a LoRA into its base (peft‘s merge_and_unload) and treat the result as a full checkpoint. The second is that mergekit ships mergekit-extract-lora, which takes --model, --base-model and --out-path plus optional --max-rank, to approximate the difference between a fine-tuned model and its base as a LoRA. That inverts the usual direction: you can compress a full fine-tune into a portable adapter.

# fold adapters into a base with peft, then merge the full models with mergekit
from transformers import AutoModelForCausalLM
from peft import PeftModel
import torch

base = AutoModelForCausalLM.from_pretrained("your-org/base-7b", torch_dtype=torch.bfloat16)
model = PeftModel.from_pretrained(base, "your-org/lora-medical")
merged = model.merge_and_unload()
merged.save_pretrained("./base-7b-medical-merged")

Merging adapters directly has a subtlety: the product B times A of two LoRA factors is what changes the weights, and averaging A and B separately is not the same as averaging the products. Merge in the full-weight domain, or use tooling that does so explicitly.

Step 4: Evaluate honestly

A merge is only as good as its evaluation. Use a harness such as lm-evaluation-harness for standard benchmarks, but also keep a small private set of prompts drawn from your real workload, including multi-turn chat if you will serve chat. Compare against each parent and against the base. The decision rule is simple: the merge must beat the best parent on the tasks you care about, without a regression on safety refusals, formatting or long-context behaviour.

# minimal side-by-side sanity check across parents and merge
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

paths = {"parent_math": "your-org/base-7b-math",
         "parent_code": "your-org/base-7b-code",
         "merged": "./merged-dare-ties"}
prompts = ["Write a Python function that returns the nth Fibonacci number.",
           "If 3x + 7 = 25, what is x? Show your steps."]

for name, p in paths.items():
    tok = AutoTokenizer.from_pretrained(p)
    m = AutoModelForCausalLM.from_pretrained(p, torch_dtype=torch.bfloat16, device_map="auto")
    for q in prompts:
        ids = tok.apply_chat_template([{"role": "user", "content": q}],
                                      add_generation_prompt=True, return_tensors="pt").to(m.device)
        out = m.generate(ids, max_new_tokens=200, do_sample=False)
        print(name, "|", tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True)[:300])
    del m; torch.cuda.empty_cache()

Greedy decoding keeps the comparison deterministic. This is a smoke test, not a benchmark: it catches catastrophic breakage such as gibberish output or a broken chat template, then hands over to a proper harness.

Cost, in rough terms

A 7B merge in bfloat16 reads about 14 GB per parent and writes about 14 GB, so the work is dominated by disk and memory bandwidth rather than FLOPs. Expect minutes on a workstation, which is the economic point: each experiment costs far less than a LoRA training run, so you can try dozens. Evaluation, not merging, becomes the bottleneck. Budget accordingly: a full benchmark sweep on a 7B model can take longer than creating twenty candidate merges.

Beyond Weight Averaging: Evolution, Frankenmerges and Mixture-of-Experts

Evolutionary merging

Hand-tuning weights and densities is guesswork, and the search space is large: per-layer coefficients across several parents. Akiba et al. (Sakana AI, arXiv 2403.13187) propose automating it. Their approach searches merging recipes with an evolutionary algorithm and operates in both parameter space and data-flow space, needing no additional training data or compute beyond evaluating candidates. They report merging Japanese and math-capable models into a Japanese math LLM that achieves strong results on Japanese benchmarks, and a culturally aware Japanese vision-language model, and released the resulting models.

Figure 4: The evolutionary merging loop. Parameter-space recipes tune layer-wise mixing coefficients, while data-flow recipes decide which layers of which models a token passes through.

Two ideas are worth separating. Parameter-space evolution tunes the same coefficients you would write in YAML, but by optimiser rather than intuition. Data-flow-space evolution is stranger: it chooses the sequence of layers (possibly drawn from different models) that activations traverse at inference, which is a learned frankenmerge. I have not re-run these experiments, and I am describing the mechanism from the abstract and the method’s structure; consult the paper and Sakana’s released code for the specific optimiser and hyperparameters.

The operational lesson is about the fitness function. Evolutionary search is a very good optimiser and therefore a very good overfitter: whatever you score, it will chase. Use a development set for search, a disjoint set for selection, and a final untouched set for the claim you publish.

Passthrough and frankenmerges

The passthrough method copies layers without arithmetic, so a config can stack layer ranges from one or more models into a deeper network, for example repeating middle layers to make a larger model. Such “depth up-scaling” merges can produce models that score well on benchmarks, but behaviour is brittle: duplicated layers feed activations to places that never saw them in training. Treat these as research toys unless you follow with continued training, and verify long-context and instruction-following behaviour specifically.

Merging into a mixture of experts

If sparse parameter averaging loses specialisation, you can keep the experts separate. mergekit-moe builds a mixture-of-experts from dense models: the base model acts as donor for attention and layer norm parameters, while each source model’s MLP becomes an expert, and the output can be in Mixtral, DeepSeek-MoE or Qwen-MoE formats. Gates can be initialised three ways: hidden (default, built from hidden-state representations of positive and negative prompts for each expert, best quality but memory-hungry), cheap_embed (raw token embeddings only, markedly less effective but runs on small hardware), or random (intended for subsequent training, as in sparse upcycling).

The related research line is Branch-Train-MiX (Sukhbaatar et al., arXiv 2403.07816): copy a seed model, train copies as domain experts in parallel, then fold their feed-forward layers into MoE layers, average the rest, and fine-tune the router. That final routing stage is what dense merging lacks. An MoE merge costs more memory than any dense merge, since all experts must be resident, but each token activates only a subset, so compute is closer to a single parent.

What changes with synthetic data

Merging pairs well with synthetic-data pipelines. You can fine-tune cheap specialists on different filtered synthetic sets and merge them rather than mixing everything in one run, which avoids data-ratio tuning. The caveat from our piece on synthetic data, distillation and model collapse applies doubly: a merge inherits every flaw of its parents, and merging parents trained on the same synthetic source does not diversify them.

Trade-offs, Gotchas, and What Goes Wrong

Leaderboard contamination and Goodhart

The most documented failure of the merging scene is evaluation. Public leaderboards use public test sets. A merge of models, any of which saw benchmark data in training, can inherit contamination, and a search procedure (human or evolutionary) that picks winners by leaderboard score will select for it. Merged models that top leaderboards frequently show weaker real-world behaviour. I will not quote numbers here because I have not verified a specific contamination study for this article; the safe protocol is to rely on private held-out evaluations that mirror your workload.

Silent capability loss

Merges can look fine on average benchmarks and fail on tails: refusal behaviour, structured output, tool-calling formats, long-context retrieval, non-English text. Safety alignment is especially fragile. If one parent is an uncensored fine-tune, the merge can lose guardrails even when the other parent kept them. Always run your safety and format checks on the merged artefact, not on its parents.

Tokenizer, template and vocabulary mismatches

Parents with different vocabularies, added special tokens or resized embeddings will merge into garbage or fail loudly. Mergekit has flags like --allow-crimes and --unsafe-truncate-embeddings whose names are warnings. A chat template mismatch is quieter: weights merge, but the model was tuned to see a different prompt format, so quality degrades without error.

Interference grows with parent count

Every method degrades as you stack more experts. Sparsification postpones the problem rather than removing it. Past a handful of parents, expect diminishing returns and consider an MoE merge or ordinary multi-task training instead.

Licences and provenance

A merged model is a derivative of every parent. Licences such as Llama community terms, non-commercial research licences, and share-alike conditions combine, and the strictest effectively governs. Mergekit itself is LGPL-3.0, which concerns the tool, not your output. Record parents, versions and configs; for regulated work this lineage is your audit trail. This article is not legal advice.

When merging beats fine-tuning, and when it does not

Merging wins when you already have good specialists, want to combine them with zero training, need to explore many combinations cheaply, or cannot access training data (privacy, licensing). It loses when you need a genuinely new capability, when parents differ in base or tokenizer, or when guarantees matter more than convenience. A short fine-tune on a blended dataset is slower but usually more predictable.

Practical Recommendations

Start from the evaluation, not the merge. Decide which tasks the merged model must excel at, assemble a private held-out set for them, and measure every parent and the base before you touch mergekit. Without that baseline you cannot tell whether a merge helped.

Then run a ladder of cheap experiments. Begin with a linear average or SLERP as the floor. Move to ties or dare_ties for three or more experts, sweeping density between roughly 0.3 and 0.7 and weights in small steps. If results are noisy, try della. Only after hand-tuned merges plateau should you consider evolutionary search, and only with a separate selection set to resist overfitting.

Finally, treat the merge as a software artefact. Pin parents by commit hash, commit the YAML, and record the mergekit version, because parameter semantics evolve. Check licences before you release, and run safety and format regression tests on the final checkpoint.

A compact checklist:

Confirm identical architecture, shared base and compatible tokenizer.
Baseline every parent on a private evaluation set.
Run linear or SLERP, then ties or dare_ties; log each config.
Require the merge to beat the best parent on target tasks.
Test refusals, tool-call formats and long context explicitly.
Reconcile licences and store the lineage record.
Quantise only after the merge is validated; see our comparison of GGUF, AWQ, GPTQ and FP8 formats.

Frequently Asked Questions

What is model merging in LLMs?

Model merging combines the weights of two or more models with the same architecture into one model, without further training. Typical inputs are fine-tuned variants of a shared base. Methods range from simple averaging and SLERP to task arithmetic, TIES, DARE and DELLA, which operate on the difference between fine-tuned and base weights. The output is a single dense model with the same inference cost as each parent.

Does model merging require a GPU or training data?

Neither is strictly required. Mergekit streams tensors lazily and can run on CPU, where the cost is mostly memory bandwidth and disk I/O, and it uses no training data. A GPU can speed it up via the --cuda flag. Gate initialisation for mixture-of-experts merges in hidden mode does run prompts through the model and can need real VRAM. Evaluation of the result is where compute is actually spent.

What is the difference between TIES and DARE merging?

Both sparsify task vectors before combining them. TIES trims by magnitude, keeping the largest changes, then elects a sign per parameter and averages only agreeing values. DARE drops a random fraction p of deltas and rescales survivors by 1/(1-p). They compose: mergekit’s dare_ties applies DARE pruning followed by TIES sign consensus. Which wins depends on the models and tasks, so test both.

When should I use SLERP instead of linear averaging?

Use SLERP when merging exactly two models. It interpolates along the arc between weight vectors rather than the straight line, which better preserves the magnitude of the weights when they point in different directions. Mergekit’s SLERP needs a base model and a parameter t, where 0 returns the base and 1 the other model, and t can vary per layer and module. For similar checkpoints from one run, the difference from linear averaging is small.

Can I merge models with different architectures or tokenizers?

Generally no. Methods operating on parameters assume matching tensor shapes, and mismatched vocabularies break embeddings and output heads. Mergekit has override flags, but they are unsafe by design. The exceptions are structural: stacking layers with passthrough, or building a mixture of experts from dense models that share the base architecture. Cross-architecture knowledge transfer is a distillation problem, not a merging problem.

Is a merged model legal to publish and use commercially?

It depends on the parents. A merged model is derived from each parent, so the licences combine and the most restrictive terms typically bind. Non-commercial or custom community licences can block commercial use or require attribution and naming conventions. Read every parent’s licence, record the lineage, and consult counsel for commercial deployment. Mergekit’s own LGPL-3.0 licence applies to the software, not to the weights you produce.