Let's look at a practical question. When you fold many specialized models into one, how much should you spend to do it well?
The real story here isn't a single method — it's a budget. Sometimes a cheap, one-shot merge is the right call; sometimes it pays to spend real computation. We'll see exactly when.
Start simple. You take one pre-trained model and fine-tune it separately for many tasks — an expert for everyday photos, one for satellite images, one for digits, and so on.
They all began from the same checkpoint; they only drifted apart during fine-tuning. That shared origin is what makes combining them possible.
Serving a whole zoo of experts is expensive, and you often can't retrain one model on everything — the data may be private, or simply gone.
Model consolidation is the fix: fold those experts back into a single model — same size, same cost — that handles every task, with no joint retraining.
There are two very different ways to do this, and they cost wildly different amounts.
The first is closed-form merging. You collect a few statistics from each expert and solve one equation. Methods like RegMean finish in about a minute. Cheap — but it only approximates what you want.
The second is end-to-end distillation — our E2E-Merge. It repeatedly runs the frozen experts and trains the merged model to reproduce their behavior. Far more expensive, but it optimizes the goal directly.
So which should you use? That's the whole question — and the honest answer is: it depends on your budget.
Here's the insight that ties them together. Both regimes are chasing the exact same target.
For every task, on that task's own inputs, the merged model should produce the same internal representation the expert would. Write that down, and you get one clean objective.
No labels are needed — the target is the expert's own output. Closed-form merging approximates this objective with a formula; distillation optimizes it directly with gradient descent.
One goal, two ways to reach it — cheap and approximate, or costly and exact.
And that's exactly where the trade-off lives. The cheap closed-form solution pays a fixed price: because it only approximates, it leaves a permanent gap.
Distillation can close that gap — but only if you feed it enough data and enough training steps. With too little of either, it actually does worse than the cheap merge.
So there is no universal winner. There is a frontier: closed-form is best under tight budgets; distillation pulls ahead once you can afford it. Let's watch that along three axes — data, compute, and memory.
First, data. How many unlabeled examples per task do you have?
With very few — say sixty-four or a hundred and twenty-eight — the cheap RegMean merge is simply better. Distillation doesn't have enough to learn from.
But the lines cross at around two hundred and fifty-six examples. Past that, distillation pulls ahead and keeps climbing — to about eighty-eight and a half percent with a thousand examples per task.
The lesson: below a couple hundred examples, don't bother with the expensive method — the closed form wins.
Second, compute — how many training steps you're willing to run.
Almost all of the benefit arrives early. Just four thousand steps recover about eighty-three percent of the total gain, jumping from eighty-four to over eighty-nine.
By eight thousand steps you're ninety-nine percent of the way there. The final stretch to twelve thousand adds only a tenth of a point.
So distillation is a diminishing-returns game — a modest compute budget already captures most of the win.
Third, memory. The distillation can run block by block, trading peak memory for accuracy.
Merge just one block at a time and you use under four gigabytes, already reaching eighty-five percent — better than the cheap baseline.
Group more blocks together and both memory and accuracy climb — up to the full run at about ninety percent and eight gigabytes.
So even on a modest GPU, you can pick the point on this curve that fits your hardware.
Put it together, and you get a simple decision rule.
Use the cheap closed-form merge when data, compute, or memory is tight — or when its approximation is already good enough.
Use end-to-end distillation when you have enough unlabeled data and offline compute, and the gap to close is large — many experts, or a weaker backbone.
And to be honest about it: on a language benchmark with a different setup, the cheap RegMean method actually beats distillation. There really is no universal winner — only the right tool for the budget.
So what happens at the generous end of the budget, where distillation gets everything it needs?
On eight image tasks with a large vision model, it reaches ninety-four point three percent — essentially the accuracy of the eight separately fine-tuned experts, in a single model.
And the harder the problem, the more it pays. Push from eight tasks to twenty, and distillation beats the best cheap baseline by nine and a half points.
The advantage grows with the number of experts, and on weaker backbones — precisely the settings where a one-shot formula struggles most.
Why does spending that compute help? Because small errors at each layer compound through depth.
Two methods can look equally accurate layer by layer, yet one ends up more than twice as far from the expert at the output. Only an end-to-end objective sees, and fixes, that final gap.
A fair warning on scope: this is mostly a vision study, all experts share one pre-trained initialization, and the comparison changes several things at once. The break-even points are guidance, not guarantees.
So here's the takeaway. Closed-form merging and end-to-end distillation aren't rivals — they're two budget regimes for the same goal.
Match each expert's behavior. Then choose how hard to work at it by what you can afford — data, compute, and memory.
The result isn't a universal winner. It's a practical frontier: spend little and get a solid merge, or spend more and approach the original experts.
The paper and code are coming soon, linked alongside this video. Thanks for watching.