E2E-Merge Resources

Under Review · Model Merging

Model Consolidation
under Budget

From closed-form merging to end-to-end distillation. Both fold task experts into one model by optimizing the same functional objective — so the real question isn't which is better, but when each is worth its cost.

Zhiren Gong, Tiantong Wang, Zihao Zeng, Yiyang Duan, Chau Yuen, Wei Yang Bryan Lim

College of Computing and Data Science, Nanyang Technological University

A cheap closed-form merge wins under tight data / compute / memory; end-to-end distillation (E2E-Merge) pays off once resources are ample and the structural gap is large — reaching 94.3% on ViT-L/14, essentially the fine-tuned experts. Not a universal winner — a frontier.

Paper · coming soon Code · coming soon ▶ Tutorial
Budget frontier: data, iteration, and memory
Figure. The budget frontier — along data, iteration, and memory, the cheaper closed-form merge and end-to-end distillation trade places.

2 regimes

Closed-form merging vs end-to-end distillation

N = 256

Data crossover: distillation overtakes here

4k steps → 83%

Of the total distillation gain, then it saturates

3.7 → 8 GB

Block-wise memory frontier: 85.3% → 90.2%

up to +9.5

Distillation's edge at the full-budget endpoint (20 tasks)

Abstract

Data-assisted model consolidation comes in two computational regimes: closed-form merging, which cheaply solves a parameter- or layer-local surrogate, and end-to-end distillation, which spends iterative computation to match expert behavior. We study when each regime is preferable rather than treating one as universally superior.

Both are organized around one task-conditioned functional objective: on inputs from task a, one shared model should reproduce expert a. Our analysis separates the fixed surrogate gap of closed-form regression from the data- and compute-dependent errors of distillation. Locally, RegMean is the exact optimizer of an expert-centered, layer-wise, isotropic surrogate; dropping cross-layer coupling incurs an exact quadratic excess loss. Under smoothness and Polyak–Łojasiewicz conditions, iterative refinement reduces the functional error with compute and admits a break-even budget.

Existing experiments map three slices of the resulting frontier — across data, compute, and memory. The evidence supports a regime view: closed-form merging is preferable under severe constraints and a tight surrogate; end-to-end distillation pays when unlabeled data and offline computation are available and the structural gap is large.

The Setup · Two Regimes

Closed-form merging — cheap

Collect a few statistics and solve once. RegMean / RegMean++ regress per-layer activations in closed form — on the order of a minute after one pass. Negligible optimization error, but a fixed structural gap: it only approximates the objective.

End-to-end distillation — costly

E2E-Merge repeatedly queries the frozen experts and, on their own unlabeled inputs, makes one shared student reproduce each expert's representation. It samples the objective directly — but pays finite-data and optimization error, and real compute.

Same goal, different cost

Model consolidation folds several experts — fine-tuned from a shared checkpoint — into one model, with no joint retraining. Both regimes chase the same thing; they just spend a different budget to get there. So the useful question is the frontier: how much accuracy each buys per unit of data, compute, and memory.

The Unifying Objective

On each task's inputs, reproduce its expert

For merged encoder g_θ and experts g_{θₐ}:

F(θ)  =  Σa   πa   Ex∼Da   ‖ gθ(x) − gθa(x) ‖²

Closed-form approximates F

  • RegMean is the exact optimizer of a layer-wise, first-order, isotropic surrogate of F.
  • Fisher merging is its diagonal analogue; averaging ignores the geometry entirely.
  • Dropping cross-layer coupling costs an exact quadratic excess — a fixed structural gap.

Distillation samples F

  • E2E-Merge minimizes F directly by gradient descent, from a task-arithmetic init, teachers frozen.
  • Under smoothness + PL, error contracts geometrically — with a computable break-even budget.
  • Distillation beats the closed-form reference once 2εₙ + δ_K < Δ_surrogate.

Representation matching supervises the full shared representation; logit distillation only controls directions the task heads can see — a separate design axis.

The Budget Frontier · Three Slices

Data budget

Closed-form wins at 64–128 examples/task; distillation overtakes at 256+.

256 ex: 86.1 (E2E) vs 84.4 (RegMean++)

Compute budget

4k steps recover 83% of the gain (89.2); it saturates by 8k (90.1 → 90.2).

RegMean++ reference: 84.2

Memory budget

Block-wise merge spans 85.3%@3.7GB → 89.8%@5.4GB → 90.2%@~8GB.

even one block beats RegMean++

Data-budget crossover — ViT-B/32, average accuracy (%)

Distinct examples / task641282565121024
RegMean++ (closed-form)83.383.984.484.384.5
E2E-Merge (distillation)81.383.586.187.788.5

The ordering reverses at 256 examples — below it, the cheap merge is simply the better bet.

Data, iteration, and memory budget frontiers
Figure. Three one-dimensional slices of the frontier — data, iteration, and memory — from the same ViT-B/32 measurements.

When Each Wins

Use closed-form merging when…

  • data is scarce (≲ a couple hundred examples/task), or
  • compute / memory is tight, or
  • the local surrogate is already tight (small structural gap).

A counter-example that proves the point: on a Flan-T5 GLUE probe, closed-form RegMean (83.0) beats iterative E2E (81.9) — different functional readouts favor the cheap regime.

Use end-to-end distillation when…

  • you have enough unlabeled data (≳ 256 examples/task) and offline compute, and
  • the structural gap is large — many experts, or a weaker backbone.

The distillation edge grows with expert count — +2.6 → +6.0 from 2 to 8 experts, and +13.7 at 20 — and on weaker backbones.

Full-budget endpoints — best closed-form reference vs E2E-Merge (avg accuracy %)

SettingClosed-form refE2E-MergeΔ
ViT-B/32 · 8 tasks84.2 (RegMean++)89.8+5.6
ViT-B/16 · 8 tasks87.3 (RegMean++)92.0+4.7
ViT-L/14 · 8 tasks90.9 (RegMean++)94.3+3.3
ViT-L/14 · 20 tasks83.4 (Iso-C)92.9+9.5

At the full budget, the merged model reaches the individual fine-tuned accuracy on ViT-L/14 (94.3 ≈ 94.3).

Why · Mechanism

Layer-wise residuals do not determine final error
Figure. Layer-wise residuals are nearly tied (2.46 vs 2.11), yet the final representation error differs 2.4× (0.55 vs 0.23) — and that ordering matches accuracy.

Per-layer optimality ≠ end-to-end optimality

A closed-form solver can be near-optimal at every layer and still land far from each expert at the output, because small residuals compound through depth. Distillation controls what actually reaches the output — which is exactly what its extra compute buys, and why the gap widens with more experts and weaker backbones.

Efficiency, honestly

We report cost as a resource vector (data, updates, memory, time) with Pareto dominance — not a fragile accuracy-per-minute scalar. The point is the frontier, not a single winner.

Distillation gain grows with expert count
Figure. The distillation advantage over closed-form grows with the number of merged experts.

Honest scope

  • A representative, not exhaustive comparison: RegMean++-vs-E2E changes target, trajectory, solver and compute at once — it measures an overall regime gap, not one isolated component.
  • CLIP-centric; all experts share a pre-trained init. The Flan-T5 probe uses different functional readouts.
  • Break-even results are conditional local bounds, not AdamW convergence guarantees. The method is label-free but not data-free.

Resources

Paper

Under review. Coming soon.

Code

Reference implementation and reproduction. Coming soon.

Tutorial

A narrated, animated ~7-minute video tour — the two regimes, the shared objective, and the budget frontier, built for a general audience.