AAAI 2026 · Under Review · Training-Free Structured Pruning

AnchorPath

Defeating the high-sparsity collapse of structured LLM pruning with a calibration-only anchored continuation — separating a fixed dense anchor from a moving expansion point, with zero weight updates.

Zhiren Gong^1,2, Zihao Zeng¹, Zijie Wang¹, Ming Xiao⁴, Chau Yuen³, Wei Yang Bryan Lim¹

¹ College of Computing and Data Science, Nanyang Technological University ² Interdisciplinary Graduate Programme, Nanyang Technological University ³ School of Electrical and Electronic Engineering, Nanyang Technological University ⁴ School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology

At 50% sparsity on LLaMA-3.1-8B, one-shot second-order pruning explodes to 4106 perplexity; AnchorPath holds it at 245 (16.8× lower) — and up to ~270× on Yi-1.5-9B — while changing only the mask.

Paper · coming soon Code · coming soon ▶ Tutorial

Headline degradation curve: one-shot vs AnchorPath perplexity across sparsity — Figure. One-shot collapses super-exponentially with sparsity; AnchorPath controls it — with no weight updates.

7

Models across 5 families

10–50%

Structured sparsity range

16.8× · up to ~270×

Lower perplexity vs one-shot at 50%

K = 1 ≡ one-shot

Exact ablation of our own method

0 weight updates

Only the mask changes

Abstract

Training-free structured pruning removes whole attention heads and FFN channel groups from a frozen LLM using only a small unlabeled calibration set — no gradients, no fine-tuning. Yet every one-shot method collapses at high sparsity.

We show this collapse is a property of the one-shot paradigm itself — a single second-order quadratic built at the dense model and extrapolated across the entire budget — not of the importance score. AnchorPath recasts compression as a calibration-only anchored continuation: it traces the pruning path in the cost budget, re-estimating local curvature at the current partially-pruned model while measuring damage against a fixed dense anchor.

Across seven models, AnchorPath is the lower envelope of the training-free baseline set and cuts perplexity by up to two orders of magnitude versus one-shot at 50% sparsity — while changing only the mask. One-shot second-order pruning is exactly the K = 1 case.

The Problem · Why One-Shot Collapses

One map, extrapolated too far

One-shot second-order pruning builds a single quadratic R(s) ≈ ½·sᵀH·s of the KL-to-teacher risk, measured at the dense model, then minimizes it over the whole budget. The map is faithful only near the dense point — deleting 40–50% of units carries you far outside its trust region, and neglected higher-order terms grow as the cube of the step.

A super-exponential signature

On LLaMA-2-7B, one-shot WikiText-2 perplexity runs 6.4 → 9.3 → 19.9 → 92.6 → 1497 at 10–50% — per-step inflation ×1.5, ×2.1, ×4.7, ×16.2 that accelerates. Two coupled causes: trust-region violation and operating-point drift — both about where the expansion is taken, so no importance score escapes it.

One-shot fits one quadratic at the dense point; AnchorPath re-anchors over K steps — Figure. (a) One-shot: anchor = expansion point, extrapolated → collapse. (b) AnchorPath: fixed anchor, moving expansion point over K steps — errors add, not multiply.

The Key Separation

One-shot conflates two roles that AnchorPath pulls apart. Separating them revives a first-order drift term that one-shot discards by construction.

Anchor — fixed

The dense teacher p₀ against which all damage is measured.
Held fixed for the entire budget, so "how far have we drifted from dense" is always well-defined.
This is what keeps the objective a clean self-distillation KL, not a moving target.

Expansion point — moving

Where the risk is re-linearized: the current partially-pruned model sₖ.
Re-estimating curvature here revives a drift term g⁽ᵏ⁾ that is zero at the dense point.
A unit that aligns with accumulated damage becomes expensive — even if it looked cheap at dense.

Method

Compression as an Anchored Continuation (calibration-only)

Instead of one quadratic spanning the whole budget, AnchorPath splits the budget into K bands and traces the constrained-optimal pruning path, re-estimating local curvature at each operating point. It is a continuation in the compression budget whose reference anchor stays fixed at the dense teacher. Nothing but the mask ever changes — no gradient descent, no reconstruction, no LoRA.

Operating-point curvature

At path point sₖ, model the extra risk of removing more as g⁽ᵏ⁾·δ + ½·δᵀH⁽ᵏ⁾·δ.
Both terms evaluated at the pruned point but measured against the fixed anchor.
g⁽ᵏ⁾ couples accumulated drift with a candidate's perturbation — the term one-shot cannot see.

K-step predictor–corrector loop

Estimate g⁽ᵏ⁾, H⁽ᵏ⁾ at the current model.
Fill the band by cost-normalized greedy selection under the fresh curvature.
Commit: physically realize the mask, recompute the drift, advance.

Why it works — errors add, not multiply

One-shot's extrapolation error is a cubic that compounds into super-exponential collapse.
K re-linearized steps convert that compounding error into an additive one (an idealized 1/K² mechanism).
Interaction-flagged refresh re-ablates only the stalest survivors each step → offline cost ≈ 4.5× one-shot at K=8, and the first step is served from cache, preserving exact K=1 equivalence.
Because only the mask changes, the gain is fully inspectable and the deploy cost is identical to one-shot.

Step 1 · Prepare

Cache the frozen dense teacher and step-0 curvature features from a small calibration set.

Output: teacher cache + the exact one-shot (K=1) starting point.

Step 2 · Continue

Trace the budget in K bands: estimate operating-point curvature, greedily fill each band, realize the mask.

Output: a path of masks that stays inside each fresh trust region.

Step 3 · Deploy

Ship the final mask — structural units are physically removed. No recovery, no weight edits.

Result: a smaller model at the same deploy cost as one-shot, but far higher quality.

Budget homotopy geometry: one-shot extrapolates one quadratic, AnchorPath re-estimates — Figure. The budget homotopy: one-shot's single quadratic leaves its trust region; AnchorPath re-estimates at K=8 operating points and tracks the lower frontier.

Dense Fisher-Gram coupling matrix — Figure. The dense curvature is strongly cross-coupled — 27% of energy off-diagonal, 89% of unit pairs coupled — so a single dense-point quadratic drifts, and refresh is needed.

Main Results

LLaMA-3.1-8B · 50%

One-shot 4106 → AnchorPath 245

16.8× lower · 2.9× below the best baseline (LLM-Pruner 718).

Yi-1.5-9B · 50%

One-shot 16510 → AnchorPath 61.7

~270× lower — nearly two orders of magnitude.

Clean sweep

Best in 28/28 cells vs one-shot; ≥ LLM-Pruner on 24/24.

The lower envelope of every training-free baseline at 50% on all seven models.

Main Degradation Table — LLaMA-3.1-8B, WikiText-2 Perplexity (no recovery)

Hint: scroll horizontally to view all columns on smaller screens.

Method	ρ=0.2	ρ=0.3	ρ=0.4	ρ=0.5
Random	22.7	56.6	4168	60360
Magnitude	36.7	2184	27101	76313
Wanda-sp	29.4	101.4	1015	17611
FLAP	37.6	1030	49940	376017
LLM-Pruner	14.5	30.0	103.3	718.2
One-shot (K=1)	14.0	33.0	245.9	4106
AnchorPath (K=8)	12.1	23.2	47.2	245
Gain vs one-shot	1.2×	1.4×	5.2×	16.8×

Cross-Model — One-shot vs AnchorPath at 50% Sparsity (WikiText-2)

Model	One-shot (K=1)	AnchorPath	Gain
Yi-1.5-9B	16510	61.7	~270×
LLaMA-3.1-8B	4106	245	16.8×
LLaMA-3.2-3B	3027	325	9.3×
LLaMA-2-7B	1497	389	3.9×
Qwen2.5-14B	300.8	156	1.9×
Qwen2.5-7B	270.1	141	1.9×
Falcon3-7B	62.5	27.8	2.2×

AnchorPath is the lower envelope of all training-free baselines — Figure. AnchorPath is the lower envelope of the full training-free degradation curve (LLaMA-3.1-8B).

Reading the results

The gain grows with sparsity — exactly where one-shot's extrapolation breaks down.
Zero-shot accuracy retention improves too: LLaMA-2-7B 7-task average at 50% is .414 vs one-shot .367 and LLM-Pruner .388 (dense .595).
At the extended frontier the separation widens further — up to 24.5× / 119× over one-shot at 60% sparsity.
Robust to calibration resampling (±4%) and holds out-of-distribution (PTB, 8–88× over one-shot).

Efficiency & Ablation

A pure quality gain at zero deploy overhead

Offline pruning cost ≈ 4.5× one-shot at K=8, τ=0.5 — the first step is served from cache.
Deployment is just a mask: identical cost to a one-shot model at the same sparsity.
LLaMA-2-7B at 50%: 6.74B → 3.90B params, 1.6× prefill speedup, 36% lower peak memory.
LLaMA-3.1-8B at 50%: 8.03B → 4.53B params, 1.78× prefill speedup.

Effect of K (LLaMA-2-7B, 50%)

Steps K	1	2	4	8
WikiText-2 ppl	1497	1205	823	389

Monotone, diminishing returns — a 3.9× reduction from re-centering alone. K never hurts; K=1 recovers one-shot exactly (Jaccard 1.0).

Perplexity monotone-decreasing in K; schedule study — Figure. Perplexity decreases monotonically in K; a front-loaded schedule helps further where collapse is sharpest.

Per-step inflation accelerates for one-shot; drift mass grows along the path — Figure. One-shot's per-step inflation accelerates (×1.5→×16); the revived first-order drift mass is zero at dense and grows along the path.

Ablation build-up (LLaMA-2-7B, 50%)

One-shot 1497 → multi-step (K=8) 389 → front-loaded schedule 126 (11.9× total on the sharpest-collapse model). Removing the drift term g regresses 389 → 668 — and is inert at low sparsity, mattering exactly where drift mass is large. Equal-cost bands are the universal default; the safety rails are conservative, not the source of the gain.

White-Box Analysis

What actually changes — and why it's inspectable

Because only the mask changes, the improvement is fully traceable. AnchorPath prunes many attention heads — 47 / 71 / 156 heads at 20 / 30 / 50% sparsity — where one-shot prunes at most one at every ratio. Those heads look cheap only from the dense point; the anchored continuation sees the damage they cause once the model has already moved.

On the perplexity–accuracy plane, the front-loaded schedule reaches BoolQ 0.676 > one-shot 0.629 while also lowering perplexity — one-shot is a strictly dominated point.

White-box: heads pruned and perplexity-accuracy plane — Figure. (a) AnchorPath removes attention heads one-shot never touches; (b) it reaches a favorable perplexity–BoolQ corner that dominates one-shot.

Honest scope

No calibration-only pruner is deploy-as-is far beyond moderate sparsity (~40%); AnchorPath's role there is to hand the most recoverable mask to a downstream recovery stage.
Generation tasks (GSM8K, HumanEval) collapse for every calibration-only pruner — restored only by orthogonal recovery, not by selection.
Gains are largest on distribution-aligned, perplexity-style objectives; logit-reshaping architectures need an adapted anchor.
The paper deliberately isolates selection; coupling with fine-tuning is composable future work.

Resources

Paper

AAAI 2026 submission — under review. Coming soon.

Code

Reference implementation and one-command reproduction. Coming soon.

Tutorial

A narrated, animated ~7-minute video tour — the collapse, the anchored continuation, and the results, built for a general audience.

Contact

For questions about this project, contact zhiren001@e.ntu.edu.sg.