Let's take a guided tour of SubspacePath Pruner — a way to specialize a large language model for a single deployment scenario, at inference time, with no scenario-specific training.
We'll build it up from the problem, to the idea, to the evidence.
Start with how these models are actually used.
A single frozen model is often invoked again and again inside one narrow context — a domain, a task, a scenario.
But it still carries the full weight of every capability it was ever trained for. For any one scenario, most of that is simply idle.
The natural fix is pruning — remove the components you don't need. But standard pruning ranks importance once, globally, and produces a single fixed compressed model.
That ranking is an average over everything. Under a specific scenario — especially an unfamiliar one — the wrong parts get cut, and accuracy turns brittle.
The alternatives cost too much. Retraining per scenario needs data and compute you may not have; routers and mixtures add parameters and runtime complexity.
So the real question is this: can we extract a scenario-specific model from a frozen base, robustly, without any scenario-specific training data?
Our answer rests on one observation, which we call subspace-pathway coupling.
In embedding space, inputs from a domain occupy a compact region — a representation subspace we call an axis.
In parameter space, the model computes through attention heads. The specific heads that carry a behavior form a pathway.
The coupling is this: inputs that share a subspace repeatedly activate the same small, stable set of head pathways — and different domains use partially separable ones.
If that holds, then locating a scenario's subspace tells us which heads it actually needs. Pruning becomes a lookup, not a guess.
SubspacePath turns this into two modules: Domain-Basis Synthesis, and Probe-based Scenario Pruning.
First, D-B-S builds the coordinate system.
From input-only pools of text, with stopwords removed, we embed each domain and reduce it to a shared low-dimensional space.
Each domain becomes an axis — the direction its inputs point along.
We then select a subset of domains that are as orthogonal as possible while still covering the semantic space, balancing separation against coverage.
The result is a stable, near-orthogonal basis — here, six domains with pairwise orthogonality above zero point seven seven — shared across every backbone.
Second, P-S-P bridges those axes to the model's heads. It has an offline stage, prepared just once.
For every layer, we train a lightweight linear probe to read how strongly a hidden state aligns with each domain axis.
For every head, we measure its residual write-back — what it adds to the stream — and how much of that energy points along each axis. That gives a head-importance score for each domain.
Some heads matter everywhere, for every domain. We find these domain-invariant backbone heads — confirmed by a Mann-Whitney test at p below zero point zero zero one — and protect them as a whitelist.
All of this is cached. It never runs again at deployment.
Then the online stage — run once per scenario, on the frozen model.
We read the first few inputs, run the calibrated probes, and estimate the scenario's domain mixture.
The entropy of that mixture gives the scenario's breadth — narrow, or broadly cross-domain — which sets how many heads to keep.
We combine the mixture with the cached importance to score every head, always keeping the whitelist, and retain the top heads under budget.
That yields one head mask — compiled once, then reused for every turn in the scenario. No optimization, no retraining.
Now the evidence. We test on XDomainBench — selected, out-of-domain, and cross-domain splits — plus cross-dataset transfer to CommonsenseQA, Natural Questions, and ARC.
On Qwen-2.5-14B under moderate pruning, recall reaches 47.8, 44.1, and 31.3 — against a dense model's 40.9, 37.2, and 22.8. The pruned model beats the full one.
On LLaMA-2-13B, it reaches 43.0, 32.5, and 20.2, versus 29.6, 26.1, and 18.4 dense — outperforming Wanda, R-I-A, DaSS, LLM-Pruner, and probe pruning at the same budget.
And the gains are largest exactly where global pruning is weakest — out-of-domain and cross-domain — where scenario-conditioned masks cut the interference between competing pathways.
Even under aggressive pruning, recall stays above the dense model on all three splits, because the method reorganizes pathways rather than just deleting parameters.
It's also efficient by construction. The heavy work is offline; online compilation takes just twenty-seven to sixty-eight milliseconds — well under a tenth of a second.
Inference speedups reach one-point-four to over three times at light pruning, and the single compiled mask is reused across the whole multi-turn scenario.
Is every part necessary? Yes. Replace the selected axes with random ones, and performance collapses to near zero.
Remove the whitelist, and out-of-domain recall falls off a cliff. Remove multi-domain mixing, and cross-dataset accuracy drops sharply. Each piece is load-bearing.
The paper is honest about scope: gains are strongest at moderate pruning, reasoning-heavy tasks are more sensitive, and results span four models from seven to fourteen billion parameters — not yet the largest.
But the core message is simple. A component's importance is not global — it is conditional on the scenario.
Find the subspace, and the pathway follows. Compile it once, reuse it, and specialize a frozen model with no training at all.
The code, the paper, and an interactive project page are linked on screen.