Let's understand CoCurve — a way to compress large language models by pruning them smarter, with no fine-tuning at all. We'll build it up from scratch, assuming no background.
The whole idea is one shift in perspective: stop ranking parts one by one, and start looking at the connections between them.
First, the goal. A large language model is built from many small parts — attention heads and feed-forward channel groups — stacked in layers.
Structured pruning makes the model smaller and faster by deleting whole parts. Training-free means we do it with just a little calibration text — no retraining.
Because we remove whole heads and whole channel groups — not scattered individual weights — the model actually runs faster and lighter on real hardware.
We measure damage simply: how far do the pruned model's predictions drift from the original? Remove the parts you can drop with the least drift.
So which parts do we remove? Almost every method gives each part an importance score — on its own.
Then it assumes the total damage of removing a whole set is just the sum of the individual scores. Add them up, cut the cheapest.
It's a clean recipe. And it hides a wrong assumption.
Here's the problem. A Transformer is not a bag of independent parts. Every layer reads from and writes to one shared workspace — the residual stream.
So removing two parts together is not the same as removing each alone. They interact through that shared stream.
Two parts that each look important might be doing the same job — partly redundant. And two parts that each look weak might only work as a pair.
Those pairs are the trap. Call them bridge units: individually low-scoring, but highly connected to the parts that depend on them.
A method that scores each part alone will happily delete a bridge unit — right alongside the very units it was holding up. The damage isn't additive; it's joint.
And you can't fix this by correlating activations — that signal is dominated by raw scale and disconnected from what the model actually outputs.
CoCurve fixes it at the source. Start from that same measure of drift — the divergence between the model and its pruned copy.
At zero pruning there is no drift and no slope, so the leading behavior is pure curvature — a single matrix that captures how damage grows.
And this one matrix has two readings. Its diagonal is exactly the classical importance score — how much each part matters on its own.
Its off-diagonal entries are something new: co-pruning curvature edges — the extra damage from removing two parts together.
Positive means the pair is jointly harmful to cut; negative means they cancel; zero means independent. And on real models, around 98 percent of the mass lives in these edges — not the diagonal everyone scores.
But measuring every pair sounds impossibly expensive — there are millions of them.
Here's the key result. In the right geometry, the entire edge matrix is a Gram product — a table of dot products — of single-part measurements.
So you only need to test each part once, on its own. From those single-part ablations, the whole web of edges falls out — with no pairwise sweeps, no gradients, and no labels.
It's the same information a full pairwise analysis would give you — at the cost of the cheapest possible experiment.
With the full matrix in hand, CoCurve prunes in a single shot. A greedy solver removes parts under one shared budget across attention and feed-forward together.
Because attention and FFN compete in the same budget, the method decides on its own how much to take from each — no hand-set ratios.
And a single dial, chosen from calibration alone, sets how much to trust the edges — from pure node saliency, up to the full edge model.
So does it work? On an instruction-tuned Llama-3.1-8B, at 20 percent pruning, here is how much of the original capability survives.
Language modeling, commonsense, knowledge — CoCurve leads the strongest baseline on all of them. It is the only training-free method strong across every capability class.
The clearest win is code. Code generation is fragile — most pruning methods wipe it out. CoCurve keeps 16 percent of it, against 11 for the best baseline.
And it is not one model. CoCurve gives the lowest perplexity on every model we tested — up to about fourteen times lower than the best baseline on a 24-billion-parameter model.
And we can prove it's the edges doing the work. Turn them off — keep only the diagonal, the classical score — and perplexity jumps to about 13.
Turn the edges back on, and it drops to 8, cutting the gap to the original model roughly in half. Code recovers about twenty-fold.
Even sharper: keep only the edges within a module, and you do worse than the diagonal alone. It is specifically the attention-to-feed-forward edges — the cross-module ones — that carry the model.
CoCurve even tells you in advance when the edges will help. The benefit grows when a model's feed-forward channels are less redundant — a number you read straight off the calibration matrix, before any benchmark.
So on low-redundancy models it leans fully on the edges; on redundant ones it damps back to a strong classical score. It never bets on edges where they won't pay off.
It's honest about limits, too. Like all training-free pruning, it targets moderate compression — past about thirty percent, everything collapses without a recovery stage.
So the takeaway is a change of unit. Importance is not a property of a part in isolation — it lives in the connections between parts.
The thing almost everyone scores — the diagonal — is a few percent of what matters. The rest is in the edges.
CoCurve reads both from a single matrix, recovers the edges cheaply, and prunes them together. Prune the edges — not just the nodes.
The project page, this tutorial, and the code and paper — coming soon — are linked on screen. Thanks for watching.