The Surprising Effectiveness of Deleting Weights in LLM Post-Training

Jack Lu¹^*, Zhenbang Yang¹^*, Mike Lasby², Tejas Pote², Yani Ioannou², and Mengye Ren¹

¹New York University, ²University of Calgary
^*Equal contribution

Advances in Neural Information Processing Systems 39 (NeurIPS 2026)

arXiv PDF Poster Code

TL;DR: Setting fewer than 0.05% of a pretrained LLM's weights to zero, with no other change, matches full fine-tuning at scale across reinforcement learning, supervised fine-tuning, and on-policy distillation.

Abstract

How little of a pretrained large language model (LLM) has to change for it to acquire new capabilities? We find that zeroing fewer than 0.05% of its weights, with no other modification, matches full fine-tuning (FFT) at scale across three LLM post-training regimes: reinforcement learning with verifiable rewards, supervised fine-tuning, and on-policy distillation. Our method, Deletion Tuning, learns a binary mask over the gate projection of transformer blocks. Two practical advantages follow: the trained mask is hundreds to thousands of times smaller than the corresponding FFT checkpoint, and Deletion Tuning forgets far less, which addresses a central challenge in continual learning. Under reinforcement learning, its prior-task accuracy stays near the base level while FFT, LoRA, and random sparse fine-tuning lose up to 5.4 points. To understand why masking alone suffices, we analyze the geometry of the learned weight changes under reinforcement learning using stable rank, spectrum drift, and principal-weight overlap. Compared with LoRA and random sparse fine-tuning, Deletion Tuning is closest to FFT across the three measures.

Overview

Deletion Tuning learns which pretrained weights to delete and writes no new values. The pretrained weights $W$ stay frozen, and each is paired with a trainable logit in $G$, the only parameters that post-training updates. During training, each weight is multiplied by $\sigma(\alpha G)$, a smooth mask between 0 and 1. The sharpness $\alpha$ rises over training until the sigmoid becomes a hard threshold at $G = 0$. At deployment, weights with positive logits (blue) keep their pretrained values and weights with negative logits (red) are set to zero (crossed out), so the trained adapter is one bit per weight.

Full fine-tuning (FFT), LoRA, and sparse fine-tuning all write new values into a pretrained model. We introduce Deletion Tuning to test whether deleting pretrained weights alone can match FFT.

Deletion alone matches full fine-tuning. Deletion Tuning is on par with FFT at the largest scale we test in GRPO, SFT, and on-policy distillation (SDFT), up to a 30B mixture-of-experts model. Under GRPO, it also solves problems that the base model fails in all 128 sampled attempts.
Deletion Tuning is compact and forgets less. Its masks are orders of magnitude smaller than FFT checkpoints in all three regimes, and it keeps prior-task accuracy throughout training.
Deletion Tuning's learned updates have FFT-like geometry. Its weight updates resemble FFT's on three geometric measures, which helps explain its accuracy.

Deletion Tuning

Deletion Tuning freezes the pretrained model and gives every weight of each gate projection its own trainable logit. During training, the weight matrix $W$ is multiplied elementwise by the soft mask $\sigma(\alpha G)$, an annealed sigmoid of the logits $G$ that lets gradients reach them:

$$ \widetilde{W} = W \odot \sigma(\alpha G). $$

The sharpness $\alpha$ rises linearly over training, so the soft mask approaches the hard mask that Deletion Tuning deploys, a binary mask that keeps each pretrained weight with a positive logit and deletes the rest:

$$ \widehat{W} = W \odot \mathbf{1}[G > 0]. $$

We initialize the logits from Wanda pruning scores and set no sparsity target, so training decides how many weights to delete. Bit-packed and compressed with zstd, the hard mask takes only 1.0 MB for Llama1B and 8.4 MB for Llama8B, whose fully fine-tuned checkpoint takes 30,633 MB. This makes adapters cheap to store and to switch: changing tasks loads only a small bitmask, applied to one shared copy of the base weights, rather than a separate full model checkpoint.

Matching Full Fine-Tuning with Reinforcement Learning

We train Deletion Tuning with GRPO in two settings. The first trains Llama3 Instruct models from 1B to 8B on GSM8K. The second trains Qwen2.5 base models from 0.5B to 14B and the Qwen3-30B-A3B mixture-of-experts model on the SimpleRL-Zoo math corpus and evaluates them on six held-out math benchmarks. We name each model by its family and size, such as Llama8B and Qwen7B. Besides FFT, we compare against LoRA at ranks 1 and 4 and against random sparse fine-tuning (MTH, from The Multiple Ticket Hypothesis) at 99% and 99.95% sparsity, across six backbones from 0.5B to 8B. Every Deletion Tuning run deletes fewer than 0.05% of the model's weights.

Deletion Tuning has the smallest adapter at every scale and matches FFT at the largest backbone of each family. Best-checkpoint validation accuracy (%) and adapter size (MB) under GRPO, on GSM8K for Llama and on the six-benchmark SimpleRL-Zoo mean for Qwen.

	Llama, GSM8K						Qwen, SimpleRL-Zoo
	1B		3B		8B		0.5B		3B		7B
Method	Acc.	Size	Acc.	Size	Acc.	Size	Acc.	Size	Acc.	Size	Acc.	Size
Base	1.21	0	53.53	0	60.27	0	10.44	0	37.73	0	40.18	0
FFT	60.50	4714	82.49	12256	83.24	30633	19.01	1885	44.35	11772	51.89	29051
LoRA r=1	59.29	2.7	82.39	5.8	83.38	10.0	20.30	2.1	43.55	7.1	52.13	9.6
LoRA r=4	60.58	10.8	82.44	23.2	82.87	40.0	20.30	8.4	43.59	28.6	51.70	38.5
MTH 99%	60.73	68.2	80.53	177.4	83.31	443.6	21.84	27.3	44.16	170.4	52.03	420.7
MTH 99.95%	58.61	4.1	82.17	10.7	83.07	26.6	20.16	1.7	43.87	13.7	51.24	25.3
Deletion Tuning	54.44	1.0	81.49	3.1	83.55	8.4	17.64	0.5	41.98	2.7	52.32	6.9

Tuning FFT's learning rate and KL coefficient does not change the parity at 7B and 8B. At smaller sizes, Deletion Tuning recovers 90% to 99% of FFT's accuracy, and the gap closes as models grow. For the mixture-of-experts model, we mask the gate projection inside each of its 128 experts and leave the router unchanged.

Deletion Tuning matches FFT on dense and mixture-of-experts models from 7B to 30B with an adapter thousands of times smaller. Best-checkpoint validation accuracy (%) and adapter size (MB) under GRPO.

	Llama8B		Qwen7B		Qwen14B		Qwen3-30B-A3B
	GSM8K		SimpleRL-Zoo		SimpleRL-Zoo		SimpleRL-Zoo
Method	Acc.	Size	Acc.	Size	Acc.	Size	Acc.	Size
Base	60.27	0	40.18	0	44.37	0	49.41	0
FFT	83.24	30633	51.89	29051	54.87	56343	58.43	116471
Deletion Tuning	83.55	8.4	52.32	6.9	55.43	11.4	58.54	29.6

Acquiring New Capabilities

On pass@k, Deletion Tuning leads the base model at every $k$. Even at $k = 128$, the base model stays 4.3 and 6.1 points below Deletion Tuning on Llama8B and Qwen7B.

Deletion Tuning acquires new capabilities. Pass@k (%), the share of problems solved within $k$ attempts, estimated from 128 samples per problem at temperature 1.0. A lead at large $k$ means solving problems the base model misses even with many attempts.

	Llama8B, GSM8K					Qwen7B, SimpleRL-Zoo
Method	k=1	2	8	32	128	k=1	2	8	32	128
Base	53.20	66.75	83.64	92.40	95.29	27.35	38.75	56.18	66.26	74.01
Deletion Tuning	81.75	88.51	95.75	98.44	99.62	49.39	53.22	64.64	73.23	80.08

Learning the Mask Drives the Gains

Wanda one-shot pruning supplies Deletion Tuning's starting scores. Applied without training, it changes accuracy by at most 2.4 points on all six backbones, while Deletion Tuning gains up to 53.2 points.

Wanda alone stays near the base model. Validation accuracy (%) of the base model, of Wanda one-shot pruning at the best of eight sparsity levels, and of the best checkpoint of the Deletion Tuning mask trained with GRPO.

	Llama, GSM8K			Qwen, SimpleRL-Zoo
Method	1B	3B	8B	0.5B	3B	7B
Base	1.21	53.53	60.27	10.44	37.73	40.18
Wanda (best $\eta$)	1.59	52.46	60.65	11.42	38.70	42.49
Deletion Tuning	54.44	81.49	83.55	17.64	41.98	52.32

Matching Full Fine-Tuning beyond Reinforcement Learning

Deletion Tuning even matches FFT under SFT on MMLU-Pro and BBEH, and under the on-policy distillation method Self-Distillation Fine-Tuning (SDFT) on tool use (ToolAlpaca) and science questions (Science). SFT imitates fixed targets token by token and produces dense updates, while reinforcement-learning updates are sparse. In SDFT, the student samples its own responses and a teacher supplies a target at every token.

Under SFT, Deletion Tuning matches FFT at 3B. Best-checkpoint evaluation accuracy (%), averaged over the 14 MMLU-Pro subjects and the 23 BBEH subtasks, and adapter size (MB) for Llama1B and Llama3B.

	MMLU-Pro				BBEH
	1B		3B		1B		3B
Method	Acc.	Size	Acc.	Size	Acc.	Size	Acc.	Size
Base	10.37	0	12.63	0	2.17	0	8.87	0
FFT	16.97	4714	21.19	12256	8.35	4714	11.39	12256
Deletion Tuning	18.62	1.0	22.28	3.1	5.65	1.0	11.39	3.1

Under SDFT, Deletion Tuning matches FFT on Qwen7B. Best-checkpoint exact-match accuracy (%) and adapter size (MB) on ToolAlpaca tool use and Science.

	ToolAlpaca		Science
Method	Acc.	Size	Acc.	Size
Base	41.24	0	32.35	0
FFT	68.04	29051	71.20	29051
Deletion Tuning	69.07	27.1	70.61	47.1

Deletion Tuning Forgets Less

During GRPO on GSM8K, Deletion Tuning's mean accuracy on six prior tasks (HellaSwag, MMLU, TruthfulQA, WinoGrande, IFEval, and HumanEval) stays within one point of the base model. FFT ends 1.3 to 3.1 points below the base, and LoRA and MTH fall up to 5.4 points. Following RL's Razor, we also track the forward KL to the base policy: Deletion Tuning's stays at or below 0.05, while FFT's reaches 1.0 on Llama1B and 1.5 on Llama8B. MTH at 99.95% sparsity trains 0.05% of all parameters, comparable to the 0.035% to 0.046% that Deletion Tuning deletes at these scales, which suggests that deletion itself keeps forgetting low. Learning new tasks while keeping old ones makes Deletion Tuning a promising candidate for continual learning.

Forward KL and change in prior-task accuracy over GRPO training on Llama1B, Llama3B, and Llama8B: Deletion Tuning stays near zero on both while FFT, LoRA, and MTH drift. — **Deletion Tuning keeps prior-task accuracy at every Llama scale while the baselines drift.** (a) Forward KL between the trained and base policies on GSM8K during GRPO training, the forgetting predictor of RL's Razor. (b) Change in mean accuracy on a six-task prior-task suite from the base model (dashed line).

Why Deletion Tuning Works

We compare each method's weight updates on three measures of update geometry: stable rank (how many directions an update uses), spectrum drift (how far the singular values move), and principal-weight overlap (how many changed weights fall on the base model's principal weights). The last two come from The Path Not Taken, which studies the updates of reinforcement learning with verifiable rewards. Like FFT, Deletion Tuning barely moves the spectrum (its drift stays within 1.4× of FFT's) and changes fewer principal weights than chance (an overlap of 0.43 to 0.46, against FFT's 0.41 to 0.46), while LoRA stays near the random baseline of 0.5 and MTH sits at it.

Stable rank, spectrum drift, and principal-weight overlap of each method's update for Llama8B and Qwen7B: Deletion Tuning is the method closest to FFT on all three. — **Deletion Tuning (blue) tracks FFT (red) most closely across three update-geometry metrics.** Values at the saved GRPO checkpoint for Llama8B (filled) and Qwen7B (open): (a) stable rank of $\Delta W$, (b) spectrum drift, and (c) principal-weight overlap.

LoRA caps the rank of its update, and MTH trains weights chosen at random before training. Deletion Tuning has neither restriction, which helps explain how its updates come close to FFT's without writing new values.

Deletion Tuning Is Robust to Hyperparameter Changes

Deletion Tuning works across a range of values for its hyperparameters: the initial sparsity (how many weights the Wanda initialization deletes), the learning rate, and the regularization strength (how strongly the mask is pushed to keep large weights). Across all 12 combinations we sweep, every setting lands within 1.6 points of the best run on Llama8B GSM8K and within 3.2 points on Qwen7B SimpleRL-Zoo, and varying the initial sparsity fourfold changes accuracy by at most 0.8 and 1.7 points.

BibTeX

@inproceedings{lu2026deleting,
  title     = {The Surprising Effectiveness of Deleting Weights in {LLM} Post-Training},
  author    = {Lu, Jack and Yang, Zhenbang and Lasby, Mike and Pote, Tejas and Ioannou, Yani and Ren, Mengye},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026}
}