Damage-Aware Bandit Pruning for Vision and Language Transformers: A Technical Analysis

Damage-Aware Bandit Pruning for Vision and Language Transformers: A Technical Analysis

Damage-aware bandit pruning is a novel approach to structured post-training pruning of transformers. This technique involves formulating structured-unit selection for language and vision transformers as a damage-aware multi-armed bandit problem under a fixed candidate-evaluation budget.

What is Damage-Aware Bandit Pruning?

Damage-aware bandit pruning is a method for selecting complete functional units in transformers whose suppression causes limited degradation. This approach is particularly useful in vision and language transformers, where the suppression of certain units can lead to significant degradation in performance.

How Does Damage-Aware Bandit Pruning Work?

The damage-aware bandit pruning method involves the following steps:

1. Candidate Evaluation: A set of candidate units (e.g., attention heads or MLP channel groups) is selected for evaluation.
2. Paired Damage Calculation: The masked loss (i.e., the loss incurred when a unit is temporarily masked) is calculated for each candidate unit. The paired damage is then computed as the difference between the masked loss and the base loss on the same batch.
3. Reward Calculation: A smooth bounded reward is calculated based on the paired damage. This reward drives either a UCB-style policy or fractional-Beta Thompson Sampling.
4. Unit Selection: The final mask is constructed sequentially by adding one unit at each step. The selected units are functionally zeroed in the original dense checkpoint.

Experimental Results

The damage-aware bandit pruning method was evaluated on various vision and language transformers, including GPT-2, OPT, Pythia, Qwen2.5, SmolLM2, ViT-B/16, DeiT-Tiny, and Swin-Tiny. The results show that the bandit methods usually reduce degradation relative to budgeted greedy in the paired language-model comparisons.

Comparison with Other Methods

The damage-aware bandit pruning method was compared with random, magnitude, static-saliency, and budgeted-greedy selection. The results show that the bandit methods outperform these baselines in most cases.

Takeaways

• Damage-aware bandit pruning is a novel approach to structured post-training pruning of transformers.
• This method involves formulating structured-unit selection as a damage-aware multi-armed bandit problem under a fixed candidate-evaluation budget.
• The damage-aware bandit pruning method was evaluated on various vision and language transformers, showing that it reduces degradation relative to budgeted greedy in most cases.

Damage-aware bandit pruning is a promising approach to structured post-training pruning of transformers. However, further research is needed to fully understand its potential and limitations.