Papers by Akash Srivastava
Hopscotch: Discovering and Skipping Redundancies in Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Hopscotch is a method that skips attention blocks with least contributions to a task . it does not modify model weights or require access to pretraining or instruction-tuning data. |
| Approach: | Hopscotch proposes a method that skips attention blocks with least contributions to a task . it introduces lightweight scaling parameters to attention and MLP blocks . |
| Outcome: | The proposed method reduces the drop in performance by 2% even after skipping four attention blocks. |