Papers by Michael Hassid
How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformers (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained language models use the attention mechanism to contextualize input inputs . but, we find that it is not as important as thought for pretrained models . |
| Approach: | They propose a probing method that replaces input-dependent attention matrices with constant ones. |
| Outcome: | The proposed method improves performance of pretrained language models without input-dependent attention. |
On Pruning State-Space LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work proposed state-space models as an efficient alternative to transformers. |
| Approach: | They propose to prune state-space models (SSMs) to reduce computation costs by using unstructured pruning methods. |
| Outcome: | The proposed pruning methods show that they can be pruned to reduce their computation costs. |
Efficient Methods for Natural Language Processing: A Survey (2023.tacl-1)
Copied to clipboard
Marcos Treviso, Ji-Ung Lee, Tianchu Ji, Betty van Aken, Qingqing Cao, Manuel R. Ciosici, Michael Hassid, Kenneth Heafield, Sara Hooker, Colin Raffel, Pedro H. Martins, André F. T. Martins, Jessica Zosa Forde, Peter Milder, Edwin Simpson, Noam Slonim, Jesse Dodge, Emma Strubell, Niranjan Balasubramanian, Leon Derczynski, Iryna Gurevych, Roy Schwartz
| Challenge: | Recent work in natural language processing (NLP) has yielded appealing results from scaling model parameters and training data, but using only scale to improve performance means resource consumption also grows. |
| Approach: | They propose to use data, time, storage, or energy to improve model performance. |
| Outcome: | The proposed methods and findings provide guidance for conducting NLP under limited resources and point towards promising research directions for developing more efficient methods. |
Finding the SWEET Spot: Analysis and Improvement of Adaptive Inference in Low Resource Settings (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained Transformer-based language models such as BERT, DeBERTa, and GPT3 have become the go-to tool in NLP. |
| Approach: | They propose an Early-Exit fine-tuning method that assigns each classifier its own set of unique model weights, not updated by other classifiers. |
| Outcome: | The proposed method outperforms Early-Exit and Multi-Model at fast speeds while maintaining comparable scores to Early- Exit at slow speeds. |
Transformers are Multi-State RNNs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Schwartz et al., 2017) have been using transformers for long-range tasks for NLP since the 1990s. |
| Approach: | They propose a transformer-only transformer with unlimited hidden state size that can be converted into bounded multistate RNNs by fixing the size of their hidden state. |
| Outcome: | The proposed compression policy outperforms baseline compression policies on long range tasks and LLMs. |
More Documents, Same Length: Isolating the Challenge of Multiple Documents in RAG (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Retrieval-Augmented Generation (RAG) enhances the accuracy of Large Language Models by leveraging relevant external documents during generation. |
| Approach: | They evaluate various language models on custom datasets derived from QA tasks . they keep context length and position of relevant information constant while varying the number of documents . |
| Outcome: | The proposed method improves the accuracy of large language models by leveraging external documents . increasing document count reduces performance by up to 20%, the authors find . |