Papers by Alessio Devoto
Steering Knowledge Selection Behaviours in LLMs via SAE-Based Representation Engineering (2025.naacl-long)
Copied to clipboard
Yu Zhao, Alessio Devoto, Giwon Hong, Xiaotang Du, Aryo Pradipta Gema, Hongru Wang, Xuanli He, Kam-Fai Wong, Pasquale Minervini
| Challenge: | Large language models store factual knowledge in their parameters but their parametric knowledge can conflict with the information provided in the context. |
| Approach: | They propose a training-free representation engineering method that uses pre-trained sparse auto-encoders to control the knowledge selection behaviour of large language models. |
| Outcome: | The proposed method can control the use of both knowledge sources to resolve knowledge conflict in open-domain question-answering tasks surpassing existing representation engineering methods (+10%) and contrastive decoding methods (+5%). |
Are We Done with MMLU? (2025.naacl-long)
Copied to clipboard
Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, Claire Barale, Robert McHardy, Joshua Harris, Jean Kaddour, Emile Van Krieken, Pasquale Minervini
| Challenge: | MMLU is widely adopted but its ground truth errors obscure the true capabilities of LLMs. |
| Approach: | They propose a framework for identifying dataset errors using a novel error annotation protocol and a subset of 5,700 manually re-annotated questions. |
| Outcome: | The proposed framework is based on 5,700 re-annotated questions from the MMLU benchmark. |
A Simple and Effective L_2 Norm-Based Strategy for KV Cache Compression (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to reduce the KV cache size involve fine-tuning the model to learn a compression strategy or leveraging attention scores to reduce sequence length. |
| Approach: | They find a correlation between the L2 norm and attention scores over cached KV pairs . they compress the KV cache based on the L1 norm of key embeddings . |
| Outcome: | The proposed approach reduces the KV cache size by 50% on language modelling and needle-in-a-haystack tasks and 90% on passkey retrieval tasks without losing accuracy. |
Attention Sinks in Diffusion Language Models (2026.findings-acl)
Copied to clipboard
Maximo Eduardo Rulli, Simone Petruzzi, Edoardo Michielon, Fabrizio Silvestri, Simone Scardapane, Alessio Devoto
| Challenge: | Masked Diffusion Language Models (DLMs) employ transformer encoders with bidirectional attention, enabling parallel token generation while maintaining competitive performance. |
| Approach: | They conduct an empirical analysis of DLM attention patterns focusing on the attention sinking phenomenon . they find that DLMs also exhibit attention sinks, but with distinct characteristics . |
| Outcome: | The proposed models employ transformer encoders with bidirectional attention, enabling parallel token generation while maintaining competitive performance. |