Papers with FIRST
FIRST: Faster Improved Listwise Reranking with Single Token Decoding (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing listwise LLMs lack efficiency as they provide ranking output in the form of a generated ordered sequence of candidate passage identifiers. |
| Approach: | They propose a listwise LLM reranking approach that leverages the first generated identifier to obtain a ranked ordering of the candidates. |
| Outcome: | The proposed approach accelerates inference by 50% while maintaining robust ranking performance with gains across BEIR benchmark. |
Overcoming a Theoretical Limitation of Self-Attention (2022.acl-long)
Copied to clipboard
| Challenge: | Hahn shows that for languages where acceptance depends on a single input symbol, a transformer’s classification decisions get closer and closer to random guessing as input strings get longer and longer. |
| Approach: | They propose a transformer that recognizes PARITY with perfect accuracy and a model that uses layer normalization to bring the cross-entropy of both models arbitrarily close to zero. |
| Outcome: | The proposed model can accept and reject strings with perfect accuracy and bring cross-entropy close to zero when they need to focus on a single position. |
FIRST: Teach A Reliable Large Language Model Through Efficient Trustworthy Distillation (2024.emnlp-main)
Copied to clipboard
KaShun Shum, Minrui Xu, Jianshu Zhang, Zixin Chen, Shizhe Diao, Hanze Dong, Jipeng Zhang, Muhammad Raza
| Challenge: | Experimental results show that a well-calibrated model is more reliable than a fine-tuned model due to “tuning-induced mis-calibration”. |
| Approach: | They propose a method which utilizes a small portion of teacher’s knowledge to obtain a reliable language model in a cost-efficient way. |
| Outcome: | The proposed method reduces the computational burden by utilizing teacher's knowledge to obtain a reliable language model in a cost-efficient way. |