Efficient Out-of-Domain Detection for Sequence to Sequence Models (2023.findings-acl)
Copied to clipboard
Artem Vazhentsev, Akim Tsvigun, Roman Vashurin, Sergey Petrakov, Daniil Vasilev, Maxim Panov, Alexander Panchenko, Artem Shelmanov
| Challenge: | Sequence-to-sequence (seq2sequ) models are a ubiquitous tool for text generation but they are not suitable for many other tasks. |
| Approach: | They propose to use UE techniques to identify out-of-domain (OOD) inputs where the model is susceptible to errors. |
| Outcome: | The proposed methods outperform heavyweight ensembles on the task of OOD detection. |
Similar Papers
Understanding and Improving Sequence-to-Sequence Pretraining for Neural Machine Translation (2022.acl-long)
Copied to clipboard
| Challenge: | Existing studies on self-supervised pretraining for machine translation have focused on the jointly pretrained decoder . |
| Approach: | They propose a method to improve neural machine translation by jointly pretrained decoder . they propose two strategies to remedy the domain and objective discrepancies . |
| Outcome: | The proposed approach improves translation performance and model robustness on three language pairs. |
Multi-Task Dense Retrieval via Model Uncertainty Fusion for Open-Domain Question Answering (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to multitask dense retrieval are not effective due to corpus inconsistency. |
| Approach: | They propose to train individual dense passage retrievers for different open-domain question-answering tasks and aggregate their predictions during test time. |
| Outcome: | The proposed method achieves state-of-the-art performance on 5 benchmark QA datasets, with up to 10% improvement in top-100 accuracy compared to a joint-training multi-task DPR on SQuAD. |
Sparse Sequence-to-Sequence Models (P19-1)
Copied to clipboard
| Challenge: | Sequence-to-sequence models are dense and assigning nonzero probability to implausible outputs. |
| Approach: | They propose a new family of -entmax transformations that includes softmax and sparsemax as particular cases and is sparser for any > 1 . they provide fast algorithms to evaluate these transformations and their gradients, which scale well for large vocabulary sizes. |
| Outcome: | The proposed models are able to produce sparse alignments and assign nonzero probability to short list of plausible outputs, sometimes rendering beam search exact. |
Transforming Sequence Tagging Into A Seq2Seq Task (2022.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained, large, generative language models have had great success in a wide range of sequence tagging and structured prediction tasks. |
| Approach: | They propose to use a new format for casting input text sentences and their output labels into the input and target of a Seq2Seq model and introduce it to test their hypothesis. |
| Outcome: | The proposed format shows to be both simpler and more effective and devoid of hallucination. |
Rethinking Model Selection and Decoding for Keyphrase Generation with Pre-trained Sequence-to-Sequence Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Keyphrase generation is a longstanding task in NLP with widespread applications. |
| Approach: | They propose a likelihood-based decode-select algorithm for seq2seq PLMs that improves greedy search by an average of 4.7% semantic F1 across five datasets. |
| Outcome: | The proposed algorithm improves greedy search by an average of 4.7% semantic F1 across five datasets. |
LUQ: Long-text Uncertainty Quantification for LLMs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing research on Uncertainty Quantification (UQ) predominantly targets short text generation, however, real-world applications often necessitate much longer responses. |
| Approach: | They propose a method that ensembles responses from multiple models and selects the response with the lowest uncertainty. |
| Outcome: | The proposed method outperforms baseline methods in correlating with the model’s factuality scores (negative coefficient of -0.85 observed for Gemini Pro). |
Denoising based Sequence-to-Sequence Pre-training for Text Generation (D19-1)
Copied to clipboard
| Challenge: | PoDA pre-trains encoders and decoders by denoising noise-corrupted text . Unlike encoder-only or decode-only methods, it can be used for text generation tasks without using any task-specific techniques. |
| Approach: | They propose a sequence-to-sequence (seq2sequ) pre-training method PoDA which denoises autoencoders by denoising noise-corrupted text. |
| Outcome: | The proposed method improves model performance over strong baselines without using any task-specific techniques and significantly speed up convergence. |
Structural generalization is hard for sequence-to-sequence models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Sequence-to-sequence models have been successful across many NLP tasks, but they have low generalization accuracy . |
| Approach: | They propose to use linguistic knowledge to overcome generalization limitations of seq2seq models . they show that human beings are able to understand and produce linguistic structures they have never observed before . |
| Outcome: | The proposed models can overcome this limitation by having linguistic knowledge built in. |
Uncertainty Estimation of Transformer Predictions for Misclassification Detection (2022.acl-long)
Copied to clipboard
Artem Vazhentsev, Gleb Kuzmin, Artem Shelmanov, Akim Tsvigun, Evgenii Tsymbalov, Kirill Fedyanin, Maxim Panov, Alexander Panchenko, Gleb Gusev, Mikhail Burtsev, Manvel Avetisian, Leonid Zhukov
| Challenge: | Uncertainty estimation (UE) of model predictions is crucial step for a variety of tasks such as active learning, misclassification detection, adversarial attack detection, etc. |
| Approach: | They propose to modify UE methods for Transformer models for misclassification detection in named entity recognition and text classification tasks to improve model expressiveness and computational performance. |
| Outcome: | The proposed methods outperform computationally intensive methods on misclassification detection tasks and are based on a large dataset of simulated datasets. |
Reconsidering LLM Uncertainty Estimation Methods in the Wild (2025.acl-long)
Copied to clipboard
Yavuz Faruk Bakman, Duygu Nur Yaldiz, Sungmin Kang, Tuo Zhang, Baturalp Buyukates, Salman Avestimehr, Sai Praneeth Karimireddy
| Challenge: | Existing studies evaluate UE methods in short-form QA settings, but real-world deployment presents several challenges. |
| Approach: | They examine UE methods' sensitivity to decision threshold selection and their robustness to query transformations such as typos and adversarial prompts. |
| Outcome: | The proposed methods exhibit robustness against typos, adversarial prompts, and prior chat history, and are highly susceptible to adversarials. |