Papers with FiD
Optimizing Retrieval-augmented Reader Models via Token Elimination (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for ODQA use a retrieval-augmented language model . a generative model can cause a significant bottleneck in decoding time . |
| Approach: | They propose to eliminate some of the retrieved information that might not contribute essential information to the answer generation process. |
| Outcome: | The proposed method reduces run-time by up to 62.2% with only 2% reduction in performance and improves performance. |
REANO: Optimising Retrieval-Augmented Reader Models through Knowledge Graph Generation (2024.acl-long)
Copied to clipboard
| Challenge: | Existing knowledge graphs suffer from incompleteness and lack information critical for answering given questions. |
| Approach: | They propose to enhance the open domain question answering model with a knowledge graph generation module that generates KGs from the passages and an answer predictor. |
| Outcome: | The proposed model improves the exact match score by 2.7% on the EntityQuestion dataset, with an average improvement of 1.8% across all the datasets. |
ListT5: Listwise Reranking with Fusion-in-Decoder Improves Zero-shot Retrieval (2024.acl-long)
Copied to clipboard
| Challenge: | Existing listwise reranking models rely on pointwise sizing of each passage . Until now, listwise models lack the ability to compare between passages at inference time . |
| Approach: | They propose a listwise reranking approach based on Fusion-in-Decoder that handles multiple candidate passages at train and inference time. |
| Outcome: | The proposed model outperforms the state-of-the-art RankT5 model on the BEIR benchmark for zero-shot retrieval task with a notable +1.3 gain in the average NDCG@10 score. |
RFiD: Towards Rational Fusion-in-Decoder for Open-Domain Question Answering (2023.findings-acl)
Copied to clipboard
| Challenge: | Open-domain Question Answering (ODQA) systems rely on spurious features instead of genuine causal relationships to generate answers. |
| Approach: | They propose a model that leverages the encoders of FiD to distinguish between causal relationships and spurious features and guides the decoder to generate answers informed by this discernment. |
| Outcome: | The proposed model improves on two ODQA datasets and shows that it can identify causal relationships and identify spurious features. |
MAFiD: Moving Average Equipped Fusion-in-Decoder for Question Answering over Tabular and Textual Data (2023.findings-eacl)
Copied to clipboard
| Challenge: | Experimental results show that Transformer-based questions have a "long" hybrid sequence over tabular and textual elements, causing long-range reasoning problems. |
| Approach: | They propose a moving average-equipped fusion-in-decoder to handle long-range reasoning problems . they use FiD and EMA to combine different levels of reasoning . |
| Outcome: | Experimental results show that the proposed model increases exact matching and F1 by 1.1 and 1.7 on the blind test set. |
FiD-Ex: Improving Sequence-to-Sequence Models for Extractive Rationale Generation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained sequence to sequence models are effective in making and generating NL explanations, but they have many shortcomings. |
| Approach: | They propose a model that uses sentence markers to eliminate explanation fabrication . they use fusion-in-decoder architecture to handle long input contexts . |
| Outcome: | The proposed model significantly improves on the ERASER explainability benchmark. |
FastFiD: Improve Inference Efficiency of Open Domain Question Answering via Sentence Selection (2024.acl-long)
Copied to clipboard
| Challenge: | Open Domain Question Answering (ODQA) is a longstanding task in Natural Language Processing that involves generating an answer solely based on a given question. |
| Approach: | They propose a novel approach that executes sentence selection on the encoded passages to enhance the inference speed while reducing the context length required for generating answers. |
| Outcome: | The proposed approach can increase inference speed by **2.3X-5.7X** while maintaining the model’s performance. |
KG-FiD: Infusing Knowledge Graph in Fusion-in-Decoder for Open-Domain Question Answering (2022.acl-long)
Copied to clipboard
Donghan Yu, Chenguang Zhu, Yuwei Fang, Wenhao Yu, Shuohang Wang, Yichong Xu, Xiang Ren, Yiming Yang, Michael Zeng
| Challenge: | Open-Domain Question Answering (ODQA) models typically include a retrieving module and a reading module. |
| Approach: | They propose a new open-domain question-answering framework that uses a knowledge-enhanced version of FiD to improve the approach. |
| Outcome: | The proposed model improves on ODQA benchmark datasets with less than 40% computation cost. |
Strong and Efficient Baselines for Open Domain Conversational Question Answering (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Current Open Domain Question Answering (ODQA) approaches to reevaluating baselines for efficiency and effectiveness have been limited due to various limitations. |
| Approach: | They propose and evaluate strong yet simple and efficient baselines by introducing a fast reranking component between the retriever and the reader and performing targeted finetuning steps. |
| Outcome: | The proposed method improves the SotA results while reducing the reader’s latency by 60%. |
FiD-ICL: A Fusion-in-Decoder Approach for Efficient In-Context Learning (2023.acl-long)
Copied to clipboard
| Challenge: | Large pre-trained models are capable of few-shot in-context learning (ICL) however, concatenated demonstrations are often excessively long and require additional computation. |
| Approach: | They propose to apply fusion-in-decoder (FiD) models to perform few-shot in-context learning (ICL) they propose to use concatenation-based, early-fusion, intermediate- and late-fusion methods to improve efficiency . |
| Outcome: | The proposed methods outperform concatenation-based models on 11 held-out tasks. |
FiDO: Fusion-in-Decoder optimized for stronger performance and faster inference (2023.findings-acl)
Copied to clipboard
Michiel de Jong, Yury Zemlyanskiy, Joshua Ainslie, Nicholas FitzGerald, Sumit Sanghai, Fei Sha, William Cohen
| Challenge: | Fusion-in-Decoder (FiD) is a powerful retrieval-augmented language model . however, the architecture used for FiD was not designed for retrieval augmented models . |
| Approach: | They propose to make FiD a modified retrieval-augmented language model with a large decoder and memory bandwidth constraints to alleviate memory bandwidth limitations. |
| Outcome: | The proposed architecture outperforms existing models on knowledge-intensive tasks even on large models on many knowledge-based tasks. |
Context Quality Matters in Training Fusion-in-Decoder for Extractive Open-Domain Question Answering (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have shown that the quantity and quality of context affect retrieval-augmented generation models during training. |
| Approach: | They propose a method to mitigate overfitting to specific context quality by introducing bias to the cross-attention distribution. |
| Outcome: | The proposed method improves retrieval-augmented generation models on different context quality. |
From Reading to Compressing: Exploring the Multi-document Reader for Prompt Compression (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have recently exhibited performance gains owing to a wide variety of prompting techniques, including Retrieval-Augmented Generation (RAG), Chain-of-Thought (CoT), and In-Context Learning (ICL). |
| Approach: | They propose a prompt compression method that captures the global context without compromising semantic consistency while detouring the necessity of pseudo-labels for training the compressor. |
| Outcome: | Empirical results show that the proposed method retains key contexts while reducing the prompt length by 80%. |