MORE: Multi-mOdal REtrieval Augmented Generative Commonsense Reasoning (2024.findings-acl)
Copied to clipboard
| Challenge: | Language Models (LLMs) have gained increasing prominence in artificial intelligence, especially Large Language Model (LLm) due to the well-recognized reporting bias, the recording of commonsense information is significantly less than its existence in reality. |
| Approach: | They propose a Multi-mOdal REtrieval framework to leverage both text and images to enhance commonsense ability of language models. |
| Outcome: | The proposed framework can leverage both text and images to enhance commonsense ability of language models. |
Similar Papers
Retrieval Augmentation for Commonsense Reasoning: A Unified Approach (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for retrieving encyclopedic knowledge lack a large corpus and effective commonsense retriever. |
| Approach: | They propose a framework for retrieval-augmented commonsense reasoning with a large commonsensense corpus and a commonseense retriever. |
| Outcome: | The proposed framework outperforms existing methods on commonsense reasoning tasks. |
Generative Data Augmentation for Commonsense Reasoning (2020.findings-emnlp)
Copied to clipboard
Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji-Ping Wang, Chandra Bhagavatula, Yejin Choi, Doug Downey
| Challenge: | Recent advances in commonsense reasoning depend on large-scale human-authored training data. |
| Approach: | They propose a generative data augmentation technique that augments human-authored training data by using pretrained language models. |
| Outcome: | The proposed technique outperforms existing methods on commonsense reasoning benchmarks and enhances out-of-distribution generalization. |
Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work has proposed to improve relevance modeling by having large language models actively involved in retrieval, i.e., to guide retrieval with generation. |
| Approach: | They propose to have large language models actively involved in retrieval to guide retrieval with generation. |
| Outcome: | The proposed method synergizes retrieval and generation in an iterative manner, and can generate better results in subsequent iterations. |
Cross-Modal Retrieval Augmentation for Multi-Modal Classification (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in using retrieval components over external knowledge sources have shown impressive results for a variety of downstream tasks in natural language processing. |
| Approach: | They propose a retrieval-augmented multi-modal transformer architecture for embedding images and captions in the same space. |
| Outcome: | The proposed approach improves visual question answering over strong baselines and hot-swapping indices. |
Retrieval Enhanced Model for Commonsense Generation (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing frameworks for commonsense generation are lacking for pre-trained models. |
| Approach: | They propose a framework that uses concept matching to retrieve prototype sentences and trainable sentence retriever to enhance pre-training and fine-tuning. |
| Outcome: | The proposed framework achieves state-of-the-art on the large-scale Common-Gen benchmark. |
Towards Multi-Modal Text-Image Retrieval to improve Human Reading (2021.naacl-srw)
Copied to clipboard
| Challenge: | In primary school, children's books, as well as in modern language learning apps, multi-modal learning strategies like illustrations of terms and phrases are used to support reading comprehension. |
| Approach: | They propose to use multi-modal transformers to train multi-dimensional models on text-image retrieval to support a user's reading comprehension of arbitrary text. |
| Outcome: | The proposed model performs poorly because of the short and relatively simple textual data that the current models are trained with. |
Retrieving Multimodal Information for Augmented Generation: A Survey (2023.findings-emnlp)
Copied to clipboard
Ruochen Zhao, Hailin Chen, Weishi Wang, Fangkai Jiao, Xuan Long Do, Chengwei Qin, Bosheng Ding, Xiaobao Guo, Minzhi Li, Xingxuan Li, Shafiq Joty
| Challenge: | Large Language Models (LLMs) are increasingly using multimodality to augment their generation ability, but there is no unified perception of at which stage and how to incorporate different modalities. |
| Approach: | They propose to use multimodality to augment Large Language Models (LLMs) this will provide scholars with a deeper understanding of the methods' applications and encourage them to adapt existing techniques to the fast-growing field of LLMs. |
| Outcome: | The proposed methods improve factuality, reasoning, interpretability, and robustness of the generated content. |
Beyond Language: Learning Commonsense from Images for Reasoning (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing commonsense reasoning methods use raw texts to perform data representation and answer prediction tasks. |
| Approach: | They propose a novel approach to learn commonsense from images instead of limited raw texts or costly knowledge bases. |
| Outcome: | The proposed approach outperforms language-based methods on commonsense reasoning problems on two commonsence reasoning problems. |
Embedding-Informed Adaptive Retrieval-Augmented Generation of Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Retrieval-augmented large language models excel in various NLP tasks but are not always helpful when the knowledge required is absent in the model. |
| Approach: | They propose to determine whether the model is knowledgeable on a query via inspecting the (contextualized) pre-trained token embeddings of LLMs. |
| Outcome: | Experiments show that the proposed approach performs better than previous approaches on various benchmarks. |
LaMI: Augmenting Large Language Models via Late Multi-Image Fusion (2026.acl-short)
Copied to clipboard
| Challenge: | Large Language Models lack visual grounding on visual reasoning, despite training on text alone. |
| Approach: | They propose a late multi-image fusion method that augments LLMs with test-time visual signals. |
| Outcome: | Using a late multi-image fusion method, the proposed model outperforms LLMs on visual reasoning and matches VLMs in vision-based tasks. |