CK-Transformer: Commonsense Knowledge Enhanced Transformers for Referring Expression Comprehension (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing frameworks for referring expression comprehension with commonsense knowledge are lacking in the field of multimodal referring . |
| Approach: | They propose a framework for commonsense knowledge Enhanced Transformers which integrates commonsensible knowledge into representations of objects in an image. |
| Outcome: | The proposed framework improves on the existing state of the art in referring expression comprehension with commonsense knowledge (CK-Transformer) it achieves 3.14% accuracy over the existing framework. |
Similar Papers
MaPPER: Multimodal Prior-guided Parameter Efficient Tuning for Referring Expression Comprehension (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for Referring Expression Comprehension (REC) lack specific domain abilities for precise local visual perception and visual-language alignment. |
| Approach: | They propose a framework for Parameter-Efficient Transfer Learning to localize a visual region via natural language using a prior-guided prior. |
| Outcome: | The proposed framework achieves the best accuracy compared to the current methods with only 1.41% tunable backbone parameters. |
Enhancing Natural Language Representation with Large-Scale Out-of-Domain Commonsense (2022.findings-acl)
Copied to clipboard
| Challenge: | Using commonsense in text understanding tasks can cause catastrophic forgetting due to domain discrepancy . previous methods of using textual descriptions as extra input information cannot apply to large-scale commonsensing. |
| Approach: | They propose to use out-of-domain commonsense to enhance text representation . they propose to integrate commonsensense descriptions into large-scale models . |
| Outcome: | The proposed model can integrate commonsense descriptions and enhance them to the target text representation without pre-training on large-scale unsupervised corpora. |
COFAR: Commonsense and Factual Reasoning in Image Search (2022.aacl-main)
Copied to clipboard
Prajwal Gatti, Abhirama Subramanyam Penamakuri, Revant Teotia, Anand Mishra, Shubhashis Sengupta, Roshni Ramnani
| Challenge: | Existing approaches to retrieve relevant images for natural language searches are limited by visual recognition and lack of commonsense reasoning. |
| Approach: | They propose a framework that leverages visual content and natural language queries to enable commonsense reasoning and factual reasoning in the image search. |
| Outcome: | The proposed framework enables commonsense and factual reasoning in image search on a COFAR dataset. |
RECANTFormer: Referring Expression Comprehension with Varying Numbers of Targets (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for REC assume that a single referring expression always refers to a one instance in the image. |
| Approach: | They propose a one-stage method that generates bounding boxes for objects referred to in natural language expressions. |
| Outcome: | The proposed method outperforms baselines in three GREC datasets. |
KM-BART: Knowledge Enhanced Multimodal BART for Visual Commonsense Generation (2021.acl-long)
Copied to clipboard
| Challenge: | Existing models for visual and language understanding are not capable of multimodal reasoning. |
| Approach: | They propose a Transformer-based sequence-to-sequence model capable of reasoning about commonsense knowledge from multimodal inputs of images and texts. |
| Outcome: | The proposed model performs state-of-the-art on the Visual Commonsense Generation task. |
Adaptive Transformers for Learning Multimodal Representations (2020.acl-srw)
Copied to clipboard
| Challenge: | Existing approaches for learning visiolinguistic representations with transformers are over-parametrized and require extensive training. |
| Approach: | They propose to extend attention spans, sparse, and structured dropout methods to learn more about how the network perceives the complexity of input sequences. |
| Outcome: | The proposed approaches improve on language semantics and visiolinguistic representations, but are often over-parametrized and require large amounts of computation. |
KARNA at COIN Shared Task 1: Bidirectional Encoder Representations from Transformers with relational knowledge for machine comprehension with common sense (D19-60)
Copied to clipboard
| Challenge: | Using Bidirectional Encoder Representations from Transformers(BERT) and external relational knowledge from ConceptNet, we are able to achieve an accuracy of 73.3 % on the official test data. |
| Approach: | They propose a model that uses Bidirectional Encoder Representations from Transformers and ConceptNet to tackle the problem of commonsense inference in natural language processing. |
| Outcome: | The proposed model achieves 73.3 % accuracy on the official test data. |
Approximating CKY with Transformers (2023.findings-emnlp)
Copied to clipboard
| Challenge: | CKY algorithm is a cubic dependence on sentence length, but transformers can be used to approximate it. |
| Approach: | They propose a transformer-based approach that approximates the CKY algorithm by directly predicting a sentence's parse and avoiding its cubic dependence on sentence length. |
| Outcome: | The proposed approach achieves better performance than comparable parsers that make use of CKY, while being faster. |
Segment-Level and Category-Oriented Network for Knowledge-Based Referring Expression Comprehension (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods employ sentence-level retrieval and fusion methods, which may lead to similarity bias and interference from irrelevant information in unstructured knowledge sentences. |
| Approach: | They propose a segment-level and category-oriented network to solve similarity bias problem by segmenting and prompting knowledge retrieval methods and a category-based grounding method. |
| Outcome: | The proposed model eliminates similarity bias and improves the overall performance of the KB-REC task. |
Editing Common Sense in Transformers (2023.emnlp-main)
Copied to clipboard
Anshita Gupta, Debanjan Mondal, Akshay Sheshadri, Wenlong Zhao, Xiang Li, Sarah Wiegreffe, Niket Tandon
| Challenge: | Currently, the performance of transformer-based model editing methods is limited to statements about encyclopedic knowledge with a single correct answer. |
| Approach: | They propose to improve MEMIT's model editing algorithm by varying edit tokens and improving the layer selection strategy to improve commonsense knowledge. |
| Outcome: | The MEMIT editing algorithm outperforms baseline models on PEP3k and 20Q datasets while fine-tuning baselines shows significant trade-offs. |