CoNAN: A Complementary Neighboring-based Attention Network for Referring Expression Generation (2020.coling-main)
Copied to clipboard
| Challenge: | Existing approaches to identify and describe complex daily scenes are limited by ambiguity. |
| Approach: | They propose a Complementary Neighboring-based Attention Network that utilizes visual differences between the target object and its highly-related neighbors as complementary features. |
| Outcome: | The proposed expression outperforms state-of-the-art models on a dataset of 3D objects. |
Similar Papers
Visual Referring Expression Recognition: What Do Systems Actually Learn? (N18-2)
Copied to clipboard
| Challenge: | Existing systems for referring expression recognition ignore linguistic structure, instead relying on shallow correlations introduced by unintended biases in the data selection and annotation process. |
| Approach: | They propose to use a system trained on the input image without the input referring expression to achieve a precision of 71.2% in top-2 predictions. |
| Outcome: | The proposed model can achieve 71.2% accuracy on the input image without the input referring expression and 84.2% on the object category given the input. |
Scene Graph Enhanced Pseudo-Labeling for Referring Expression Comprehension (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Referring expression comprehension is a visual-linguistic task that involves localizing objects in images based on textual referring expressions. |
| Approach: | They propose a scene graph-based framework that generates high-quality pseudo region-query pairs . their method captures relationships between objects in images and generates expressions enriched with relation information. |
| Outcome: | The proposed framework outperforms existing methods by 10%, 12%, and 11% on RefCOCO, RefCoCO+, and Ref COCOg datasets. |
Prompting Vision-Language Models For Aspect-Controlled Generation of Referring Expressions (2024.findings-naacl)
Copied to clipboard
Danfeng Guo, Sanchit Agarwal, Arpit Gupta, Jiun-Yu Kao, Emre Barut, Tagyoung Chung, Jing Huang, Mohit Bansal
| Challenge: | Referring Expression Generation (REG) is the task of generating a descriptive caption that uniquely identifies a given target in the scene. |
| Approach: | They propose an Aspect-Controlled REG task which requires generating a referring expression conditioned on the input aspect(s) by changing the input input such as color, location, action etc. |
| Outcome: | The proposed model beats all prior works in the CIDEr score and achieves comparable performance to training with 100% of real data. |
Building Joint Relationship Attention Network for Image-Text Generation (2022.coling-1)
Copied to clipboard
| Challenge: | et al., 2017) focus on visual features individually, while ignoring relationship information among image features that provides important guidance for generating sentences. |
| Approach: | They propose a joint relationship attention network that explores the relationships among image features. |
| Outcome: | The proposed method achieves state-of-the-art performance on large-scale datasets and on Flickr30k datasets. |
FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression Comprehension (2024.emnlp-main)
Copied to clipboard
| Challenge: | Referring Expression Comprehension (REC) is a cross-modal task that objectively evaluates the capabilities of language understanding, image comprehension, and language-to-image grounding. |
| Approach: | They propose to use a new reference expression comprehension (REC) dataset to evaluate the capabilities of language understanding, image comprehension, and language-to-image grounding. |
| Outcome: | The proposed model is able to reject scenarios where the target object is not visible in the image, a key aspect often overlooked in existing models and approaches. |
Convolutional Self-Attention Networks (N19-1)
Copied to clipboard
| Challenge: | Existing models of self-attention networks lack the ability to capture dependencies regardless of distance and can be enhanced with multi-head attention. |
| Approach: | They propose a convolutional self-attention network which can be enhanced by multi-head attention by allowing the model to attend to information from different representation subspaces. |
| Outcome: | The proposed model outperforms existing models on improving locality of SANs on different language pairs and model settings. |
NeuralREG: An end-to-end approach to referring expression generation (P18-1)
Copied to clipboard
| Challenge: | Referring Expression Generation models typically rely on features such as salience and grammatical function to make decisions about form and content. |
| Approach: | They propose a new approach that makes decisions about form and content in one go . they use a delexicalized version of the WebNLG corpus to test the approach . |
| Outcome: | The proposed approach significantly improves over two strong baselines. |
Contrastive Learning-Enhanced Nearest Neighbor Mechanism for Multi-Label Text Classification (2022.acl-short)
Copied to clipboard
| Challenge: | Existing methods for multi-label text classification neglect the knowledge from the existing similar instances when predicting labels of a specific text. |
| Approach: | They propose a k nearest neighbor mechanism which retrieves several neighbor instances and interpolates the model output with their labels. |
| Outcome: | Extensive experiments show that the proposed method can bring significant performance improvements to multiple MLTC models including state-of-the-art pretrained and non-pretrained ones. |
Comparing Neighbors Together Makes it Easy: Jointly Comparing Multiple Candidates for Efficient and Effective Retrieval (2024.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that using only bi-encoders as an intermediate reranker can improve top-1 accuracy with negligible slowdown (7%). |
| Approach: | They propose a framework that compares a query and multiple embeddings of similar candidates through shallow self-attention layers, delivering rich representations contextualized to each other. |
| Outcome: | The proposed framework compares a query and multiple embeddings of similar candidates through shallow self-attention layers, delivering rich representations contextualized to each other. |
Cross-layer Attention Sharing for Pre-trained Large Language Models (2026.tacl-1)
Copied to clipboard
Yongyu Mu, Yuzhang Wu, Yuchun Fan, Chenglong Wang, Hengyu Li, Jiali Zeng, Qiaozhi He, Murun Yang, Fandong Meng, Jie Zhou, Tong Xiao, Jingbo Zhu
| Challenge: | Existing studies focus on compressing the Key-Value cache or grouping attention heads, while overlooking redundancy between layers. |
| Approach: | They propose a lightweight substitute for self-attention in well-trained LLMs that uses feed-forward networks to align attention heads between adjacent layers and low-rank matrices to approximate differences in layer-wise attention weights. |
| Outcome: | The proposed model reduces redundancy by sharing weights across layers while maintaining high response quality while reducing redundant calculations within 53% 84% of the total layers. |