Robust and Interpretable Grounding of Spatial References with Relation Networks (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models for understanding spatial references in text are vulnerable to noise in input text or state observations. |
| Approach: | They propose a text-conditioned relation network with a cross-modal attention module to capture fine-grained spatial relations between entities and a model that is robust and interpretable. |
| Outcome: | The proposed model improves performance on three tasks with a 17% improvement in predicting goal locations and a 15% improvement in robustness compared to state-of-the-art systems. |
Similar Papers
Attention as Grounding: Exploring Textual and Cross-Modal Attention on Entities and Relations in Language-and-Vision Transformer (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing work has focused on what is captured by multi-modal architectures. |
| Approach: | They propose a multi-modal transformer that learns syntactic and semantic representations about entities and relations grounded in objects at the level of masked self-attention and cross-modal attention. |
| Outcome: | The proposed model learns syntactic and semantic representations about objects and relations cross-modally and unimodally. |
RE2: Region-Aware Relation Extraction from Visually Rich Documents (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing studies on relation extraction from visually rich documents focus on layout structure and Optical Character Recognition (OCR) results. |
| Approach: | They propose a relation extraction tool that leverages layout structure among entity blocks to improve relation prediction. |
| Outcome: | The proposed model outperforms existing models on a wide range of domains and languages. |
Visually Guided Spatial Relation Extraction from Text (N18-2)
Copied to clipboard
| Challenge: | Existing studies show that spatial relations can be extracted with a good accuracy, but spatial relation extraction is still challenging. |
| Approach: | They propose to use visual modality to fill the information gap in the text modality and resolve spatial semantic ambiguities. |
| Outcome: | The proposed model fills the information gap in the text modality and resolves spatial semantic ambiguities. |
Representation, Learning and Reasoning on Spatial Language for Downstream NLP Tasks (2020.emnlp-tutorials)
Copied to clipboard
| Challenge: | In this tutorial, we discuss the cutting-edge research results and existing challenges related to spatial language understanding including semantic annotations, existing corpora, symbolic and sub-symbolic representations, qualitative spatial reasoning, spatial common sense, deep and structured learning models. |
| Approach: | This tutorial presents cutting-edge research results and current challenges related to spatial language understanding including semantic annotations, existing corpora, symbolic and sub-symbolic representations, qualitative spatial reasoning, spatial common sense, deep and structured learning models. |
| Outcome: | This paper reviews the cutting-edge research results and current challenges related to spatial language understanding including semantic annotations, existing corpora, symbolic and sub-symbolic representations, qualitative spatial reasoning, spatial common sense, deep and structured learning models. |
Document-Level N-ary Relation Extraction with Multiscale Representation Learning (N19-1)
Copied to clipboard
| Challenge: | Existing work on cross-sentence relation extraction is limited to three consecutive sentences, which severely limits recall. |
| Approach: | They propose a multiscale neural architecture for document-level n-ary relation extraction that combines representations learned over various text spans throughout the document and across the subrelation hierarchy. |
| Outcome: | The proposed system outperforms existing methods on biomedical machine reading. |
Uncover the Ground-Truth Relations in Distant Supervision: A Neural Expectation-Maximization Framework (D19-1)
Copied to clipboard
| Challenge: | Existing methods for relation extraction assume that text is noisy, but its corresponding labels are clean. |
| Approach: | They propose a framework that combines neural network and probabilistic modelling to denoise noisy relation labels. |
| Outcome: | The proposed framework improves the current art in uncovering the ground-truth relation labels. |
FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression Comprehension (2024.emnlp-main)
Copied to clipboard
| Challenge: | Referring Expression Comprehension (REC) is a cross-modal task that objectively evaluates the capabilities of language understanding, image comprehension, and language-to-image grounding. |
| Approach: | They propose to use a new reference expression comprehension (REC) dataset to evaluate the capabilities of language understanding, image comprehension, and language-to-image grounding. |
| Outcome: | The proposed model is able to reject scenarios where the target object is not visible in the image, a key aspect often overlooked in existing models and approaches. |
KnowDR-REC: Auditing Knowledge-Conditioned Visual Grounding in Referring Expression Comprehension (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation metrics suggest that Multimodal large language models have acquired fine-grained visual grounding capabilities. |
| Approach: | They propose a benchmark to assess Referring Expression Comprehension (REC) that uses intra-image visual cues to localize target objects and a controllable evaluation mechanism to test sensitivity to fine-grained factual changes. |
| Outcome: | The proposed benchmarks show that multimodal large language models have a high level of performance on the RefCOCO family of benchmarks. |
A Hybrid Model of Classification and Generation for Spatial Relation Extraction (2022.coling-1)
Copied to clipboard
| Challenge: | Existing studies only focus on spatial relations extraction as a classification task . spatial information is one kind of critical information for natural language understanding . |
| Approach: | They propose a hybrid model that generates null-role relations and extracts non-null-rol . they propose varying kinds of schemes to represent spatial relation . |
| Outcome: | The proposed model outperforms the baselines on the spatial relation extraction task on SpaceEval. |
Understanding Spatial Relations through Multiple Modalities (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing work on common sense reasoning and understanding of spatial relations is limited. |
| Approach: | They propose a spatial model that uses both textual and visual information to predict spatial relations between two entities in an image. |
| Outcome: | The proposed model improves prediction accuracy and coverage and deals with unseen subjects, objects and relations. |