WikiScenes with Descriptions: Aligning Paragraphs and Sentences with Images in Wikipedia Articles (2024.starsem-1)
Copied to clipboard
| Challenge: | Existing work on processing image-text alignment in multimodal documents has been unsupervised, facing the challenge of missing evaluation and training data. |
| Approach: | They propose to provide one of the first datasets that provides ground-truth annotations of image-text alignments in multi-paragraph multi-image articles. |
| Outcome: | The proposed dataset can be used to study phenomena of visual language grounding in longer documents and assess retrieval capabilities of language models trained on captioning data. |
Similar Papers
Leverage Points in Modality Shifts: Comparing Language-only and Multimodal Word Representations (2023.starsem-1)
Copied to clipboard
| Challenge: | a recent study of the effect of visual grounding on language representations has given a new life to the debate around extractability and quality of semantic information in representations trained solely on textual input. |
| Approach: | They compare word embeddings from vision-and-language models to text-only models . they identify meaning properties and relations that characterize words whose embeddements are most affected by visual grounding . |
| Outcome: | The proposed model differs from text-only models on semantic representations of language . the study is the first large-scale study of the effect of visual grounding on language representations . |
Scalable Performance Analysis for Vision-Language Models (2023.starsem-1)
Copied to clipboard
| Challenge: | a new method to probe vision-language models is proposed that does not require data annotation and makes use of existing datasets. |
| Approach: | They propose a method that extracts features from a vision-language benchmark and measures their correlation with the output of the target model. |
| Outcome: | The proposed method is scalable and does not require data annotation . it can be used with other models and benchmarks, and is available at https://github.com/MichiganNLP/Scalable-VLM-Probing. |
CRAPES:Cross-modal Annotation Projection for Visual Semantic Role Labeling (2023.starsem-1)
Copied to clipboard
| Challenge: | Existing approaches to image comprehension limit the image to a single action, while text-based approaches label all actions in a sentence. |
| Approach: | They propose to expand GSR to follow more liberal text-based approach to action and participant identification. |
| Outcome: | The proposed approach improves image comprehension on a SWiG dataset by 28.6 points. |
Guiding Zero-Shot Paraphrase Generation with Fine-Grained Control Tokens (2023.starsem-1)
Copied to clipboard
| Challenge: | Sequence-to-sequence paraphrase generation models struggle with the generation of diverse paraphrases. |
| Approach: | They propose a translation-based guided paraphrase generation model that learns useful features for promoting surface form variation in generated paraphrases from cross-lingual parallel data. |
| Outcome: | The proposed model learns useful features for promoting surface form variation in generated paraphrases from cross-lingual parallel data. |
Exploring Factual Entailment with NLI: A News Media Study (2024.starsem-1)
Copied to clipboard
| Challenge: | Recent studies have focused on the relationship between factuality and Natural Language Inference (NLI). |
| Approach: | They propose a novel annotation scheme that models factual rather than textual entailment and use it to annotate a dataset of naturally occurring sentences from news articles. |
| Outcome: | The proposed annotation scheme can be used to model factual relationships on a dataset of naturally occurring sentences from news articles. |
Query Generation Using GPT-3 for CLIP-Based Word Sense Disambiguation for Image Retrieval (2023.starsem-1)
Copied to clipboard
| Challenge: | Existing studies show that human-like prompts with quotes benefit both CLIP and GPT-3 as implicit word sense disambiguation components. |
| Approach: | They propose using the GPT-3 as a query generator for the backend of CLIP as an implicit word sense disambiguation component for the SemEval 2023 shared task Visual Word Sense Disambiguation. |
| Outcome: | The proposed query generator for CLIP is an implicit word sense disambiguation component for the SemEval 2023 shared task Visual Word Sense Disambiguation (VWSD). human-like prompts adapted for WSD with quotes benefit both CLIP and GPT-3, whereas plain phrases or poorly templated prompts give the worst results. |
Testing Paraphrase Models on Recognising Sentence Pairs at Different Degrees of Semantic Overlap (2023.starsem-1)
Copied to clipboard
| Challenge: | Existing models for paraphrase detection are not suitable for many applications . existing datasets ignore and fail to test models in this setup . |
| Approach: | They propose to use adversarial paradigms to test paraphrase detection models . they propose to examine the sensitivity to different degrees of semantic overlap . |
| Outcome: | Empirical results show that paraphrase models and different sentence encoders appear successful on evaluations, but measuring the degree of semantic overlap remains a big challenge for them. |
Comparison and Combination of Sentence Embeddings Derived from Different Supervision Signals (2022.starsem-1)
Copied to clipboard
| Challenge: | Existing methods to derive sentence embeddings have not been well understood what properties are captured in the resulting sentences depending on the supervision signals. |
| Approach: | They propose to combine two types of sentence embedding methods with similar architectures and tasks to investigate their properties. |
| Outcome: | The proposed methods perform better on unsupervised and downstream tasks than the proposed methods on untrained STS tasks and probing tasks. |
Inducing Language-Agnostic Multilingual Representations (2021.starsem-1)
Copied to clipboard
| Challenge: | Cross-lingual representations have the potential to make NLP techniques available to the vast majority of languages in the world, but they currently require large pretraining corpora or access to typologically similar languages. |
| Approach: | They propose to remove language identity signals from multilingual embeddings by re-aligning vector spaces of target languages to a pivot source language and removing language-specific means and variances. |
| Outcome: | The proposed approaches reduce cross-lingual transfer gap by 8.9 points (m-BERT) and 18.2 points (XLM-R) on average across all tasks and languages. |
Measuring Alignment Bias in Neural Seq2seq Semantic Parsers (2022.starsem-1)
Copied to clipboard
| Challenge: | Sequence-to-sequence semantic parsers with attention mechanisms have changed the research landscape . emergence of seq2seq models have led to questions about alignments . |
| Approach: | They investigate whether seq2seq models can handle both simple and complex alignments. |
| Outcome: | The proposed model performs better on monotonic and complex alignments compared to monotonic models . |