Proceedings of the 13th Joint Conference on Lexical and Computational Semantics (*SEM 2024)
MASSIVE Multilingual Abstract Meaning Representation: A Dataset and Baselines for Hallucination Detection (2024.starsem-1)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (AMR) is a semantic formalism that captures the core meaning of an utterance. |
| Approach: | They propose to use AMR to map meanings of 1,685 utterances to 50+ languages to build a dataset 20 times larger than existing resources. |
| Outcome: | The proposed dataset covers more languages, has more utterances, and has localized or translated entities for each language. |
How Does Stereotype Content Differ across Data Sources? (2024.starsem-1)
Copied to clipboard
| Challenge: | Existing studies of stereotypes using rating scales capture beliefs and opinions about different social groups. |
| Approach: | They compare stereotype-relevant measures of social group social status with traditional scales and a word-list generation task using free-text data. |
| Outcome: | The results compare with traditional surveys and a spontaneous word-list generation task. |
Polysemy through the lens of psycholinguistic variables: a dataset and an evaluation of static and contextualized language models (2024.starsem-1)
Copied to clipboard
| Challenge: | Polysemes are words that can have different senses depending on context . traditionally, NLP models assume that each sense should be given a separate representation in a lexicon, thus limiting the amount of evidence that can be gained from their use. |
| Approach: | They propose a framework to model polysemes as a continuous variation in psycholinguistic properties of a word in context without postulating jumps between senses. |
| Outcome: | The proposed framework accommodates different sense interpretations, without postulating clear-cut jumps between senses. |
Post-Hoc Answer Attribution for Grounded and Trustworthy Long Document Comprehension: Task, Insights, and Challenges (2024.starsem-1)
Copied to clipboard
| Challenge: | Existing work on attribution of answer text to source document is limited . existing systems are prone to generating answers lacking sufficient grounding to knowledge sources . |
| Approach: | They propose to use existing datasets to assess the strengths and weaknesses of existing systems for this task. |
| Outcome: | The proposed system is based on retrieval-based and textual entailment-based optimal selection attribution systems. |
A Benchmark Suite of Japanese Natural Questions (2024.starsem-1)
Copied to clipboard
| Challenge: | Existing studies to solve QA tasks in an integrated manner are not available in other languages because of the lack of QA datasets. |
| Approach: | They build a Japanese version of Natural Questions using natural questions from query logs of a search engine and crowdsource it using crowdsourcing. |
| Outcome: | The proposed datasets are based on natural questions from Japanese search engines and crowdsourced. |
ROUGE-K: Do Your Summaries Have Keywords? (2024.starsem-1)
Copied to clipboard
| Challenge: | Existing evaluation metrics for extreme summarization models do not pay explicit attention to keywords in summaries, leaving developers ignorant of their presence. |
| Approach: | They propose a keyword-oriented evaluation metric, dubbed ROUGE-K, which quantifies how well summaries include keywords. |
| Outcome: | The proposed model can be extended to include more keywords while keeping the overall quality. |
Investigating Aspect Features in Contextualized Embeddings with Semantic Scales and Distributional Similarity (2024.starsem-1)
Copied to clipboard
| Challenge: | Aspect is a linguistic category describing how actions and events unfold over time. |
| Approach: | They propose to use semantic projections to examine whether the vector dimensions of annotated verbs reflect human linguistic distinctions. |
| Outcome: | The proposed models encode the aspects of stativity, durativity and telicity in most of their layers, while durativité is the most challenging feature. |
WikiScenes with Descriptions: Aligning Paragraphs and Sentences with Images in Wikipedia Articles (2024.starsem-1)
Copied to clipboard
| Challenge: | Existing work on processing image-text alignment in multimodal documents has been unsupervised, facing the challenge of missing evaluation and training data. |
| Approach: | They propose to provide one of the first datasets that provides ground-truth annotations of image-text alignments in multi-paragraph multi-image articles. |
| Outcome: | The proposed dataset can be used to study phenomena of visual language grounding in longer documents and assess retrieval capabilities of language models trained on captioning data. |
Relevance, Diversity, and Exclusivity: Designing Keyword-augmentation Strategy for Zero-shot Classifiers (2024.starsem-1)
Copied to clipboard
| Challenge: | Existing methods incorporate semantically similar keywords related to class names, but the properties of effective keywords remain unclear. |
| Approach: | They propose a method for acquiring keywords that satisfy these properties without additional knowledge bases or data. |
| Outcome: | The proposed method outperforms existing methods in fully zero-shot and generalized zero- shot settings. |
Lexical Substitution as Causal Language Modeling (2024.starsem-1)
Copied to clipboard
| Challenge: | Existing methods for lexical substitution task lacks autoregressive decoding capabilities. |
| Approach: | They propose a framework that uses causal language modeling (CLM) for lexical substitution task. |
| Outcome: | The proposed system outperforms GeneSis, the best previously published supervised LST method. |
Paraphrase Identification via Textual Inference (2024.starsem-1)
Copied to clipboard
| Challenge: | Paraphrase identification (PI) and natural language inference (NLI) are important tasks in natural language processing. |
| Approach: | They propose a method for paraphrase identification and natural language inference using an NLI system to solve these tasks. |
| Outcome: | The proposed method outperforms dedicated PI models on PI datasets and provides insights into limitations of current benchmarks. |
Identifying Emotional and Polar Concepts via Synset Translation (2024.starsem-1)
Copied to clipboard
| Challenge: | Emotion identification and polarity classification seek to determine sentiment expressed by a writer. |
| Approach: | They propose a translation-based method for labeling each individual word sense and lexical concept into 20 different languages and translate them into multilingual sentiment lexicons. |
| Outcome: | The proposed method outperforms existing methods and is available on GitHub . it contains 12,429 emotional synsets and 15,567 polar synset. |
A Closer Look at Claim Decomposition (2024.starsem-1)
Copied to clipboard
| Challenge: | Recent work uses claim decomposition to determine how well supported a claim is for applications in factual precision of generated text, entailment of human generated text and claim verification. |
| Approach: | They propose an LLM-based approach to generating decompositions inspired by Bertrand Russell’s theory of logical atomism and neo-Davidsonian semantics and demonstrate its improved decomposing quality over previous methods. |
| Outcome: | The proposed method improves on the FActScore and a Bertrand Russell-inspired approach to generating decompositions inspired by neo-Davidsonian semantics and improves decomposability quality. |
Speedy Gonzales: A Collection of Fast Task-Specific Models for Spanish (2024.starsem-1)
Copied to clipboard
| Challenge: | Large language models (LLMs) are a common and successful approach to language and retrieval tasks. |
| Approach: | They evaluate the available large language models in Spanish and then use knowledge distillation to refine and distill them. |
| Outcome: | The proposed models are fine-tuned and distilled on knowledge distillation and are available on huggingface.co/dccuchile. |
Exploring Factual Entailment with NLI: A News Media Study (2024.starsem-1)
Copied to clipboard
| Challenge: | Recent studies have focused on the relationship between factuality and Natural Language Inference (NLI). |
| Approach: | They propose a novel annotation scheme that models factual rather than textual entailment and use it to annotate a dataset of naturally occurring sentences from news articles. |
| Outcome: | The proposed annotation scheme can be used to model factual relationships on a dataset of naturally occurring sentences from news articles. |
The Emergence of High-Level Semantics in a Signaling Game (2024.starsem-1)
Copied to clipboard
| Challenge: | a symbol grounding problem has been raised in recent years in AI . we show that neural agents can communicate high-level semantic concepts . |
| Approach: | They propose to use an adversarial agent to train neural agents in a signaling game . they show that the agents can communicate high-level semantic concepts rather than low-level features . |
| Outcome: | The proposed method can learn to communicate high-level semantic concepts . it also produces an appropriate training signal when no other method is available . |
PDDLEGO: Iterative Planning in Textual Environments (2024.starsem-1)
Copied to clipboard
| Challenge: | Existing methods to plan in textual environments rely on a fully-observed environment where all entity states are known, but are not interpretable. |
| Approach: | They propose to use LLMs to generate a formal representation of the environment that can be solved by a symbolic planner. |
| Outcome: | The proposed model outperforms existing methods in the Coin Collector simulation and Cooking World simulations. |
VOLIMET: A Parallel Corpus of Literal and Metaphorical Verb-Object Pairs for English–German and English–French (2024.starsem-1)
Copied to clipboard
| Challenge: | Metaphorical language is a complex interplay of cultural and linguistic elements that characterizes metaphorical language . a corpus of parallel sentences containing gold standard alignments of metaphorical verb-object pairs and literal paraphrases is presented . |
| Approach: | They propose to analyze metaphorical verb-object pairs and literal paraphrases in parallel sentences from English to German and French. |
| Outcome: | The proposed corpus of 2,916 parallel sentences reveals monolingual patterns for metaphorical vs. literal uses in English . cross-lingually, the results show a rich variability in translations as well as different behaviors for the two target languages . |
Improving Word Sense Induction through Adversarial Forgetting of Morphosyntactic Information (2024.starsem-1)
Copied to clipboard
| Challenge: | Contextualized word representations from pre-trained language models encode more information than is necessary for the identification of word senses and some of this information affect performance negatively in unsupervised settings. |
| Approach: | They propose to use a framework to erase specific information from pre-trained word models and create feature-invariant representations that are invariant to these ‘nuisance features’. |
| Outcome: | The proposed framework erases information from the representations of pre-trained language models, thereby creating feature-invariant representations. |
What’s wrong with your model? A Quantitative Analysis of Relation Classification (2024.starsem-1)
Copied to clipboard
| Challenge: | A major trend in NLP research aims at designing more sophisticated setups to improve the state-of-the-art (SOTA) on a target task. |
| Approach: | They propose an in-depth analysis suite for Relation Classification to be used for prediction tasks. |
| Outcome: | The proposed model improves over the baseline by >3 Micro-F1 . the proposed model is based on a case study and a preliminary error-guided analysis . |
Disambiguating Emotional Connotations of Words Using Contextualized Word Representations (2024.starsem-1)
Copied to clipboard
| Challenge: | BERT, RoBERTa, XLNet, and GPT-2 models effectively discern emotional connotations of words, demonstrating superior performance and greater resilience against biases. |
| Approach: | They propose to use contextualized word representations to examine how words can be used to distinguish emotional connotations across contexts. |
| Outcome: | The proposed models show that they can distinguish emotional connotations of words in different contexts. |
Length-Aware Multi-Kernel Transformer for Long Document Classification (2024.starsem-1)
Copied to clipboard
| Challenge: | Existing SOTA models segment long texts into equal-length snippets, but they have new challenges of context fragmentation and generalizability due to sentence boundaries and varying text lengths. |
| Approach: | They propose a Length-Aware Multi-Kernel Transformer to encode long documents by transformers and vectorize text length by the kernels to promote model robustness over varying document lengths. |
| Outcome: | The proposed model outperforms existing models on five benchmarks from health and law domains up to an absolute 10.9% improvement. |
Investigating Wit, Creativity, and Detectability of Large Language Models in Domain-Specific Writing Style Adaptation of Reddit’s Showerthoughts (2024.starsem-1)
Copied to clipboard
| Challenge: | Recent Large Language Models (LLMs) have shown the ability to generate content that is difficult or impossible to distinguish from human writing. |
| Approach: | They compare GPT-2 and GPT-Neo fine-tuned on Reddit data and GTP-3.5 invoked in a zero-shot manner, against human-authored texts. |
| Outcome: | The proposed model can generate short, creative texts that are difficult to distinguish from human writing, but human evaluators rate them worse than the model. |
Multilingual and Code-Switched Sentence Ordering (2024.starsem-1)
Copied to clipboard
| Challenge: | Prior research has focused on English language structures and multilingual contexts . however, there are several shortcomings with specialized sentence ordering models and advanced Large Language Models like GPT-4. |
| Approach: | They propose a multilingual sentence order task that extends SO to diverse narratives across 12 languages and code-switched texts. |
| Outcome: | The proposed task extends SO to diverse narratives across 12 languages, including challenging code-switched texts. |
HANS, are you clever? Clever Hans Effect Analysis of Neural Systems (2024.starsem-1)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been exhibiting outstanding abilities to reason around cognitive states, intentions, and reactions of all people involved, letting humans guide and comprehend day-to-day social interactions effectively. |
| Approach: | They propose to use multiple-choice questions (MCQ) benchmarks to assess LLMs' ability to reason around cognitive states, intentions, and reactions of all people involved to investigate their resilience abilities. |
| Outcome: | The proposed models exhibit exceptional abilities to reason around cognitive states, intentions, and reactions of all people involved, letting humans guide and comprehend day-to-day social interactions effectively. |
Exploring Semantics in Pretrained Language Model Attention (2024.starsem-1)
Copied to clipboard
| Challenge: | Abstract Meaning Representations (AMRs) encode the semantics of sentences in the form of graphs. |
| Approach: | They propose to use attention heads of two LMs to detect semantic relations encoded in AMRs. |
| Outcome: | The proposed models detect semantic relations without fine tuning, using both unsupervised and supervised learning techniques. |
Enhancing Self-Attention via Knowledge Fusion: Deriving Sentiment Lexical Attention from Semantic-Polarity Scores (2024.starsem-1)
Copied to clipboard
| Challenge: | Existing methods to inject lexical features into self-attention mechanisms have shown remarkable performance across various downstream tasks in NLP. |
| Approach: | They propose to inject lexical features into the self-attention mechanism of Transformer-based models by injecting lexicon-based Sentiment Lexical Attention into the attention scores throughout the training process. |
| Outcome: | The proposed method shows significant performance improvements on the NSMC sentiment classification benchmark and is able to perform in out-of-domain tasks. |
Handling Ontology Gaps in Semantic Parsing (2024.starsem-1)
Copied to clipboard
| Challenge: | Existing methods to detect hallucinations in closed-ontology models are limited by ontology gaps. |
| Approach: | They propose a framework for stimulating and analyzing NSP model hallucinations . they propose 'hallucination simulation framework' to detect hallucinosities in presence of ontology gaps . |
| Outcome: | The proposed framework improves the F1-Score and the IQ Pro benchmark datasets. |
PipeNet: Question Answering with Semantic Pruning over Knowledge Graphs (2024.starsem-1)
Copied to clipboard
| Challenge: | Existing approaches to utilizing explicit knowledge graphs (KGs) are limited by the number of nodes in the subgraph. |
| Approach: | They propose a grounding-pruning-reasoning pipeline to prune noisy nodes in subgraphs to improve the efficiency of graph reasoning with KG. |
| Outcome: | The proposed method reduces computation cost and memory usage while obtaining decent representation of pruned subgraphs. |
A Trip Towards Fairness: Bias and De-Biasing in Large Language Models (2024.starsem-1)
Copied to clipboard
| Challenge: | a little or a large bias in CtB-LLMs may cause huge harm . LLaMA and OPT families have an important bias in gender, race, religion, and profession. |
| Approach: | They propose to debiase three families of Very Large-Language Models with LORA to reduce bias by 4.12 points in the normalized stereotype score. |
| Outcome: | The proposed model reduces bias up to 4.12 points in the normalized stereotype score. |
Compositional Structured Explanation Generation with Dynamic Modularized Reasoning (2024.starsem-1)
Copied to clipboard
| Challenge: | Large-scale language models have shown remarkable performance on reasoning tasks such as reading comprehension, natural language inference, story generation, etc. |
| Approach: | They propose a compositional structured explanation generation task to test a model's ability to generalize from generating entailment trees to more steps, focusing on the length and shapes of engorgement trees. |
| Outcome: | The proposed model shows competitive compositional generalization abilities in a generation setting. |
Inspecting Soundness of AMR Similarity Metrics in terms of Equivalence and Inequivalence (2024.starsem-1)
Copied to clipboard
| Challenge: | Existing Abstract Meaning Representation (AMR) similarity metrics have less investigated their soundness . |
| Approach: | They propose a new experimental method to evaluate soundness of AMR similarity metrics in terms of equivalence and inequivalentity. |
| Outcome: | The proposed method satisfies the soundness criteria of existing AMR similarity metrics and improves them by proposing a revised metric, SMATCH . |
Sõnajaht: Definition Embeddings and Semantic Search for Reverse Dictionary Creation (2024.starsem-1)
Copied to clipboard
| Challenge: | Existing systems that use exact term matching to find words are based on information retrieval. |
| Approach: | They propose to use pre-trained language models and approximate nearest neighbors search algorithms to enhance and enrich an Estonian lexicon resource by introducing cross-lingual reverse dictionary functionality powered by semantic search. |
| Outcome: | The proposed system produces a 1 and 2 routs in the monolingual and cross-lingual settings using the unlabeled evaluation approach. |
Do large language models and humans have similar behaviours in causal inference with script knowledge? (2024.starsem-1)
Copied to clipboard
| Challenge: | Recent studies show pre-trained language models have superior language understanding abilities, including zero-shot causal reasoning. |
| Approach: | They used a script-based story to manipulate event B in a story which causally depends on a previous event A. |
| Outcome: | The results show that only recent LLMs, like GPT-3 or Vicuna, correlate with human behavior in the A B condition. |
EDM3: Event Detection as Multi-task Text Generation (2024.starsem-1)
Copied to clipboard
| Challenge: | Existing methods for Event Detection (ED) cannot easily leverage pre-trained semantic knowledge. |
| Approach: | They propose to decompose and reformulate ED and fine-tune over its atomic subtasks to enhance knowledge transfer while mitigating prediction error propagation inherent in pipelined approaches. |
| Outcome: | The proposed method achieves state-of-the-art performance on RAMS, MAVEN, and MLEE, while achieving 90% accuracy over rare event types. |