Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023)

Revisiting Syntax-Based Approach in Negation Scope Resolution (2023.starsem-1)

Challenge:	a recent study of the effect of visual grounding on language representations has given a new life to the debate around extractability and quality of semantic information in representations trained solely on textual input.
Approach:	They compare word embeddings from vision-and-language models to text-only models . they identify meaning properties and relations that characterize words whose embeddements are most affected by visual grounding .
Outcome:	The proposed model differs from text-only models on semantic representations of language . the study is the first large-scale study of the effect of visual grounding on language representations .

Copied to clipboard

Asahi Yoshida, Yoshihide Kato, Shigeki Matsubara

Syntactic Text Processing

Challenge:	Existing methods for negation scope resolution without syntactic structure performed better.
Approach:	They re-evaluate the effectiveness of syntactic structure in negation scope resolution . they replace the parser employed in previous studies with state-of-the-art parsers .
Outcome:	The proposed method performs better without syntactic structure than previous methods . the proposed method is comparable to state-of-the-art methods based on heuristic rules .

When Truth Matters - Addressing Pragmatic Categories in Natural Language Inference (NLI) by Large Language Models (LLMs) (2023.starsem-1)

Copied to clipboard

Reto Gubelmann, Aikaterini-lida Kalouli, Christina Niklaus, Siegfried Handschuh

Analyzing Syntactic Generalization Capacity of Pre-trained Language Models on Japanese Honorific Conversion (2023.starsem-1)

Challenge:	In this paper, we examine the ability of large language models (LLMs) to accommodate different pragmatic sentence types, such as questions, commands, and sentence fragments for natural language inference (NLI).
Approach:	They propose to fine-tune large language models to accommodate different sentence types for natural language inference (NLI) they also explore ChatGPT's concept of entailment by using a symbolic semantic parser.
Outcome:	The proposed models can accommodate different sentence types without losing too much accuracy on MNLI-matched models.

Copied to clipboard

Ryo Sekizawa, Hitomi Yanaka

Improving Toponym Resolution with Better Candidate Generation, Transformer-based Reranking, and Two-Stage Resolution (2023.starsem-1)

Challenge:	Using Japanese honorifics requires knowledge of grammatical rules and contextual information, such as social relationships.
Approach:	They propose a Japanese honorific conversion task that considers social relationships among people mentioned in a conversation.
Outcome:	The proposed model performed better on the context-aware task than the prompt-based one.

Copied to clipboard

Zeyu Zhang, Steven Bethard

CRAPES:Cross-modal Annotation Projection for Visual Semantic Role Labeling (2023.starsem-1)

Challenge:	Geocoding is the task of converting location mentions in text into structured data that encodes the geospatial semantics.
Approach:	They propose a new architecture that generates a list of geospatial ontologies and reranks them using a transformer-based neural network.
Outcome:	The proposed architecture achieves state-of-the-art on multiple datasets.

Copied to clipboard

Abhidip Bhattacharyya, Martha Palmer, Christoffer Heckman

Multimodality

Challenge:	Existing approaches to image comprehension limit the image to a single action, while text-based approaches label all actions in a sentence.
Approach:	They propose to expand GSR to follow more liberal text-based approach to action and participant identification.
Outcome:	The proposed approach improves image comprehension on a SWiG dataset by 28.6 points.

Not All Counterhate Tweets Elicit the Same Replies: A Fine-Grained Analysis (2023.starsem-1)

Copied to clipboard

Abdullah Albanyan, Ahmed Hassan, Eduardo Blanco

Challenge:	a recent survey found 41% of people reported online harassment on a personal level . a counterhate argument can effectively limit the spread of hate speech, but it can also exacerbate it .
Approach:	They analyze 2,621 replies to counterhate arguments countering hateful tweets and analyze their responses . they find that half of the replies disagree with the argument, and this kind of reply often supports the hateful Tweet .
Outcome:	The proposed method can anticipate the kind of replies a counterhate argument will elicit.

Evaluating Factual Consistency of Texts with Semantic Role Labeling (2023.starsem-1)

Copied to clipboard

Jing Fan, Dennis Aumiller, Michael Gertz

Language models are not naysayers: an analysis of language models on negation benchmarks (2023.starsem-1)

Challenge:	Existing evaluation methods rely on task-specific language models, which in turn hampers interpretation of generated scores.
Approach:	They propose a reference-free evaluation metric for text summarization that measures factuality . their method generates fact tuples from Semantic Role Labels, applied to both input and summary texts.
Outcome:	The proposed evaluation metric is comparable with state-of-the-art methods and has a stable generalization across datasets.

Copied to clipboard

Thinh Hung Truong, Timothy Baldwin, Karin Verspoor, Trevor Cohn

JSEEGraph: Joint Structured Event Extraction as Graph Parsing (2023.starsem-1)

Challenge:	Negation has been shown to be a major bottleneck for masked language models, such as BERT, but whether this finding still holds for larger-sized auto-regressive language models has not been studied comprehensively.
Approach:	They evaluate the ability of current-generation auto-regressive language models to handle negation using a wide range of benchmarks and models.
Outcome:	The proposed models are compared against a wide range of negation benchmarks and show that they are insensitive to negation, inability to capture the lexical semantics of negations, and failure to reason under negation.

Copied to clipboard

Huiling You, Lilja Vrelid, Samia Touileb

Information Extraction & Text Mining

Challenge:	Existing approaches model event extraction using simplified datasets or sequence-labeling-based encodings.
Approach:	They propose a graph-based event extraction framework that explicitly encodes entities and events in a single semantic graph.
Outcome:	The proposed framework can handle nested event structures and solve different IE tasks jointly.

Generative Data Augmentation for Aspect Sentiment Quad Prediction (2023.starsem-1)

Copied to clipboard

An Wang, Junfeng Jiang, Youmi Ma, Ao Liu, Naoaki Okazaki

Sentiment Analysis

Challenge:	Existing approaches to analyze text contain rewrites and inconsistency between text and quads.
Approach:	They propose a new approach to analyze aspect terms, opinion terms, sentiment polarity in text . they augment quads and train a quads-to-text model to generate corresponding texts .
Outcome:	The proposed method outperforms existing methods and achieves state-of-the-art performance on two datasets.

Are Language Models Sensitive to Semantic Attraction? A Study on Surprisal (2023.starsem-1)

Copied to clipboard

Yan Cong, Emmanuele Chersoni, Yu-yin Hsu, Alessandro Lenci

Syntax and Semantics Meet in the “Middle”: Probing the Syntax-Semantics Interface of LMs Through Agentivity (2023.starsem-1)

Challenge:	In psycholinguistics, semantic attraction is a sentence processing phenomenon in which a given argument violates the selectional requirements of a verb but is not perceived by comprehenders due to its attraction to another noun in the same sentence.
Approach:	They used autoregressive language models to compute the sentence-level and target phrase-level Surprisal scores of a psycholinguistic dataset on semantic attraction.
Outcome:	The proposed models are sensitive to semantic attraction, leading to reduced Surprisal scores, although none perfectly matches the human behavioral pattern.

Copied to clipboard

Lindia Tjuatja, Emmy Liu, Lori Levin, Graham Neubig

Can Pretrained Language Models Derive Correct Semantics from Corrupt Subwords under Noise? (2023.starsem-1)

Challenge:	a recent study examined how large language models handle interactions in meaning across words and larger syntactic forms.
Approach:	They propose to use a dataset to examine the linguistic properties of optionally transitive English verbs to examine their agentivity.
Outcome:	The proposed model outperforms all other models in the evaluation dataset . the results are better correlated with human judgements than syntactic and semantic corpus statistics .

Copied to clipboard

Xinzhe Li, Ming Liu, Shang Gao

How Are Idioms Processed Inside Transformer Language Models? (2023.starsem-1)

Challenge:	Existing studies have shown that Pretrained Language Models (PLMs) perform poorly under noise due to subword segmentation.
Approach:	They propose a framework for subword segmentation that provides a systematic categorization of segmentation corruption under noise and evaluation protocols by generating contrastive datasets with canonical-noisy word pairs.
Outcome:	The proposed framework provides a systematic categorization of segmentation corruption under noise and evaluation protocols by generating contrastive datasets with canonical-noisy word pairs.

Copied to clipboard

Ye Tian, Isobel James, Hye Son

Is Shortest Always Best? The Role of Brevity in Logic-to-Text Generation (2023.starsem-1)

Challenge:	idioms are prevalent in natural language, but how do they be processed?
Approach:	They analyze the embeddings of idiomatic and literal expressions across all layers of the networks at both the sentence and word levels.
Outcome:	The proposed models represent idioms distinctively compared to literal language, the study finds .

Copied to clipboard

Eduardo Calò, Jordi Levy, Albert Gatt, Kees Van Deemter

Seeking Clozure: Robust Hypernym extraction from BERT with Anchored Prompts (2023.starsem-1)

Challenge:	Logical formulae are essential for scholars in many fields, including linguistics and artificial intelligence.
Approach:	They propose to use a Quantified Boolean Formulae (QBFs) problem to find the shortest formulae as input for a "logic-to-text" generation system.
Outcome:	The proposed approach improves the comprehensibility and fluency of the generated texts.

Copied to clipboard

Chunhua Liu, Trevor Cohn, Lea Frermann

LEXPLAIN: Improving Model Explanations via Lexicon Supervision (2023.starsem-1)

Challenge:	Existing methods for extracting hypernym knowledge from large language models are unclear whether they fail due to a lack of knowledge or shortcomings.
Approach:	They propose to use pattern-based hypernym extraction as a diagnostic tool to examine hypernomy knowledge encoded in BERT.
Outcome:	The proposed method compares the results of two different methods on six English data sets and on challenge sets of rare and abstract concepts.

Copied to clipboard

Orevaoghene Ahia, Hila Gonen, Vidhisha Balachandran, Yulia Tsvetkov, Noah A. Smith

KGLM: Integrating Knowledge Graph Structure in Language Models for Link Prediction (2023.starsem-1)

Challenge:	Existing methods that extract features from input text to explain a classifier's prediction are limiting to models that are faithful to their predictions.
Approach:	They propose a framework for guiding model explanations by supervising them explicitly using task-related lexicons to direct supervise model explanation.
Outcome:	The proposed method improves model explanations without sacrificing performance on sentiment analysis and toxicity detection tasks while demoting spurious correlations with African American English dialects.

Copied to clipboard

Jason Youn, Ilias Tagkopoulos

Probing Out-of-Distribution Robustness of Language Models with Parameter-Efficient Transfer Learning (2023.starsem-1)

Challenge:	Knowledge graphs are incomplete in the information they represent, necessitating knowledge graph completion tasks.
Approach:	They propose a new entity/relation embedding layer that learns to differentiate distinctive entity and relation types, thus allowing the model to learn the structure of the knowledge graph.
Outcome:	The proposed language model learns to differentiate distinct entity and relation types, thus learning the structure of the knowledge graph.

Copied to clipboard

Hyunsoo Cho, Choonghyun Park, Junyeob Kim, Hyuhng Joon Kim, Kang Min Yoo, Sang-goo Lee

Limits for learning with language models (2023.starsem-1)

Challenge:	Pre-trained language models (PLMs) are gaining popularity on many benchmarks, but it is uncertain whether they can handle inputs that have been distributionally shifted.
Approach:	They evaluated various PETL techniques to detect out-of-distribution changes as the size of the PLM grows or the transfer methods are altered.
Outcome:	The proposed methods can detect out-of-distribution changes as the size of the PLM grows or the transfer methods are altered.

Copied to clipboard

Nicholas Asher, Swarnadeep Bhar, Akshay Chaturvedi, Julie Hunter, Soumya Paul

Does Character-level Information Always Improve DRS-based Semantic Parsing? (2023.starsem-1)

Challenge:	Recent studies show that large language models fail to capture important aspects of linguistic meaning . authors argue that LLMs cannot learn fundamental semantic properties defined in formal semantics .
Approach:	They propose a theoretical explanation for some of the observed failings of large language models . they show that LLMs cannot learn certain fundamental semantic properties .
Outcome:	The proposed model fails to learn semantic entailment and consistency as defined in formal semantics, the authors argue . their model fails on tasks that require engorgements and deep linguistic understanding, they argue - but not on universal quantification.

Copied to clipboard

Tomoya Kurosawa, Hitomi Yanaka

Testing Paraphrase Models on Recognising Sentence Pairs at Different Degrees of Semantic Overlap (2023.starsem-1)

Challenge:	incorporating character-level information does not improve the performance in English and German, and is not sensitive to correct character order in Dutch.
Approach:	They propose to incorporate character-level representations into a neural semantic parser for Discourse Representation Structures and to test their performance using order of character sequences.
Outcome:	The proposed parser improves in English, German, Dutch, and Italian in four languages.

Copied to clipboard

Qiwei Peng, David Weir, Julie Weeds

„Mann“ is to “Donna” as「国王」is to « Reine » Adapting the Analogy Task for Multilingual and Contextual Embeddings (2023.starsem-1)

Challenge:	Existing models for paraphrase detection are not suitable for many applications . existing datasets ignore and fail to test models in this setup .
Approach:	They propose to use adversarial paradigms to test paraphrase detection models . they propose to examine the sensitivity to different degrees of semantic overlap .
Outcome:	Empirical results show that paraphrase models and different sentence encoders appear successful on evaluations, but measuring the degree of semantic overlap remains a big challenge for them.

Copied to clipboard

Timothee Mickus, Eduardo Calò, Léo Jacqmin, Denis Paperno, Mathieu Constant

Representation Learning

Challenge:	a lack of comparable multilingual benchmarks and a consensual evaluation protocol for contextual models remains an open question.
Approach:	They propose a multilingual analogy dataset and evaluate human and contextual embedding performance.
Outcome:	The proposed dataset evaluates human and contextual embedding models on the analogy task.

Scalable Performance Analysis for Vision-Language Models (2023.starsem-1)

Copied to clipboard

Santiago Castro, Oana Ignat, Rada Mihalcea

PCFG-Based Natural Language Interface Improves Generalization for Controlled Text Generation (2023.starsem-1)

Challenge:	a new method to probe vision-language models is proposed that does not require data annotation and makes use of existing datasets.
Approach:	They propose a method that extracts features from a vision-language benchmark and measures their correlation with the output of the target model.
Outcome:	The proposed method is scalable and does not require data annotation . it can be used with other models and benchmarks, and is available at https://github.com/MichiganNLP/Scalable-VLM-Probing.

Copied to clipboard

Jingyu Zhang, James Glass, Tianxing He

True Detective: A Deep Abductive Reasoning Benchmark Undoable for GPT-3 and Challenging for GPT-4 (2023.starsem-1)

Challenge:	Existing work on controlled text generation (CTG) assumes a control interface of categorical attributes.
Approach:	They propose a natural language interface to embed control attributes into natural language commands and propose variants of existing CTG models that take commands as input.
Outcome:	The proposed model can generalize to unseen attributes and unsealed attribute combinations.

Copied to clipboard

Maksym Del, Mark Fishel

Guiding Zero-Shot Paraphrase Generation with Fine-Grained Control Tokens (2023.starsem-1)

Challenge:	Large language models (LLMs) have demonstrated solid zero-shot reasoning capabilities, which is reflected in their performance on the current test tasks.
Approach:	They propose a benchmark consisting of 191 long-form mystery narratives constructed as detective puzzles.
Outcome:	The proposed benchmark outperforms random models on the current test tasks while state-of-the-art models only solve 38% of puzzles.

Copied to clipboard

Teemu Vahtola, Mathias Creutz, Jrg Tiedemann

A Tale of Two Laws of Semantic Change: Predicting Synonym Changes with Distributional Semantic Models (2023.starsem-1)

Challenge:	Sequence-to-sequence paraphrase generation models struggle with the generation of diverse paraphrases.
Approach:	They propose a translation-based guided paraphrase generation model that learns useful features for promoting surface form variation in generated paraphrases from cross-lingual parallel data.
Outcome:	The proposed model learns useful features for promoting surface form variation in generated paraphrases from cross-lingual parallel data.

Copied to clipboard

Bastien Lietard, Mikaela Keller, Pascal Denis

Challenge:	Lexical Semantic Change is the study of how the meaning of words evolves through time.
Approach:	They propose to use distributional models to detect whether LD or LPC operate for given word pairs.
Outcome:	The proposed frameworks achieve a balanced accuracy above 0.6 on the dataset.

Semantically-informed Hierarchical Event Modeling (2023.starsem-1)

Copied to clipboard

Shubhashis Roy Dipta, Mehdi Rezaee, Francis Ferraro

Knowledge Representation

Challenge:	Existing approaches to event modeling combine sequential latent variables with semantic ontological knowledge to improve representational capabilities.
Approach:	They propose a doubly hierarchical semi-supervised event modeling framework that provides structural hierarchy while accounting for ontological hierarchy.
Outcome:	The proposed model outperforms state-of-the-art models by 8.5% across two datasets and four metrics.

Representation of Lexical Stylistic Features in Language Models’ Embedding Space (2023.starsem-1)

Copied to clipboard

Qing Lyu, Marianna Apidianaki, Chris Callison-burch

Event Semantic Knowledge in Procedural Text Understanding (2023.starsem-1)

Challenge:	lexical stylistic notions such as complexity, formality, and figurativeness can be identified in pretrained Language Models . static embeddings encode these features more accurately at the level of words and phrases whereas contextualized LMs perform better on sentences.
Approach:	They propose to derive a vector representation for stylistic notions from seed pairs . they find that static embeddings encode stylistic features more accurately .
Outcome:	The proposed representations can be used to characterize new texts in terms of these dimensions using a small number of seed pairs.

Copied to clipboard

Ghazaleh Kazeminejad, Martha Palmer

Information Retrieval

Challenge:	Annotators’ reliance on commonsense knowledge to annotate implicit state information is a challenge for entity state tracking.
Approach:	They propose a method for entity state tracking that incorporates commonsense entity-centric knowledge from ConceptNet into a BERT-based neural-symbolic architecture.
Outcome:	The proposed model outperforms existing models on the ProPara dataset and is domain-agnostic.

Leveraging Active Learning to Minimise SRL Annotation Across Corpora (2023.starsem-1)

Copied to clipboard

Skatje Myers, Martha Palmer

Estimating Semantic Similarity between In-Domain and Out-of-Domain Samples (2023.starsem-1)

Challenge:	In this paper, we investigate the application of active learning to semantic role labeling (SRL) using Bayesian Active Learning by Disagreement (BALD).
Approach:	They propose a sentence-focused selection method that is based off of previous methods of using model dropout to approximate a Gaussian process for SRL.
Outcome:	The proposed selection method improves on three different domain corpora on three domains with a large and diverse corpus.

Copied to clipboard

Rhitabrat Pokharel, Ameeta Agrawal

Query Generation Using GPT-3 for CLIP-Based Word Sense Disambiguation for Image Retrieval (2023.starsem-1)

Challenge:	Prior work typically defines out-of-domain (OOD) or out- of-distribution (OOdist) samples as those that originate from dataset(s) or source(s), but for the same task.
Approach:	They propose to use supervised methods to identify OOD/OODist samples without using a trained model.
Outcome:	The proposed method is able to identify OOD/OODist samples without a trained model.

Copied to clipboard

Xiaomeng Pan, Zhousi Chen, Mamoru Komachi

Functional Distributional Semantics at Scale (2023.starsem-1)

Challenge:	Existing studies show that human-like prompts with quotes benefit both CLIP and GPT-3 as implicit word sense disambiguation components.
Approach:	They propose using the GPT-3 as a query generator for the backend of CLIP as an implicit word sense disambiguation component for the SemEval 2023 shared task Visual Word Sense Disambiguation.
Outcome:	The proposed query generator for CLIP is an implicit word sense disambiguation component for the SemEval 2023 shared task Visual Word Sense Disambiguation (VWSD). human-like prompts adapted for WSD with quotes benefit both CLIP and GPT-3, whereas plain phrases or poorly templated prompts give the worst results.

Copied to clipboard

Chun Hei Lo, Hong Cheng, Wai Lam, Guy Emerson

Challenge:	Functional Distributional Semantics is a linguistically motivated framework for modelling lexical and sentence-level semantics with truth-conditional functions using distributional information.
Approach:	They propose a more expressive lexical model that works over a continuous semantic space.
Outcome:	The proposed model improves performance and flexibility and is compatible with present-day machine learning frameworks.

FEED PETs: Further Experimentation and Expansion on the Disambiguation of Potentially Euphemistic Terms (2023.starsem-1)

Copied to clipboard

Patrick Lee, Iyanuoluwa Shode, Alain Trujillo, Yuan Zhao, Olumide Ojo, Diana Plancarte, Anna Feldman, Jing Peng

Monolingual Phrase Alignment as Parse Forest Mapping (2023.starsem-1)

Challenge:	Existing work on euphemism disambiguation tasks has focused on transformers . euphorias are expressions that soften the message they convey, therefore dictionary-based approaches are ineffective .
Approach:	They propose to annotate PETs for vagueness and use transformers to classify PETs . they perform euphemism disambiguation experiments in three different languages .
Outcome:	The proposed models perform well in English euphemism disambiguation task . preliminary results will be used to launch future work .

Copied to clipboard

Sora Kadotani, Yuki Arase

Syntactic Text Processing

Challenge:	Existing methods for phrase alignment are based on unordered tree mapping . syntactic ambiguities can affect alignment quality, so we expand it to parse forests instead of 1-best trees.
Approach:	They propose to expand existing method to align parse forests rather than 1-best trees, where syntactic structures and phrase alignment are simultaneously identified.
Outcome:	The proposed method improves the state-of-the-art method by aligning forests rather than 1-best trees.

Empirical Sufficiency Lower Bounds for Language Modeling with Locally-Bootstrapped Semantic Structures (2023.starsem-1)

Copied to clipboard

Jakob Prange, Emmanuele Chersoni

Probing neural language models for understanding of words of estimative probability (2023.starsem-1)

Challenge:	a recent attempt at language modeling with predicted semantic structure failed to establish empirical lower bounds on what could have made the attempt successful.
Approach:	They propose a concise binary vector representation of semantic structure at the lexical level and evaluate how good an incremental tagger needs to be to achieve better-than-baseline performance.
Outcome:	The proposed model can achieve better-than-baseline performance without losing its main advantages and lower bounds on prediction quality can't be established via a single score alone.

Copied to clipboard

Damien Sileo, Marie-francine Moens

Arithmetic-Based Pretraining Improving Numeracy of Pretrained Language Models (2023.starsem-1)

Challenge:	Words of Estimative Probability (WEP) are phrases used to express the plausibility of a statement.
Approach:	They propose to use a UNLI dataset to assess language models' ability to process WEPs.
Outcome:	The proposed model can accurately capture the consensual probability level associated with each WEP.

Copied to clipboard

Dominic Petrak, Nafise Sadat Moosavi, Iryna Gurevych

Robust Integration of Contextual Information for Cross-Target Stance Detection (2023.starsem-1)

Challenge:	Recent work suggests that pretrained language models perform below their capabilities when applied out-of-the-box on tasks that require understanding and working with numbers.
Approach:	They propose an extended pretraining approach that addresses both in one extended step . they propose a novel extended pre training objective called Inferable Number Prediction Task to improve numeracy.
Outcome:	The proposed approach improves reading comprehension and inference-on-tables tasks without architectural changes or pretraining from scratch.

Copied to clipboard

Tilman Beck, Andreas Waldis, Iryna Gurevych

Adverbs, Surprisingly (2023.starsem-1)

Challenge:	Existing stance detection models do not take into account relevant contextual information which allows for inferring the stance correctly.
Approach:	They propose an approach to integrate contextual information as text into pretrained language models by prompting large language models.
Outcome:	The proposed approach outperforms baselines on a large and diverse stance detection benchmark in a cross-target setup, i.e. for targets unseen during training.

Copied to clipboard

Dmitry Nikolaev, Collin Baker, Miriam R. L. Petruck, Sebastian Padó

Challenge:	adverbs are the part of speech (POS) that has seen the least attention in computational linguistics due to its challenging nature.
Approach:	They propose to use Frame Semantics to characterize word meaning to uncover systematic gaps in adverb accounts.
Outcome:	The proposed approach can describe ambiguity, semantic roles, and null instantiation of adverbs.

Can Sequence-to-Sequence Transformers Naturally Understand Sequential Instructions? (2023.starsem-1)

Copied to clipboard

Xiang Zhou, Aditya Gupta, Shyam Upadhyay, Mohit Bansal, Manaal Faruqui