LSC-Eval: A General Framework to Evaluate Methods for Assessing Dimensions of Lexical Semantic Change Using LLM-Generated Synthetic Data (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for measuring Lexical Semantic Change are lacking historical benchmarks. |
| Approach: | They propose a three-stage general-purpose evaluation framework that simulates theory-driven LSC using In-Context Learning and a lexical database. |
| Outcome: | The proposed framework evaluates the sensitivity of computational methods to synthetic change and their suitability for detecting change in specific dimensions and domains. |
Similar Papers
A Multidimensional Framework for Evaluating Lexical Semantic Change with Social Science Applications (2024.acl-long)
Copied to clipboard
| Challenge: | Historical linguists have identified multiple forms of lexical semantic change. |
| Approach: | They propose a framework for integrating and evaluating lexical semantic changes in historical linguists and a unified computational methodology for evaluating them concurrently. |
| Outcome: | The proposed framework enables lexical semantic change to be mapped economically and systematically and has applications in computational social science. |
A Systematic Comparison of Contextualized Word Embeddings for Lexical Semantic Change (2024.naacl-long)
Copied to clipboard
| Challenge: | Contextualized embeddings are the preferred tool for modeling Lexical Semantic Change (LSC) current evaluations focus on a specific task known as Graded Change Detection (GCD) however, performance comparisons between different approaches are often misleading due to diverse settings. |
| Approach: | They evaluate the performance of contextualized embeddings for Lexical Semantic Change (LSC) they break the problem into Word-in-Context (WiC) and Word Sense Induction (WSI) tasks . |
| Outcome: | The proposed model outperforms other models on eight available benchmarks for Lexical Semantic Change (LSC) while comparable to GPT-4. |
Definition generation for lexical semantic change detection (2024.findings-acl)
Copied to clipboard
| Challenge: | a number of studies have attempted to bridge the gap between lexical semantic change detection and sense-based LSCD methods. |
| Approach: | They propose a sense distribution based LSCD method which uses contextualized word definitions as 'senses' they argue that the method preserves interpretability and allows to inspect the reasons behind a specific shift in terms of discrete definitions-as-sense. |
| Outcome: | The proposed method outperforms previous sense-based methods on five datasets and three languages and preserves interpretability and allows to inspect the reasons behind a specific shift in terms of discrete definitions-as-senses. |
Evaluating Lexical Proficiency in Neural Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Recent advances in Natural Language Processing have been significantly shaped by the Deep Learning tsunami and the introduction of Transformer-based Language Models. |
| Approach: | They validated a framework to assess the lexical proficiency and linguistic creativity of Transformer-based Language Models (LMs) by analyzing performance of LMs of different sizes across tasks involving the generation, definition, and contextual usage of lexicals, neologisms, and nonce words. |
| Outcome: | The framework evaluates LMs in mono- and multilingual configuration across tasks involving the generation, definition, and contextual usage of lexicalized words, neologisms, and nonce words. |
Analysing Lexical Semantic Change with Contextualised Word Representations (2020.acl-main)
Copied to clipboard
| Challenge: | Existing studies on lexical semantic change have focused on detecting and characterising word meaning shifts using distributional semantic models. |
| Approach: | They propose a method that exploits the BERT neural language model to obtain representations of word usages, clusters these representations into usage types, and measures change along time with three proposed metrics. |
| Outcome: | The proposed method captures a variety of synchronic and diachronic linguistic phenomena and is highly reproducible and reproducible. |
Semantic-Eval : A Semantic Comprehension Evaluation Framework for Large Language Models Generation without Training (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have emerged as key drivers of progress in the field of natural language processing. |
| Approach: | They propose a framework that assesses LLM-generated text based on semantic understanding. |
| Outcome: | The proposed framework surpasses traditional evaluation metrics and lags behind GPT-4. |
A Wind of Change: Detecting and Evaluating Lexical Semantic Change across Times and Domains (P19-1)
Copied to clipboard
| Challenge: | Existing models for diachronic and synchronic detection of lexical semantic divergences are superficial and lack of comparison. |
| Approach: | They propose to extend benchmark models on a common state-of-the-art evaluation task . they also demonstrate that the same evaluation task and modelling approaches can be utilised for synchronic detection of domain-specific sense divergences in the field of term extraction. |
| Outcome: | The proposed model can be utilised for the detection of domain-specific sense divergences in the field of term extraction. |
EvalSense: A Framework for Domain-Specific LLM (Meta-)Evaluation (2026.eacl-demo)
Copied to clipboard
| Challenge: | EvalSense is a flexible framework for constructing domain-specific evaluation suites for large language models . it provides out-of-the-box support for a broad range of model providers and evaluation strategies . |
| Approach: | They propose a framework for constructing domain-specific evaluation suites for large language models. |
| Outcome: | The proposed framework provides out-of-the-box support for a broad range of model providers and evaluation strategies. |
XL-LEXEME: WiC Pretrained Model for Cross-Lingual LEXical sEMantic changE (2023.acl-short)
Copied to clipboard
| Challenge: | Existing approaches to the Word in Context task use cross-encoders, which prevent the possibility of deriving comparable word embeddings. |
| Approach: | They propose a Lexical Semantic Change Detection model that extends SBERT, highlighting the target word in the sentence. |
| Outcome: | The proposed model outperforms the state-of-the-art on the multilingual benchmarks for SemEval-2020 Task 1 - Lexical Semantic Change (LSC) Detection and the RuShiftEval shared task. |
Navigating the Modern Evaluation Landscape: Considerations in Benchmarks and Frameworks for Large Language Models (LLMs) (2024.lrec-tutorials)
Copied to clipboard
| Challenge: | General-purpose Language Models have changed the world of Natural Language Processing, if not the world itself. |
| Approach: | This tutorial will lay the foundations and explain the basics of evaluation and compare traditional methods to newly developed methods. |
| Outcome: | The tutorial assumes little familiarity with metrics, datasets, prompts and benchmarks . it will compare traditional methods to newly developed methods . |