ProLex: A Benchmark for Language Proficiency-oriented Lexical Substitution (2024.findings-acl)
Copied to clipboard
| Challenge: | Lexical Substitution fails to consider substitutes of equal or higher proficiency than the target word. |
| Approach: | They propose a task to find appropriate substitutes for a given word in a context sentence but not those that are of equal or higher proficiency than the target. |
| Outcome: | The proposed model outperforms ChatGPT by an average of 3.2% in F-score and achieves comparable results with GPT-4 on ProLex. |
Similar Papers
CILex: An Investigation of Context Information for Lexical Substitution Methods (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods for lexical substitution rely on manually curated lexicals and contextual word embedding models. |
| Approach: | They propose a method that uses contextual sentence embeddings to generate substitutes for a target word given a context and a model that captures additional context information complimenting contextual word embedders. |
| Outcome: | The proposed method is state-of-the-art on the widely used LS07 and CoInCo datasets with P@1 scores of 55.96% and 57.25% for lexical substitution. |
Swords: A Benchmark for Lexical Substitution with Improved Data Coverage and Quality (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing benchmarks for lexical substitution depend on human recall as the only source of data, authors say . existing benchmarks lack coverage of the appropriate substitutes that would be most helpful to humans . |
| Approach: | They propose a benchmark for lexical substitution to find appropriate substitutes for a target word in context . existing benchmarks depend on human recall as the only source of data, they argue . |
| Outcome: | The new benchmark for lexical substitution uses a context-free thesaurus . it has 3x as many substitutes per target word for the same quality, and substitutes are 1.4x more appropriate . |
Extending the gold standard for a lexical substitution task: is it worth it? (L18-1)
Copied to clipboard
| Challenge: | a lexical substitution task requires systems to identify words that are semantically close to the target and to select among candidates those that best fit the context. |
| Approach: | They propose to use a lexical substitution task to evaluate systems' performance . they use 300 sentences containing a target word and a second dataset based on the same data . |
| Outcome: | The proposed model is based on a set of 300 sentences containing a target word . the proposed model has not been evaluated to our knowledge . |
LexSubCon: Integrating Knowledge from Lexical Resources into Contextual Embeddings for Lexical Substitution (2022.acl-long)
Copied to clipboard
| Challenge: | Lexical substitution is the task of generating meaningful substitutes for a word in a given textual context. |
| Approach: | They propose an end-to-end lexical substitution framework based on contextual embedding models that can identify highly-accurate substitute candidates. |
| Outcome: | The proposed framework outperforms state-of-the-art embedding models on LS07 and CoInCo benchmark datasets by at least 2% over existing embeddable models. |
BERT-based Lexical Substitution (P19-1)
Copied to clipboard
| Challenge: | Existing approaches to lexical substitution tend to overlook good substitute candidates that are not the synonyms of the target words in the lexicals and fail to take into account the substitution’s influence on the global context of the sentence. |
| Approach: | They propose an end-to-end BERT-based lexical substitution approach which proposes and validates substitute candidates without using annotated data or manually curated resources. |
| Outcome: | The proposed approach performs well in proposing and ranking substitute candidates, achieving the state-of-the-art results in both LS07 and LS14 benchmarks. |
Combination of Contextualized and Non-Contextualized Layers for Lexical Substitution in French (2022.lrec-1)
Copied to clipboard
| Challenge: | Lexical substitution task requires to substitute a target word by candidates in a given context. |
| Approach: | They propose a method to find synonyms for a target word and rank them based on the context of the sentence. |
| Outcome: | The proposed method increases the BERT based system on the OOT measure but decreases on the BEST measure in the SemDis 2014 benchmark. |
Always Keep your Target in Mind: Studying Semantics and Improving Performance of Neural Lexical Substitution (2020.coling-main)
Copied to clipboard
| Challenge: | Lexical substitution is a powerful technology used in various NLP applications . it generates plausible words that can replace a given word in a textual context . |
| Approach: | They propose to use a large-scale comparative study to compare lexical substitution methods . they compare existing and new methods using word sense induction datasets . |
| Outcome: | The proposed methods improve competitive results by incorporating information about the target word into the models. |
Contextualized context2vec (D19-55)
Copied to clipboard
| Challenge: | Lexical substitution ranks substitution candidates from the viewpoint of paraphrasability for a target word in a given sentence. |
| Approach: | They propose a method that combines two approaches to contextualize word embeddings for lexical substitution. |
| Outcome: | The proposed method outperforms the current state-of-the-art method and assigns English proficiency levels to all target words and substitution candidates. |
ParaLS: Lexical Substitution via Pretrained Paraphraser (2023.acl-long)
Copied to clipboard
| Challenge: | Lexical substitution (LS) is an extremely powerful technology that can be used as a backbone of various NLP applications such as writing assistance. |
| Approach: | They propose two simple decoding strategies that focus on the variations of the target word during decoding to generate substitutes from a paraphraser. |
| Outcome: | The proposed methods outperform state-of-the-art LS methods based on pre-trained language models on three benchmarks. |
ChiKhaPo: A Large-Scale Multilingual Benchmark for Evaluating Lexical Comprehension and Generation in Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for large language models (LLMs) are restricted to high- or mid-resource languages, and evaluate performance on higher-order tasks in reasoning and generation. |
| Approach: | They propose a multilingual benchmarking tool to evaluate lexical comprehension and generation abilities of large language models. |
| Outcome: | The proposed benchmarks cover 2700+ languages and surpasses existing benchmarks in terms of language coverage. |