Extrinsic Evaluation of French Dependency Parsers on a Specialized Corpus: Comparison of Distributional Thesauri (2020.lrec-1)
Copied to clipboard
| Challenge: | Using a frequency-based method, we can identify subsets of the same word contexts without any reference data. |
| Approach: | They compare 11 different French dependency parsers on a specialized corpus to generate distributional thesauri using a frequency-based method. |
| Outcome: | The proposed method can identify relevant subsets without reference data and the similarity is confirmed on a restricted distributional benchmark. |
Similar Papers
Unveiling Strengths and Weaknesses of NLP Systems Based on a Rich Evaluation Corpus: The Case of NER in French (2024.lrec-main)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is an applicative task for which annotation schemes vary . a lack of robustness of some tools towards textual variation limits evaluation . |
| Approach: | They propose a gold corpus for french annotated with a rich tagset that enables comparison with multiple annotation schemes. |
| Outcome: | The proposed framework enables a fair comparison of NER systems across textual genres and annotation schemes. |
BERTrade: Using Contextual Embeddings to Parse Old French (2022.lrec-1)
Copied to clipboard
| Challenge: | a growing interest in digital humanities for automatic processing and annotation of historical texts is generating new models for historical languages. |
| Approach: | They use POS-tagging and dependency parsing to evaluate contextual word embedding models . Old French is one of the historical languages for which they have the largest amount of syntactically annotated data . |
| Outcome: | The proposed model can be used to improve performance in Old French, the authors show . they use POS-tagging and dependency parsing to evaluate the model's quality . |
Quantifying training challenges of dependency parsers (C18-1)
Copied to clipboard
| Challenge: | a new metric is introduced to evaluate the difficulty to learn a given class of dependencies . a series of systematic computations using that metric have revealed interesting properties of the 3 considered parsing algorithms . |
| Approach: | They introduce a new metric to evaluate the difficulty to learn a given class of dependencies . they use it to characterize the information conveyed by cross-lingual parsers . |
| Outcome: | The proposed metric reveals the kind of dependencies that require high effort during training . it also shows that cross-lingual parsers can provide better quality information . |
Leveraging Meta-Embeddings for Bilingual Lexicon Extraction from Specialized Comparable Corpora (C18-1)
Copied to clipboard
| Challenge: | Recent studies on bilingual lexicon extraction from specialized comparable corpora show differences in performance . lack of large specialized corporan to build efficient representations can be partially explained . |
| Approach: | They propose to use character-based embedding models to combine different embeddable models . they emphasize how character-driven embeddance models outperform other models on quality . |
| Outcome: | The proposed model outperforms other models on quality of extracted bilingual lexicons . comparable corpora are an interesting and practical alternative to parallel corporation . |
Revisiting the Effects of Leakage on Dependency Parsing (2022.findings-acl)
Copied to clipboard
| Challenge: | Recent work shows that treebank size and linguistic variation are important factors that explain the variation in dependency parsing performance. |
| Approach: | They propose a measure of leakage that explains and correlates with observed performance variation. |
| Outcome: | The proposed measure explains and correlates with observed performance variation. |
WaCadie: Towards an Acadian French Corpus (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing corpora do not exist for many languages and language varieties, such as Acadian French. |
| Approach: | They propose to build a corpus of Acadian French using web-as-corpus methodologies . they use domain crawling, social media scraping, and search engines to create corpus . |
| Outcome: | The proposed corpus includes some traces of Acadian French, but it is not available for many languages and language varieties, such as Acadinian French. |
Extending the gold standard for a lexical substitution task: is it worth it? (L18-1)
Copied to clipboard
| Challenge: | a lexical substitution task requires systems to identify words that are semantically close to the target and to select among candidates those that best fit the context. |
| Approach: | They propose to use a lexical substitution task to evaluate systems' performance . they use 300 sentences containing a target word and a second dataset based on the same data . |
| Outcome: | The proposed model is based on a set of 300 sentences containing a target word . the proposed model has not been evaluated to our knowledge . |
A Deeper Look into Dependency-Based Word Embeddings (N18-4)
Copied to clipboard
| Challenge: | Word embeddings trained with dependency contexts excel at different tasks, and enhanced dependencies often improve performance. |
| Approach: | They propose to use dependency-based word embeddings to capture semantic similarity rather than relatedness. |
| Outcome: | The results show that word embeddings trained with Universal and Stanford dependencies excel at different tasks and that enhanced dependencies often improve performance. |
Building Static Embeddings from Contextual Ones: Is It Useful for Building Distributional Thesauri? (2022.lrec-1)
Copied to clipboard
| Challenge: | contextual language models are dominant in the field of Natural Language Processing, but they are not suitable for all uses. |
| Approach: | They propose a method for building word or type-level embeddings from contextual models . they evaluate a large set of English nouns from the perspective of extracting semantic similarity relations . |
| Outcome: | The proposed method can be used to build word or type embeddings from contextual models . it can be exploited for a wide set of English nouns, showing it can improve distributional thesauri . |
Towards Robust Comparisons of NLP Models: A Case Study (2025.coling-main)
Copied to clipboard
| Challenge: | Existing statistical tests to compare the test scores of different NLP models have been proposed to account for nuisance factors such as noise, randomness, or hyperparameter values. |
| Approach: | They propose a regression analysis which isolates the effect of nuisance factors from the effects of the models’ capabilities. |
| Outcome: | The proposed model is able to show that the difference between BioLinkBERT and MSR BiomedBERT is 7 times smaller than previously reported. |