| Challenge: | Recent work shows that treebank size and linguistic variation are important factors that explain the variation in dependency parsing performance. |
| Approach: | They propose a measure of leakage that explains and correlates with observed performance variation. |
| Outcome: | The proposed measure explains and correlates with observed performance variation. |
Similar Papers
Some Languages Seem Easier to Parse Because Their Treebanks Leak (2020.emnlp-main)
Copied to clipboard
| Challenge: | Cross-language differences in (universal) dependency parsing performance are mostly attributed to treebank size, average sentence length, average dependency length, morphological complexity, and domain differences. |
| Approach: | They compute graph isomorphisms and find that treebank size is a factor that influences parsing performance. |
| Outcome: | The results show that the overlap between training and test graphs explain more of the observed variation than standard explanations such as the above. |
Investigating How Pre-training Data Leakage Affects Models’ Reproduction and Detection Capabilities (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies do not examine how leaked instances in training datasets influence LLMs’ output and detection capabilities. |
| Approach: | They conduct an experimental survey to examine the relationship between data leakage in training datasets and its effects on the generation and detection by Large Language Models (LLMs). |
| Outcome: | The results show that enhancing leakage detection through few-shot learning can help mitigate the impact of the leakage rate in the training data on detection performance. |
Replicating and Extending “Because Their Treebanks Leak”: Graph Isomorphism, Covariants, and Parser Performance (2021.acl-short)
Copied to clipboard
| Challenge: | a small sample size and unreliable results suggest a correlation between parser performance and graph isomorphism is not observed in the wild. |
| Approach: | They propose to replicate a study which found graph isomorphism is a non-trivial variable . they also bin sentences by length and find correlation between parser performance and isopathism disappears . |
| Outcome: | The results show that the original analysis was unreliable and had methodological issues . the study also bin sentences by length and shows that the correlation between parser performance and graph isomorphism disappears when controlling for covariants. |
Memorization vs. Generalization : Quantifying Data Leakage in NLP Performance Evaluation (2021.eacl-main)
Copied to clipboard
| Challenge: | Public datasets are often used to evaluate the efficacy and generalizability of state-of-the-art methods for many tasks in natural language processing (NLP). |
| Approach: | They identify leakage of training data into test data on several publicly available datasets used to evaluate NLP tasks, including named entity recognition and relation extraction. |
| Outcome: | The proposed model can memorize and generalize data on several publicly available datasets and is compared against previously unseen data. |
Quantifying training challenges of dependency parsers (C18-1)
Copied to clipboard
| Challenge: | a new metric is introduced to evaluate the difficulty to learn a given class of dependencies . a series of systematic computations using that metric have revealed interesting properties of the 3 considered parsing algorithms . |
| Approach: | They introduce a new metric to evaluate the difficulty to learn a given class of dependencies . they use it to characterize the information conveyed by cross-lingual parsers . |
| Outcome: | The proposed metric reveals the kind of dependencies that require high effort during training . it also shows that cross-lingual parsers can provide better quality information . |
Parser Training with Heterogeneous Treebanks (P18-2)
Copied to clipboard
| Challenge: | In the 2017 CoNLL Shared Task on Universal Dependency Parsing, 25 languages have more than one treebank . many teams did not take advantage of the multiple treebanks, however, and trained one model per treebank instead of one model for each language. |
| Approach: | They propose a method to make the most of heterogeneous treebanks when training a monolingual parser. |
| Outcome: | The proposed method improves on training with multiple treebanks for a single language. |
Please Mind the Root: Decoding Arborescences for Dependency Parsing (2020.emnlp-main)
Copied to clipboard
| Challenge: | a dependency tree has a root constraint, but only one edge may emanate from the root node. |
| Approach: | They propose an algorithm which enforces a root constraint without compromising the original runtime. |
| Outcome: | The proposed algorithm satisfies the constraint without compromising the original runtime. |
Treebank Embedding Vectors for Out-of-Domain Dependency Parsing (2020.acl-main)
Copied to clipboard
| Challenge: | a recent advance in monolingual dependency parsing is the idea of a treebank embedding vector . this allows the model to prefer training data from one treebank over another at test time . |
| Approach: | They propose a method to predict a treebank vector for sentences that do not come from a particular treebank . they also explore what happens when they move away from predefined treebank embedding vectors . |
| Outcome: | The proposed method can predict treebank vectors for sentences that do not come from a treebank used in training with sufficient accuracy for nine out of ten languages. |
On Leakage of Code Generation Evaluation Datasets (2024.findings-emnlp)
Copied to clipboard
Alexandre Matton, Tom Sherborne, Dennis Aumiller, Elena Tommasone, Milad Alizadeh, Jingyi He, Raymond Ma, Maxime Voisin, Ellen Gilsenan-McMahon, Matthias Gallé
| Challenge: | In this paper, we discuss contamination by code generation test sets in large language models. |
| Approach: | They propose to use Python to test code generation test sets for contamination . they find that code generation is an important skill for large language models to master . |
| Outcome: | The proposed benchmarks are uncontaminated and provide a new insight into code generation. |
A Deeper Look into Dependency-Based Word Embeddings (N18-4)
Copied to clipboard
| Challenge: | Word embeddings trained with dependency contexts excel at different tasks, and enhanced dependencies often improve performance. |
| Approach: | They propose to use dependency-based word embeddings to capture semantic similarity rather than relatedness. |
| Outcome: | The results show that word embeddings trained with Universal and Stanford dependencies excel at different tasks and that enhanced dependencies often improve performance. |