| Challenge: | English part-of-speech taggers make egregious errors related to noun-verb ambiguity, despite having achieved 97%+ accuracy on the WSJ Penn Treebank since 2002. |
| Approach: | They propose to use a WSJ dataset to identify 30,000 examples of noun-verb ambiguity . they find that english part-of-speech taggers make egregious errors related to nouns and verbs . |
| Outcome: | The proposed model improves on the WSJ Penn Treebank by 14% and 52% relative to the previous model. |
Similar Papers
That was the last straw, we need more: Are Translation Systems Sensitive to Disambiguating Context? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models for translation of ambiguous text use context to disambiguate meaning . current models for MTs consistently translate English idioms literally, whereas LMs are context-aware . |
| Approach: | They use a dataset of 512 pairs of English sentences to study semantic ambiguities . they use literal and figurative idioms to disambiguate intended meaning . |
| Outcome: | The results show that current models translate English idioms literally, even when the context suggests a figurative interpretation. |
How Bad are PoS Tagger in Cross-Corpora Settings? Evaluating Annotation Divergence in the UD Project. (N19-1)
Copied to clipboard
| Challenge: | Using annotation variation principles, Part-of-Speech tagging performance degrades when applied to test sentences that depart from training data. |
| Approach: | They propose to use the annotation variation principle to identify inconsistencies between annotations . they also evaluate their impact on prediction performance . |
| Outcome: | The proposed method can detect errors in gold standard annotations and improve prediction performance. |
Reasoning about Ambiguous Definite Descriptions (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing resources to evaluate reasoning are not well suited to investigate the capability of resolving ambiguities by explicit reasoning. |
| Approach: | They propose to use ambiguous definite descriptions to create a benchmark dataset which requires models to resolve ambiguity by explicit reasoning. |
| Outcome: | The proposed model includes all information required to resolve the ambiguity in the prompt, which means a model does not require anything but reasoning to do well. |
Error Analysis of Uyghur Name Tagging: Language-specific Techniques and Remaining Challenges (L18-1)
Copied to clipboard
| Challenge: | despite efforts at name tagging, there is limited understanding on the performance ceiling . despite the high-resource language, there are very few natural language processing tools available . |
| Approach: | They propose to use a machine learning model to identify Uyghur name tagger errors . they conclude that such a model is unlikely to be effective for Uygur, or low-resource languages . |
| Outcome: | The proposed model is unlikely to be effective for Uyghur, or low-resource languages in general, the authors argue . they show that the proposed model can be used for high-res languages with superficial features . |
Distant Supervision from Disparate Sources for Low-Resource Part-of-Speech Tagging (D18-1)
Copied to clipboard
| Challenge: | Low-resource languages lack manual annotated data to learn basic models such as part-of-speech (POS) taggers. |
| Approach: | They propose a cross-lingual neural part-of-speech tagger that learns from disparate sources of distant supervision in a uniform framework. |
| Outcome: | The proposed model scales to hundreds of low-resource languages without access to gold annotated data. |
Constructions Are So Difficult That Even Large Language Models Get Them Right for the Wrong Reasons (2024.lrec-main)
Copied to clipboard
| Challenge: | In this paper, we examine the ability of large language models (LLMs) to identify different meanings in sentences that are superficially similar. |
| Approach: | They propose a challenge dataset for NLP with large lexical overlap which minimises the possibility of models discerning entailment solely based on token distinctions. |
| Outcome: | The proposed model fails to distinguish between constructions with three classes of adjectives which cannot be distinguished by surface features. |
Understanding Cross-Lingual Alignment—A Survey (2024.findings-acl)
Copied to clipboard
| Challenge: | Cross-lingual alignment is the meaningful similarity of representations across languages in multilingual language models. |
| Approach: | They propose a taxonomy of methods to improve cross-lingual alignment . they argue that an effective trade-off between language-neutral and language-specific information is key . |
| Outcome: | The proposed methods can be applied to encoder models and encoder-decoder-only models . they show that language-neutral and language-specific information is key . |
We’re Afraid Language Models Aren’t Modeling Ambiguity (2023.emnlp-main)
Copied to clipboard
Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah Smith, Yejin Choi
| Challenge: | Ambiguity is an intrinsic feature of natural language, allowing us to anticipate misunderstandings and revise our interpretations as listeners. |
| Approach: | They use AmbiEnt to capture ambiguity in a sentence and analyze it to evaluate pretrained LMs. |
| Outcome: | The proposed model can flag political claims in the wild that are misleading due to ambiguity. |
Incorporating Contextual Information for Language-Independent, Dynamic Disambiguation Tasks (L18-1)
Copied to clipboard
| Challenge: | a proposed multimodal system can resolve syntactic ambiguities by exploiting external evidence, says a researcher . a parser that processes linguistic information is expected to handle syntakically unambiguous sentences, but it cannot. |
| Approach: | They propose to exploit external contextual information to resolve ambiguous sentences . they propose to use data-driven and grammar-based approaches to solve ambiguities . |
| Outcome: | The proposed system confirms this hypothesis in experiments on syntactically ambiguous sentences. |
CHAmbi: A New Benchmark on Chinese Ambiguity Challenges for Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Ambiguity is an inherent feature of language, whose management is crucial for effective communication and collaboration. |
| Approach: | They propose a dataset to evaluate LLMs' ability to handle ambiguity in Chinese by using a specialized Chinese multi-label disambiguation dataset formatted in Natural Language Inference. |
| Outcome: | The CHAmbi dataset comprises 4,991 pairs of premises and hypotheses, including 824 examples featuring a wide range of ambiguities. |