| Challenge: | Using crowdsourcing, we have found that inter-annotator disagreement is at least partly caused by ambiguity inherent to the text and frames. |
| Approach: | They propose a crowdsourcing approach to capture inter-annotator disagreement by a list of frames with disagreement-based scores that express the confidence with which each frame applies to the word. |
| Outcome: | The proposed approach captures disagreement between the annotations of 1,000 word-sentence pairs and scores on the likelihood that each frame applies to the word. |
Similar Papers
Frame Semantics across Languages: Towards a Multilingual FrameNet (C18-3)
Copied to clipboard
| Challenge: | This workshop will present current research on aligning Frame Semantic resources across languages . resources based on FrameNet have been created for roughly a dozen languages based upon Fillmore's Frame Sementics . |
| Approach: | This workshop will present current research on aligning Frame Semantic resources across languages . resources based on FrameNet have been created for roughly a dozen languages based upon Fillmore's Frame Sementics . |
| Outcome: | This workshop will present current research on aligning Frame Semantic resources across languages and automatic frame semantic parsing in English and other languages. |
Crowdsourcing in the Development of a Multilingual FrameNet: A Case Study of Korean FrameNet (2020.lrec-1)
Copied to clipboard
| Challenge: | Using current methods, the construction of multilingual FrameNets is expensive and complex. |
| Approach: | They evaluated whether crowdsourcing approaches captured cross-cultural and cross-linguistic meanings . they found that crowd workers made intuitive choices comparable to trained FrameNet experts . |
| Outcome: | The results are now available in Korean FrameNet 1.1. |
Transfer of Frames from English FrameNet to Construct Chinese FrameNet: A Bilingual Corpus-Based Approach (L18-1)
Copied to clipboard
| Challenge: | Current publicly available Chinese FrameNet has a relatively low coverage of frames and lexical units compared with other languages. |
| Approach: | They propose an automatic way to construct Chinese FrameNet using a sentence-aligned English-Chinese bilingual corpus. |
| Outcome: | The proposed resource can provide frame recommendations acceptable by annotators. |
Semantic Frame Induction from a Real-World Corpus (2025.acl-srw)
Copied to clipboard
| Challenge: | Existing studies on semantic frame induction have demonstrated that pre-trained language models (PLMs) have led to more accurate results. |
| Approach: | They conduct semantic frame induction using the Colossal Clean Crawled Corpus and assess the applicability of existing frame inducing methods to real-world data. |
| Outcome: | The proposed methods outperform existing methods on real-world data and can induce frames corresponding to novel concepts. |
Cross-lingual Linking of Automatically Constructed Frames and FrameNet (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing semantic frame resources have been manually elaborated, but manual development is labor-intensive. |
| Approach: | They propose to link Japanese frames to English FrameNet by using cross-lingual word embeddings and a model that takes only the frame-evoking words into account. |
| Outcome: | The proposed model will facilitate the development of cross-lingual frame resources. |
Would you describe a leopard as yellow? Evaluating crowd-annotations with justified and informative disagreement (2020.coling-main)
Copied to clipboard
| Challenge: | Existing evaluation methods rely on agreement between annotators, which implies a single correct interpretation. |
| Approach: | They propose an agreement-independent quality metric based on answer-coherence to evaluate on expected disagreement. |
| Outcome: | The proposed model shows that agreement is the most important indicator of quality in semantic annotation tasks. |
Enriching Frame Representations with Distributionally Induced Senses (L18-1)
Copied to clipboard
| Challenge: | lexical resource that enriches Framester knowledge graph with semantic features from text corpora . paves way for development of novel, deeper semantic-aware applications . |
| Approach: | They propose a lexical resource that enriches the Framester knowledge graph with semantic features from text corpora. |
| Outcome: | The proposed resource enables the development of deeper semantic-aware applications . it combines knowledge from text and symbolic representations of events and participants . |
Do LLMs Encode Frame Semantics? Evidence from Frame Identification (2025.emnlp-main)
Copied to clipboard
| Challenge: | Using the FrameNet lexical resource, we evaluate large language models under prompt-based inference and observe that they can perform frame identification effectively even without explicit supervision. |
| Approach: | They evaluate large language models under prompt-based inference and observe that they encode latent knowledge of frame semantics. |
| Outcome: | The proposed model can generate coherent frame definitions while generalizing well to out-of-domain benchmarks. |
We’re Afraid Language Models Aren’t Modeling Ambiguity (2023.emnlp-main)
Copied to clipboard
Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah Smith, Yejin Choi
| Challenge: | Ambiguity is an intrinsic feature of natural language, allowing us to anticipate misunderstandings and revise our interpretations as listeners. |
| Approach: | They use AmbiEnt to capture ambiguity in a sentence and analyze it to evaluate pretrained LMs. |
| Outcome: | The proposed model can flag political claims in the wild that are misleading due to ambiguity. |
Subjective Crowd Disagreements for Subjective Data: Uncovering Meaningful CrowdOpinion with Population-level Learning (2023.acl-long)
Copied to clipboard
| Challenge: | Annotator disagreements are resolved before learning takes place, but researchers question the performance of a system when annotators disagree. |
| Approach: | They propose a method that uses language features and label distributions to pool similar items into larger labels. |
| Outcome: | The proposed method is based on five publicly available datasets with varying levels of disagreements on social media and in the wild using a dataset from Facebook. |