| Challenge: | Lexical-semantic resources like WordNet are a fundamental resource for many NLP and semantic applications. |
| Approach: | They propose a crowdsourcing workflow that consists of synset localization and validation . they use inter-rater agreement metrics to estimate the precision of the results . |
| Outcome: | The proposed method is cost-effective and provides a good trade-off between quality and speed of progress. |
Similar Papers
Crowdsourcing in the Development of a Multilingual FrameNet: A Case Study of Korean FrameNet (2020.lrec-1)
Copied to clipboard
| Challenge: | Using current methods, the construction of multilingual FrameNets is expensive and complex. |
| Approach: | They evaluated whether crowdsourcing approaches captured cross-cultural and cross-linguistic meanings . they found that crowd workers made intuitive choices comparable to trained FrameNet experts . |
| Outcome: | The results are now available in Korean FrameNet 1.1. |
Crowdsourcing Natural Language Data at Scale: A Hands-On Tutorial (2021.naacl-tutorials)
Copied to clipboard
| Challenge: | a tutorial on crowdsourcing for efficient data annotation will introduce crowdsourcing and provide an overview of the technology. |
| Approach: | This tutorial will introduce users to efficient data annotation via crowdsourcing marketplaces. |
| Outcome: | This tutorial will introduce users to the use of crowdsourcing for data annotation. |
A Crowdsourced Frame Disambiguation Corpus with Ambiguity (N19-1)
Copied to clipboard
| Challenge: | Using crowdsourcing, we have found that inter-annotator disagreement is at least partly caused by ambiguity inherent to the text and frames. |
| Approach: | They propose a crowdsourcing approach to capture inter-annotator disagreement by a list of frames with disagreement-based scores that express the confidence with which each frame applies to the word. |
| Outcome: | The proposed approach captures disagreement between the annotations of 1,000 word-sentence pairs and scores on the likelihood that each frame applies to the word. |
Inferences for Lexical Semantic Resource Building with Less Supervision (2020.lrec-1)
Copied to clipboard
| Challenge: | lexical semantic resources may be built using various approaches such as extraction from corpora, integration of relevant pieces of knowledge from pre-existing knowledge resources and endogenous inference. |
| Approach: | They propose a method where the resource building process appears as a self learning process . they propose lexical and semantic resource building based on inference . |
| Outcome: | The proposed method reduces the human effort needed for lexical semantic resource building. |
Lexical Resource Mapping via Translations (2022.lrec-1)
Copied to clipboard
| Challenge: | a lexical resource associates words with concepts in multiple languages, which makes it difficult to combine information from multiple resources. |
| Approach: | They propose a translation-based approach to mapping lexical resources . they use word-concept pairs to align WordNet/BabelNet to CLICS and OmegaWiki . |
| Outcome: | The proposed method achieves state-of-the-art accuracy without other sources of knowledge . it can be framed as word sense disambiguation, and it can improve on existing methods . |
Creating Expert Knowledge by Relying on Language Learners: a Generic Approach for Mass-Producing Language Resources by Combining Implicit Crowdsourcing and Language Learning (2020.lrec-1)
Copied to clipboard
Lionel Nicolas, Verena Lyding, Claudia Borg, Corina Forascu, Karën Fort, Katerina Zdravkova, Iztok Kosem, Jaka Čibej, Špela Arhar Holdt, Alice Millour, Alexander König, Christos Rodosthenous, Federico Sangati, Umair ul Hassan, Anisia Katinskaia, Anabela Barreiro, Lavinia Aparaschivei, Yaakov HaCohen-Kerner
| Challenge: | Lack of wide-coverage and high-quality LRs is a longstanding issue in natural language processing (NLP) however, there are no large initiatives of similar scale for creating new LR or improving existing ones. |
| Approach: | They propose a generic approach to combine implicit crowdsourcing and language learning to mass-produce language resources (LRs) they describe its core paradigm that consists in pairing specific types of LRs with specific exercises . |
| Outcome: | The proposed approach can be used in several learning scenarios to produce a multitude of NLP resources and alleviate the long-standing issue of the lack of LRs. |
A Dataset of Crowdsourced Word Sequences: Collections and Answer Aggregation for Ground Truth Creation (D19-59)
Copied to clipboard
| Challenge: | Existing work on answer aggregation for labels is limited . existing work on label aggregations is limited to label . |
| Approach: | They propose three approaches to extractive word sequence aggregation from translated sentences generated by multiple workers. |
| Outcome: | The proposed dataset contains translated sentences generated from multiple workers. |
Learning from Measurements in Crowdsourcing Models: Inferring Ground Truth from Diverse Annotation Types (C18-1)
Copied to clipboard
| Challenge: | Annotated corpora are often assigned to internet workers whose judgments are reconciled by crowdsourcing models. |
| Approach: | They propose a framework for learning from rich prior knowledge to combine annotations with different structures. |
| Outcome: | The proposed model compares favorably with previous work and enables active sample selection to reduce annotation effort. |
Quality Control for Crowdsourced Bilingual Dictionary in Low-Resource Languages (2022.lrec-1)
Copied to clipboard
| Challenge: | a recent study shows that crowdsourcing is becoming mainstream to create bilingual dictionaries . the number of people who can speak multiple low-resource languages is limited and the average ability of workers is low. |
| Approach: | They propose a method to aggregate the answers of evaluation tasks by majority voting . they use hyper questions to evaluate the reliability of workers and task-allocation method to select high-quality workers . |
| Outcome: | The proposed method improves quality of bilingual dictionaries by integrating answers by majority voting. |
What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks? (2021.acl-long)
Copied to clipboard
| Challenge: | Despite the importance of datasets for natural language understanding, there has been little attention on crowdsourcing methods for collecting datasets. |
| Approach: | They compare the effectiveness of crowdsourcing methods for boosting NLU example difficulty with training crowdworkers instead of expert judgments. |
| Outcome: | The proposed method is ineffective for boosting NLU example difficulty, but it is not effective for training crowdworkers and qualifying workers based on expert judgments. |