Non-Programmers Can Label Programs Indirectly via Active Examples: A Case Study with Text-to-SQL (2023.emnlp-main)
Copied to clipboard
| Challenge: | Using APEL, non-programmers can annotate natural language utterances with complex programs that represent their meaning. |
| Approach: | They introduce a framework in which non-programmers select among candidate programs generated by a seed semantic parser. |
| Outcome: | The proposed framework achieves the same annotation accuracy as the original expert annotators (75%) and exposes many subtle errors in the original annotations. |
Similar Papers
Natural Language Annotations for Reasoning about Program Semantics (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Xu et al., 2022) and Tafjord eet . al. 2021) have shown that programming assistants can explain their work by grounding natural language inference in code. |
| Approach: | They propose a dataset and protocol for annotating programs with natural language predicates at a finer granularity than code comments without relying on internal compiler representations. |
| Outcome: | The proposed method can be used to ground natural language inference in code without static analysis and without internal compiler representations. |
ALToolbox: A Set of Tools for Active Learning Annotation of Natural Language Texts (2022.emnlp-demos)
Copied to clipboard
Akim Tsvigun, Leonid Sanochkin, Daniil Larionov, Gleb Kuzmin, Artem Vazhentsev, Ivan Lazichny, Nikita Khromov, Danil Kireev, Aleksandr Rubashevskii, Olga Shahmatova, Dmitry V. Dylov, Igor Galitskiy, Artem Shelmanov
| Challenge: | Currently, the framework supports text classification, sequence tagging, and seq2seq tasks. |
| Approach: | They propose an open-source framework for active learning annotation in natural language processing that provides an easy-to-deploy GUI annotation tool directly in the Jupyter IDE. |
| Outcome: | The proposed framework reduces computational overhead and duration of AL iterations and increases annotated data reusability. |
Rethinking Annotation: Can Language Learners Contribute? (2023.acl-long)
Copied to clipboard
| Challenge: | Researchers have traditionally recruited native speakers to provide annotations for benchmark datasets, but there are languages for which recruiting native speakers is difficult. |
| Approach: | They recruit 36 language learners and provide two types of additional resources and perform mini-tests to measure their language proficiency. |
| Outcome: | The proposed method improves learners' language proficiency in terms of vocabulary and grammar. |
AMR Parsing is Far from Solved: GrAPES, the Granular AMR Parsing Evaluation Suite (2023.emnlp-main)
Copied to clipboard
| Challenge: | Abstract Meaning Representation parsers have improved in recent years, but not solved. |
| Approach: | They propose an evaluation suite that evaluates AMR parsers on a range of phenomena . they find that current parser outputs are far from satisfactory . |
| Outcome: | The proposed evaluation suite reveals the abilities and shortcomings of current parsers. |
Platforms for Non-speakers Annotating Names in Any Language (P18-4)
Copied to clipboard
| Challenge: | Traditionally, native speakers of a language have been asked to annotate a corpus in that language. |
| Approach: | They propose two annotation platforms that allow an English speaker to annotate names for any language without knowing the language. |
| Outcome: | The proposed annotations achieved state-of-the-art performance on two surprise languages and ten languages at TAC-KBP EDL2017. |
Language-to-Code Translation with a Single Labeled Example (2024.emnlp-main)
Copied to clipboard
Kaj Bostrom, Harsh Jhamtani, Hao Fang, Sam Thomson, Richard Shin, Patrick Xia, Benjamin Van Durme, Jason Eisner, Jacob Andreas
| Challenge: | In-Context Inverse Programming (ICIP) bootstraps a language-to-code system using mostly unlabeled programs written using a potentially unfamiliar library or API. |
| Approach: | They propose a method for bootstrapping a language-to-code system using mostly unlabeled programs written using a potentially unfamiliar library or API. |
| Outcome: | The proposed method outperforms baselines across nine domains and 100 examples in a “nearly unsupervised” setting. |
Beyond Labels: Empowering Human Annotators with Natural Language Explanations through a Novel Active-Learning Architecture (2023.findings-emnlp)
Copied to clipboard
Bingsheng Yao, Ishan Jindal, Lucian Popa, Yannis Katsis, Sayan Ghosh, Lihong He, Yuxuan Lu, Shashank Srivastava, Yunyao Li, James Hendler, Dakuo Wang
| Challenge: | Existing low-resource learning techniques focus on label annotation while neglecting the natural language explanation of a data point. |
| Approach: | They propose a novel architecture that leverages an explanation-generation model to produce explanations guided by human explanations and a prediction model that utilizes generated explanations toward prediction faithfully. |
| Outcome: | The proposed architecture produces explanations guided by human explanations, a prediction model that utilizes generated explanations toward prediction faithfully, and a data diversity-based AL sampling strategy that benefits from the explanation annotations. |
Proceedings of the First Workshop on Aggregating and Analysing Crowdsourced Annotations for NLP (D19-59)
Copied to clipboard
| Challenge: | The first workshop on crowdsourcing for NLP is open to all . |
| Approach: | The first workshop on crowdsourcing annotations for NLP is held at the acl.com . the workshop will focus on methods for aggregating and analysing crowdsourced data for Nl-specific tasks. |
| Outcome: | The first workshop on crowdsourcing for NLP received 16 submissions and accepted 7 . the workshop will focus on ambiguous, subjective or ambiguity analysis of crowdsourced data . |
Commonsense Reasoning for Natural Language Processing (2020.acl-tutorials)
Copied to clipboard
| Challenge: | In this tutorial, we will outline the various types of commonsense knowledge and discuss techniques to gather and represent commonsence knowledge. |
| Approach: | This tutorial will provide researchers with the critical foundations and recent advances in commonsense representation and reasoning. |
| Outcome: | This tutorial will outline the various types of commonsense and discuss techniques to gather and represent commonsence knowledge while highlighting the challenges specific to this type of knowledge (e.g., reporting bias). |
A Structured Syntax-Semantics Interface for English-AMR Alignment (N18-1)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (AMR) annotations do not require explicit mapping between elements of an AMR and the corresponding elements of the sentence that evoke them. |
| Approach: | They devised an expressive framework to align AMR graphs to dependency graphs . their framework explains how 97% of AMR edges are evoked by words or syntax . |
| Outcome: | The proposed framework explains how 97% of AMR edges are evoked by words or syntax. |