| Challenge: | Existing methods for Appraisal annotation are descriptive and lack of data hinders progress . |
| Approach: | They propose to use annotated data to measure the performance of automated Appraisal annotations in a publicly available dataset. |
| Outcome: | The proposed methods show poor agreement at more detailed categories and fair agreement at coarse-level categories. |
Similar Papers
Task and Sentiment Adaptation for Appraisal Tagging (2023.eacl-main)
Copied to clipboard
| Challenge: | Appraisal framework in linguistics defines the framework for fine-grained evaluations and opinions. |
| Approach: | They propose to use language models to automatically identify and annotate text segments for appraisal. |
| Outcome: | The proposed model achieves superior performance than baseline adapter-based models and other neural classification models for cross-domain and cross-language settings. |
Automatic Argument Quality Assessment - New Datasets and Methods (D19-1)
Copied to clipboard
Assaf Toledo, Shai Gretz, Edo Cohen-Karlik, Roni Friedman, Elad Venezian, Dan Lahav, Michal Jacovi, Ranit Aharonov, Noam Slonim
| Challenge: | 6.3k arguments were collected from contributors of various levels, and are released as part of this work. |
| Approach: | They propose to use a language model to annotate arguments for argument ranking and argument-pair classification. |
| Outcome: | The proposed methods outperform state-of-the-art methods in the argument ranking task and argument-pair classification task. |
Automatic Article Commenting: the Task and Dataset (P18-2)
Copied to clipboard
| Challenge: | Existing methods to make comments on articles are based on human-annotated subsets, but they are not suitable for online forums. |
| Approach: | They propose to use a large-scale Chinese corpus with millions of real comments and a human-annotated subset characterizing the comments’ varying quality to generalize a broad set of popular reference-based metrics. |
| Outcome: | The proposed model incorporates human-annotated subset characterizing the comments’ varying quality and shows that it is more accurate than previous models. |
APPReddit: a Corpus of Reddit Posts Annotated for Appraisal (2022.lrec-1)
Copied to clipboard
Marco Antonio Stranisci, Simona Frenda, Eleonora Ceccaldi, Valerio Basile, Rossana Damiano, Viviana Patti
| Challenge: | Existing resources for emotion recognition are lacking for appraisal models. |
| Approach: | They propose to use APPReddit to annotate non-experimental data according to Appraisal theories . they compare it with enISEAR, a corpus of events created in an experimental setting and annotated according to this theory. |
| Outcome: | The proposed model predicts four appraisal dimensions without significant loss . the proposed model is compared with enISEAR, a corpus of events created in an experimental setting and annotated for appraisal. |
A Tutorial on Evaluation Metrics used in Natural Language Generation (2021.naacl-tutorials)
Copied to clipboard
| Challenge: | This tutorial presents the evolution of automatic evaluation metrics to their current state along with emerging trends in this field. |
| Approach: | This tutorial presents the evolution of automatic evaluation metrics to their current state . it aims to assess the extent of scientific progress made and identify areas/components that need improvement . |
| Outcome: | This tutorial presents the evolution of automatic evaluation metrics to their current state along with emerging trends in this field. |
Evaluating Subjective Cognitive Appraisals of Emotions from Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work on automatic prediction of cognitive appraisals has focused on physiological aspects of emotions. |
| Approach: | They present a dataset that assesses 24 appraisal dimensions across 241 Reddit posts . they find that open-source models fail to automatically assess and explain cognitive appraisals . |
| Outcome: | The proposed dataset assesses 24 appraisal dimensions across 241 reddit posts. |
A Systematic Review of Reproducibility Research in Natural Language Processing (2021.eacl-main)
Copied to clipboard
| Challenge: | Despite the recent progress in reproducibility, the field is far from reaching a consensus on how reproducibility should be defined, measured and addressed. |
| Approach: | They propose to provide a wide-angle snapshot of current work on reproducibility in NLP. |
| Outcome: | The proposed work will provide a wide-angle snapshot of current work on reproducibility in NLP. |
Analysis of Automatic Annotation Suggestions for Hard Discourse-Level Tasks in Expert Domains (P19-1)
Copied to clipboard
Claudia Schulz, Christian M. Meyer, Jan Kiesewetter, Michael Sailer, Elisabeth Bauer, Martin R. Fischer, Frank Fischer, Iryna Gurevych
| Challenge: | Existing deep learning methods require large amounts of training data to achieve reasonable performance. |
| Approach: | They propose to generate automatic annotation suggestions for a discourse-level sequence labelling task that requires extensive domain expertise. |
| Outcome: | The proposed model improves with newly annotated texts while introducing no biases. |
Appraisal Theories for Emotion Classification in Text (2020.coling-main)
Copied to clipboard
| Challenge: | Automatic emotion categorization is based on textual units assigned to an emotion from a predefined inventory, for instance following the basic emotion classes proposed by Paul Ekman (1999) or Plutchik (2001). |
| Approach: | They propose to make automatic emotion categorization explicit by following theories of cognitive appraisal of events and show their potential for emotion classification when being encoded in classification models. |
| Outcome: | The proposed models improve the classification of discrete emotion categories by using appraisal dimension assignments in event descriptions. |
AttributionBench: How Hard is Automatic Attribution Evaluation? (2024.findings-acl)
Copied to clipboard
| Challenge: | generative search engines enhance the reliability of large language model responses by providing cited evidence. |
| Approach: | They propose to use a benchmark to evaluate whether a large language model supports the generated responses or not . |
| Outcome: | The proposed benchmark shows that even a fine-tuned GPT-3.5 only achieves around 80% macro-F1 under a binary classification formulation. |