Papers with QWK
Unleashing Large Language Models’ Proficiency in Zero-shot Essay Scoring (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in automated essay scoring (AES) have relied on labeled essays, requiring tremendous cost and expertise for their acquisition. |
| Approach: | They propose a zero-shot prompting framework that automatically decomposes writing proficiency into distinct traits and generates scoring criteria for each trait. |
| Outcome: | The proposed framework outperforms straightforward prompting (Vanilla) on TOEFL11 and ASAP, while the small-sized Llama2-13b-chat significantly outperformed ChatGPT. |
Multi-task Learning for Automated Essay Scoring with Sentiment Analysis (2020.aacl-srw)
Copied to clipboard
| Challenge: | Automated Essay Scoring (AES) is a process that aims to alleviate the workload of graders and improve the feedback cycle in educational systems. |
| Approach: | They propose to combine two tasks, sentiment analysis and AES by utilizing multi-task learning to combine sentiment features extracted from opinion expressions. |
| Outcome: | The proposed model produces a QWK of 0.763 on the Automated StudentAssessment Prize (ASAP) benchmark. |
Preventing Critical Scoring Errors in Short Answer Scoring with Confidence Estimation (2020.acl-srw)
Copied to clipboard
Hiroaki Funayama, Shota Sasaki, Yuichiroh Matsubayashi, Tomoya Mizumoto, Jun Suzuki, Masato Mita, Kentaro Inui
| Challenge: | Recent Short Answer Scoring systems use Quadratic Weighted Kappa (QWK) but it is unsatisfactory when measuring their effectiveness in actual usage. |
| Approach: | They propose a task formulation of Short Answer Scoring (SAS) that matches actual usage and extracts as many scoring predictions that are not critical scoring errors (CSEs). |
| Outcome: | The proposed system predicts scores with zero critical scoring errors (CSEs) for 50% of test data at maximum by filtering out low-reliability predictions on the basis of a certain confidence estimation. |
Towards Prompt Generalization: Grammar-aware Cross-Prompt Automated Essay Scoring (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to score essays on unseen prompts are challenging to use in educational situations. |
| Approach: | They propose a grammar-aware cross-prompt trait scoring model which internally captures prompt-independent syntactic aspects to learn generic essay representation. |
| Outcome: | Empirical results show that the proposed model improves prompt-independent and grammar-related traits and achieves notable QWK gains in the most challenging cross-prompt scenario. |
“So You Think You’re Funny?”: Rating the Humour Quotient in Standup Comedy (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing datasets for humour classification are limited due to the subjectivity of the content and the multiple interpretations of the data. |
| Approach: | They propose to annotate a multi-modal humour-annotated dataset using stand-up comedy clips and compute a humor quotient using the audience's laughter. |
| Outcome: | The proposed scoring mechanism is validated by comparing with manual scoring methods and achieves an accuracy of 0.813 in terms of QWK. |
Autoregressive Multi-trait Essay Scoring via Reinforcement Learning with Scoring-aware Multiple Rewards (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing reinforcement learning (RL) applications in AES are limited to classification models despite associated performance degradation. |
| Approach: | They propose to integrate actual evaluation schemes into the training process by designing QWK-based rewards with a mean-squared error penalty for multi-trait AES. |
| Outcome: | The proposed scoring-aware multi-reward reinforcement learning integrates actual evaluation schemes into the training process. |
MAPLE: A Meta-learning Framework for Cross-Prompt Essay Scoring (2026.findings-acl)
Copied to clipboard
| Challenge: | Current approaches to automate essay scoring (AES) treat each writing task as a separate task, resulting in inconsistent performance. |
| Approach: | They propose a meta-learning framework that leverages prototypical networks to learn transferable representations across different writing prompts. |
| Outcome: | The proposed framework outperforms baseline models on ELLIPSE and ASAP (English) and LAILA (Arabic) on three diverse datasets. |