Enhancing Marker Scoring Accuracy through Ordinal Confidence Modelling in Educational Assessments (2025.acl-industry)
Copied to clipboard
| Challenge: | Automated Essay Scoring (AES) systems aim to evaluate the quality of candidate writing using computational methods. |
| Approach: | They propose a model that assigns a confidence score to each automated score to ensure it meets high reliability standards. |
| Outcome: | The proposed model achieves an F1 score of 0.97 and releases 47% of predicted scores with 100% CEFR agreement and 99% with at least 95% CEFR agreeance compared to the standalone model where all predicted scores are released. |
Similar Papers
Beyond the Score: Uncertainty-Calibrated LLMs for Automated Essay Assessment (2025.emnlp-main)
Copied to clipboard
| Challenge: | Automated Essay Scoring (AES) systems attain near–human agreement on some public benchmarks, but real-world adoption is limited. |
| Approach: | They propose a distribution-free wrapper that equips any classifier with set-valued outputs enjoying formal coverage guarantees. |
| Outcome: | The proposed model achieves coverage targets while keeping prediction sets compact. |
Beyond the Gold Standard in Analytic Automated Essay Scoring (2025.acl-srw)
Copied to clipboard
| Challenge: | Automated Essay Scoring (AES) is a new approach to assessing writing practice . traditional holistic scoring methods are not reliable and lack formative feedback in the classroom. |
| Approach: | They propose to combine analytic and holistic AES to create a system that learns from individual raters instead of gold standard labels. |
| Outcome: | The proposed system learns from individual raters instead of gold standard labels. |
Automated Essay Scoring: A Reflection on the State of the Art (2024.emnlp-main)
Copied to clipboard
| Challenge: | Automated essay scoring (AES) is a key application of natural language processing . it is based on a holistic score that summarizes the essay's overall quality . |
| Approach: | aaron carroll: automated essay scoring is one of the most important applications in NLP . carroll says the task is still far from being solved, but it's still progressing steadily . he says it'll be interesting to see how researchers can improve performance numbers . |
| Outcome: | a new neural model can beat existing models on a standard evaluation dataset, authors say . authors: the current model is not enough to improve performance numbers . they say it could spark discussion among researchers on how to move forward . |
Automatic Essay Scoring Incorporating Rating Schema via Reinforcement Learning (D18-1)
Copied to clipboard
| Challenge: | Existing systems for automatic essay scoring are trained to predict the score of each essay at a time without considering rating schema. |
| Approach: | They propose a reinforcement learning framework that incorporates quadratic weighted kappa as guidance to optimize the scoring system. |
| Outcome: | Experiments on benchmark datasets show the proposed framework is effective. |
Enhancing Automated Essay Scoring Performance via Fine-tuning Pre-trained Language Models with Combination of Regression and Ranking (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work on sentence prediction tasks uses shallow neural networks to learn essay representations and constrain calculated scores with regression loss or ranking loss. |
| Approach: | They propose to use a pre-trained language model to learn text representations first and then to constrain the scores with regression loss or ranking loss. |
| Outcome: | The proposed model outperforms state-of-the-art models on the Automated Student Assessment Prize dataset. |
PsyScore: A Psychometrically-Aware Framework for Trait-Adaptive Essay Scoring and ZPD-Scaffolded Feedback (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to Automated Essay Scoring (AES) treat scoring and feedback as separate components, resulting in fragmentation. |
| Approach: | They propose a psychometrically-aware framework that integrates diagnostic assessment with instructional scaffolding through a shared latent ability representation. |
| Outcome: | The proposed framework integrates diagnostic assessment with instructional scaffolding through a shared latent ability representation. |
Autoregressive Score Generation for Multi-trait Essay Scoring (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing holistic approaches to score essays using pre-trained BERT-based models are inefficient, leading to inferior qualities in data-scarce traits. |
| Approach: | They propose an autoregressive prediction of multi-trait scores using pre-trained T5 models. |
| Outcome: | The proposed model shows over 5% improvement in prompts and traits compared to previous models . |
Conundrums in Cross-Prompt Automated Essay Scoring: Making Sense of the State of the Art (2024.acl-long)
Copied to clipboard
| Challenge: | Automated essay scoring (AES) is a task of assigning a single score to an essay . authors abandon sophisticated neural architectures and develop a simple feature-based approach . |
| Approach: | a team of researchers develop a feature-based approach to cross-prompt automated essay scoring that adopts a simple neural architecture. |
| Outcome: | a new approach to cross-prompt automated essay scoring can achieve state-of-the-art results. |
Multi-task Learning for Automated Essay Scoring with Sentiment Analysis (2020.aacl-srw)
Copied to clipboard
| Challenge: | Automated Essay Scoring (AES) is a process that aims to alleviate the workload of graders and improve the feedback cycle in educational systems. |
| Approach: | They propose to combine two tasks, sentiment analysis and AES by utilizing multi-task learning to combine sentiment features extracted from opinion expressions. |
| Outcome: | The proposed model produces a QWK of 0.763 on the Automated StudentAssessment Prize (ASAP) benchmark. |
Prompt- and Trait Relation-aware Cross-prompt Essay Trait Scoring (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing systems assume to grade essays on same prompt as used in training and assign only a holistic score. |
| Approach: | They propose a prompt- and trait relation-aware cross-prompt essay trait scorer that encodes prompt-awful essay representation by essay-promotion attention and utilizing the topic-coherence feature extracted by the topic model. |
| Outcome: | The proposed model shows state-of-the-art results for all prompts and traits. |