Generating and Evaluating Tests for K-12 Students with Language Model Simulations: A Case Study on Sentence Reading Efficiency (2023.emnlp-main)
Copied to clipboard
| Challenge: | Developing an educational test can be expensive and time-consuming, as each item must be written by experts and then evaluated by collecting hundreds of student responses. |
| Approach: | They propose to fine-tune large language models to simulate how previous students would have responded to unseen items to generate high-quality parallel tests. |
| Outcome: | The proposed test forms are designed to be content-equivalent and produce identical individual scores as the original test form. |
Similar Papers
The Program Testing Ability of Large Language Models for Code (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Recent development of large language models (LLMs) for code shows promise in achieving code intelligence. |
| Approach: | They explore the ability of large language models to generate automated test cases . they show +11.77% and +4.22% higher code pass rates on HumanEval+ . |
| Outcome: | The proposed models show higher pass rates on humanEval+ compared with the current state-of-the-art models. |
Evaluating Large Language Models on Controlled Generation Tasks (2023.emnlp-main)
Copied to clipboard
Jiao Sun, Yufei Tian, Wangchunshu Zhou, Nan Xu, Qian Hu, Rahul Gupta, John Wieting, Nanyun Peng, Xuezhe Ma
| Challenge: | Recent studies have looked into the ability of large language models in various benchmark tasks, including question generation, reading comprehension, multilingual and etc. However, few studies investigate the controllability of large languages. |
| Approach: | They propose to compare large language models with state-of-the-start finetuned smaller models to find that large language model controls are comparable to smaller models. |
| Outcome: | The proposed model can meet hard constraints and perform better than state-of-the-art models. |
Machine Learning–Driven Language Assessment (2020.tacl-1)
Copied to clipboard
| Challenge: | Language proficiency tests are cumbersome to create and maintain, and items may be copied and leaked or simply used too often. |
| Approach: | They propose a method that uses machine learning and natural language processing to induce proficiency scales and linguistic models to estimate item difficulty directly for computer-adaptive testing. |
| Outcome: | The proposed method produces scores that are reliable and reliable while generating item banks large enough to satisfy security requirements. |
Evaluating Large Language Models on Wikipedia-Style Survey Generation (2024.findings-acl)
Copied to clipboard
Fan Gao, Hang Jiang, Rui Yang, Qingcheng Zeng, Jinghui Lu, Moritz Blum, Tianwei She, Yuang Jiang, Irene Li
| Challenge: | Recent studies have shown that large language models can perform well in general tasks, but their effectiveness and limitations in domainspecific tasks remain unclear. |
| Approach: | They examine the proficiency of Large Language Models (LLMs) in generating succinct survey articles specific to the niche field of NLP in computer science. |
| Outcome: | The LLMs perform better in generating succinct survey articles specific to the niche field of NLP in computer science, compared to human-authored surveys, but they exhibit bias in evaluation. |
Evaluating Language Models as Synthetic Data Generators (2025.acl-long)
Copied to clipboard
Seungone Kim, Juyoung Suk, Xiang Yue, Vijay Viswanathan, Seongyun Lee, Yizhong Wang, Kiril Gashteovski, Carolin Lawrence, Sean Welleck, Graham Neubig
| Challenge: | Prior studies have focused on developing effective data generation methods, but lack systematic comparison of different LMs as data generators in a unified setting. |
| Approach: | They propose to use a benchmark to compare language models' data generation abilities against a set of standardized settings and metrics. |
| Outcome: | The proposed benchmark provides standardized settings and metrics to evaluate LMs’ data generation abilities. |
Take Out Your Calculators: Estimating the Real Difficulty of Question Items with LLM Student Simulations (2026.findings-acl)
Copied to clipboard
| Challenge: | Standardized math assessments require expensive human pilot studies to establish the difficulty of test items. |
| Approach: | They propose to use large language models to model difficulty of multiple-choice math questions for real-world students. |
| Outcome: | The proposed model predicts difficulty of multiple-choice math questions for students . correlations between model and real-world difficulty are high, the authors show . |
Evaluating Large Language Models via Linguistic Profiling (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) undergo extensive evaluation against various benchmarks collected in established leaderboards to assess their performance across multiple tasks. |
| Approach: | They propose a new evaluation methodology to test LLMs' sentence generation abilities under specific linguistic constraints. |
| Outcome: | The proposed evaluation methodology is based on the 'linguistic profiling' approach and is not intended to be a task-oriented evaluation. |
Using LLMs to simulate students’ responses to exam questions (2024.findings-emnlp)
Copied to clipboard
Luca Benedetto, Giovanni Aradelli, Antonia Donvito, Alberto Lucchetti, Andrea Cappelli, Paula Buttery
| Challenge: | Existing studies have used Large Language Models to simulate students answering exam questions . a proposed prompt for GPT-3.5 is not suitable for all LLMs, and there is no correlation between the quality of the rationales obtained with the model and the accuracy of the student simulation task. |
| Approach: | They propose a large language model prompt engineered for GPT-3.5 that can be used to answer exam questions simulating students of different skill levels. |
| Outcome: | The proposed prompt is robust to different educational domains and generalise to data unseen during prompt engineering phase. |
Can Large Language Models Automatically Score Proficiency of Written Essays? (2024.lrec-main)
Copied to clipboard
| Challenge: | Automated essay scoring (AES) is one of the earliest research problems in natural language processing. |
| Approach: | They propose to use large language models to analyze and score written essays using four different prompts. |
| Outcome: | The proposed models show comparable performance on four different prompts and a slight advantage over the state-of-the-art models. |
Uniform Complexity for Text Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models do not capture factors that contribute to producing consistent text. |
| Approach: | They propose a benchmark test to evaluate text complexity in generative models by observing linguistic properties of input prompts. |
| Outcome: | The proposed model fails to preserve complexity of input prompts even if finetuned with professionally written texts. |