Papers by Hiroaki Funayama
Preventing Critical Scoring Errors in Short Answer Scoring with Confidence Estimation (2020.acl-srw)
Copied to clipboard
Hiroaki Funayama, Shota Sasaki, Yuichiroh Matsubayashi, Tomoya Mizumoto, Jun Suzuki, Masato Mita, Kentaro Inui
| Challenge: | Recent Short Answer Scoring systems use Quadratic Weighted Kappa (QWK) but it is unsatisfactory when measuring their effectiveness in actual usage. |
| Approach: | They propose a task formulation of Short Answer Scoring (SAS) that matches actual usage and extracts as many scoring predictions that are not critical scoring errors (CSEs). |
| Outcome: | The proposed system predicts scores with zero critical scoring errors (CSEs) for 50% of test data at maximum by filtering out low-reliability predictions on the basis of a certain confidence estimation. |
Japanese-English Sentence Translation Exercises Dataset for Automatic Grading (2024.eacl-srw)
Copied to clipboard
| Challenge: | Sentence translation exercises (STEs) are used in the early stages of L2 language learning. |
| Approach: | They propose to formalize the task of automatic assessment of Sentence Translation Exercises (STEs) they use a dataset of 21 questions and 3,498 student responses to test the framework . |
| Outcome: | The proposed task is based on a dataset of 21,498 student responses between Japanese and English and compares them to baseline models with fine-tuned BERT and GPT-3.5 with few-shot learning. |
Assessing Step-by-Step Reasoning against Lexical Negation: A Case Study on Syllogism (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) take advantage of step-by-step reasoning instructions . negation is a core linguistic phenomenon that is difficult to process . |
| Approach: | They examine the step-by-step reasoning ability of large language models with a focus on negation . negation is a core linguistic phenomenon that is difficult to process . |
| Outcome: | The proposed models perform better when using chain-of-thought prompting . the results highlight unique limitations in each LLM family . |