Papers by Thao Pham
The Base-Rate Effect on LLM Benchmark Performance: Disambiguating Test-Taking Strategies from Benchmark Performance (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Using the MMLU dataset, we show that the base-rate probability (BRP) differences across answer tokens are significant and affect task performance. |
| Approach: | They propose a cloze-based task that measures the BRP effect on model accuracy by counterfactual prompting and propose clozing to disambiguate test-taking ability from task performance. |
| Outcome: | The proposed version of the MMLU task disambiguates test-taking ability from task performance and reports the latter. |