Papers with Phun-Bench
Phun-Bench: Evaluating LLMs on Phonological Understanding in Chinese (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks on LLMs’ phonological abilities are either solvable through rote memorization or intertwined with other abilities, making them inadequate to measure LLM’s genuine ability in *phonological understanding*. |
| Approach: | They propose to use a Chinese benchmark to evaluate LLMs' phonological understanding to test their ability to recall correct pronunciations. |
| Outcome: | The proposed benchmarks show that LLMs excel at recalling correct pronunciations, but struggle to leverage phonological knowledge in the flexible and intuitive way that human speakers do. |