| Challenge: | Linguistically informed analyses of language models (LMs) contribute to understanding and improvement of such models. |
| Approach: | They introduce a corpus of Chinese linguistic minimal pairs (CLiMP) to investigate what knowledge Chinese LMs acquire. |
| Outcome: | The proposed corpus of Chinese linguistic minimal pairs (CLiMP) covers 9 major Chinese linguist phenomena. |
Similar Papers
SLING: Sino Linguistic Evaluation of Large Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Using pre-trained language models, we find that the accuracy of LMs is far below human performance. |
| Approach: | They propose a benchmark of Sino LINGuistics which consists of 38K sentence pairs in Mandarin Chinese grouped into 9 high-level linguistic phenomena. |
| Outcome: | The proposed model performs better on local phenomena than hierarchical models and has a strong gender and number bias. |
A Systematic Assessment of Language Models with Linguistic Minimal Pairs in Chinese (2026.tacl-1)
Copied to clipboard
Yikang Liu, Yeting Shen, Hongao Zhu, Lilong Xu, Zhiheng Qian, Siyuan Song, Kejia Zhang, Jialong Tang, Pei Zhang, Baosong Yang, Rui Wang, Hai Hu
| Challenge: | Using sub-linear length normalized log-probabilities (SLLN-LP), we find unequal lengths of sentences in minimal pairs difficult for LMs even up to 32B parameters. |
| Approach: | They propose to use ZhoBLiMP as a linguistic minimal pair benchmark for Chinese language models to mitigate biases. |
| Outcome: | The proposed metric mitigates biases in Chinese language models with over 100 paradigms . Anaphor, Quantifiers, and Ellipsis are difficult for LMs even up to 32B parameters . |
JBLiMP: Japanese Benchmark of Linguistic Minimal Pairs (2023.findings-eacl)
Copied to clipboard
| Challenge: | In this paper, we compare syntactic knowledge of language models across different languages. |
| Approach: | They introduce a dataset for targeted syntactic evaluations of language models in Japanese. |
| Outcome: | The proposed dataset compares the syntactic knowledge of language models across languages. |
BLiMP: The Benchmark of Linguistic Minimal Pairs for English (2020.tacl-1)
Copied to clipboard
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, Samuel R. Bowman
| Challenge: | Recent studies have examined how linguistic knowledge of language models (LMs) varies across English phenomena. |
| Approach: | They propose a benchmark to evaluate linguistic knowledge of language models on major grammatical phenomena in English. |
| Outcome: | The proposed benchmark evaluates the linguistic knowledge of language models on major grammatical phenomena in English. |
CxMP: A Linguistic Minimal-Pair Benchmark for Evaluating Constructional Understanding in Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Understanding language acquisition in language models remains an open question, yet many benchmarks focus on grammatical acceptability, with far less attention to interpreting meanings conveyed by grammatological forms. |
| Approach: | They propose a benchmark to evaluate constructional understanding in language models using a controlled minimal-pair. |
| Outcome: | The proposed benchmarks show that understanding of constructions develops more slowly and remains limited even in large language models (LLMs). |
QFrBLiMP: a Quebec-French Benchmark of Linguistic Minimal Pairs (2026.findings-eacl)
Copied to clipboard
| Challenge: | Specifically, these minimal pairs are created by manually modifying sentences extracted from an official online resource maintained by a Québec government institution. |
| Approach: | They propose to use the Quebec-French Benchmark of Linguistic Minimal Pairs to evaluate LLMs’ linguistic knowledge of prominent grammatical phenomena in Quebec-french. |
| Outcome: | The proposed corpus evaluates LLMs’ linguistic knowledge of prominent grammatical phenomena in Quebec-French. |
MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs (2026.tacl-1)
Copied to clipboard
| Challenge: | MultiBLiMP 1.0 is a massively multilingual benchmark of linguistic minimal pairs covering 101 languages and 2 types of subject-verb agreement. |
| Approach: | They propose to use multilingual benchmarks to evaluate linguistic minimal pairs in 101 languages and 2 types of subject-verb agreement to create the minimal pairs. |
| Outcome: | The proposed benchmark covers 101 languages and 2 types of subject-verb agreement, and contains more than 128,000 minimal pairs. |
CMMLU: Measuring massive multitask language understanding in Chinese (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing large language models struggle to achieve an accuracy of even 60%, which is the pass mark for Chinese exams. |
| Approach: | They propose to use CMMLU to evaluate Chinese multilingual and Chinese LLMs in a comprehensive benchmark that covers various subjects and settings. |
| Outcome: | The proposed benchmark covers natural sciences, social sciences, engineering, and the humanities and aims to improve on existing models. |
Can Large Language Models Be Good Language Teachers? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have achieved remarkable success across diverse domains, but their potential as effective language teachers remains inadequately assessed. |
| Approach: | They propose a framework to evaluate Chinese language teachers' pedagogical competence against international standards. |
| Outcome: | The proposed framework evaluates 13 latest multilingual and Chinese LLMs against international standards for Chinese language teachers. |
Chinese SimpleQA: A Chinese Factuality Evaluation for Large Language Models (2025.acl-long)
Copied to clipboard
Yancheng He, Shilong Li, Jiaheng Liu, Yingshui Tan, Weixun Wang, Hui Huang, Xingyuan Bu, Hangyu Guo, Chengwei Hu, Boren Zheng, Zhuoran Lin, Dekai Sun, Zhicheng Zheng, Wenbo Su, Bo Zheng
| Challenge: | Current frontier models sometimes generate false outputs or answers that are not substantiated by evidence. |
| Approach: | They propose Chinese SimpleQA, a Chinese benchmark to evaluate LLMs' factuality . they focus on Chinese language over 6 major topics with 99 diverse subtopics . |
| Outcome: | The Chinese SimpleQA benchmark evaluates the factuality ability of LLMs . the questions and answers are short and easy-to-evaluate . |