Papers with CLiMP
CLiMP: A Benchmark for Chinese Language Model Evaluation (2021.eacl-main)
Copied to clipboard
| Challenge: | Linguistically informed analyses of language models (LMs) contribute to understanding and improvement of such models. |
| Approach: | They introduce a corpus of Chinese linguistic minimal pairs (CLiMP) to investigate what knowledge Chinese LMs acquire. |
| Outcome: | The proposed corpus of Chinese linguistic minimal pairs (CLiMP) covers 9 major Chinese linguist phenomena. |
SLING: Sino Linguistic Evaluation of Large Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Using pre-trained language models, we find that the accuracy of LMs is far below human performance. |
| Approach: | They propose a benchmark of Sino LINGuistics which consists of 38K sentence pairs in Mandarin Chinese grouped into 9 high-level linguistic phenomena. |
| Outcome: | The proposed model performs better on local phenomena than hierarchical models and has a strong gender and number bias. |