Papers by Khai Doan
Fumbling in Babel: An Investigation into ChatGPT’s Language Identification Ability (2024.findings-naacl)
Copied to clipboard
| Challenge: | ChatGPT is a powerful NLP tool but its language identification abilities are unclear. |
| Approach: | They compile a benchmark comprising 670 languages representing 23 language families spoken in five continents and compare their language identification abilities to ChatGPT's (both GPT-3.5 and GPT-4) performance. |
| Outcome: | The proposed model performs poorly on African languages, while GPT-3.5 and GPT-4 perform poorly on English, Afrikaans, Arabic, Indonesian, Italian, Mandarin Chinese, and several more. |
The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing instruction tuned large language models (LLMs) struggle to understand cross-lingual sociopragmatic meaning (SM) lack of comprehensive investigation into their ability to understand SM is partly due to SM not being adequately represented in any of the existing benchmarks. |
| Approach: | They evaluate the performance of instruction tuned large language models (LLMs) on a multilingual benchmark specifically designed for SM understanding. |
| Outcome: | The proposed benchmark outperforms instruction tuned large language models on a wide range of tasks but falls behind task-specific finetuned models. |