ViGLUE: A Vietnamese General Language Understanding Benchmark and Analysis of Vietnamese Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing benchmarks for natural language understanding have been suggested, but there is a lack of such a benchmark in Vietnamese due to the difficulty in accessing datasets or the scarcity of task-specific datasets. |
| Approach: | They propose to use a benchmark to evaluate Vietnamese language models in a variety of tasks and areas to explore the relationship between specific tasks and the number of shots. |
| Outcome: | The proposed benchmark contains twelve tasks and encompasses over ten areas and subjects, enabling it to evaluate models comprehensively over a broad spectrum of aspects. |
Similar Papers
VLUE: A New Benchmark and Multi-task Knowledge Transfer Learning for Vietnamese Natural Language Understanding (2024.findings-naacl)
Copied to clipboard
| Challenge: | a lack of standard evaluation metrics and benchmarks makes it difficult to identify strengths of Vietnamese NLP models. |
| Approach: | They propose to establish a standardized set of benchmarks for Vietnamese NLU . they propose to evaluate Vietnamese language understanding models using a pre-trained model . |
| Outcome: | The proposed model combines proficiency of a multilingual pre-trained model with Vietnamese linguistic knowledge. |
VMLU Benchmarks: A comprehensive benchmark toolkit for Vietnamese LLMs (2025.acl-long)
Copied to clipboard
Cuc Thi Bui, Nguyen Truong Son, Truong Van Trang, Lam Viet Phung, Pham Nhut Huy, Hoang Anh Le, Quoc Huu Van, Phong Nguyen-Thuan Do, Van Le Tran Truc, Duc Thanh Chau, Le-Minh Nguyen
| Challenge: | The evolution of Large Language Models (LLMs) has underscored the need for benchmarks designed for various languages and cultural contexts. |
| Approach: | They propose to use Vietnamese multitask language understanding (VMLU) benchmarks to assess different capabilities of LLMs, including general knowledge, reading comprehension, reasoning, and conversational skills. |
| Outcome: | The VMLU Benchmarks assess LLMs' general knowledge, reading comprehension, reasoning, and conversational skills. |
ViGPTQA - State-of-the-Art LLMs for Vietnamese Question Answering: System Overview, Core Models Training, and Evaluations (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Large language models (LLMs) and their applications in low-resource languages are limited due to lack of training data and benchmarking datasets. |
| Approach: | They propose a question-response system for Vietnamese that uses LLMs . they propose to open-source the model and train it on benchmark datasets based on Vietnamese data . |
| Outcome: | The proposed question answering system for Vietnamese is open-source and performant . it can learn and capture human-like text, but there is a gap in evaluations for Vietnamese . |
Revealing Weaknesses of Vietnamese Language Models Through Unanswerable Questions in Machine Reading Comprehension (2023.eacl-srw)
Copied to clipboard
| Challenge: | Existing problems in Vietnamese Machine Reading Comprehension systems are limited due to multilinguality, which limits the ability of multilingual models to develop state-of-the-art systems. |
| Approach: | They propose to modify the process of annotating unanswerable questions to improve the quality of unanswered questions to a higher level of difficulty for Machine Reading Comprehension systems to solve. |
| Outcome: | The proposed modification improves the quality of unanswerable questions to a higher level of difficulty for Machine Reading Comprehension systems to solve. |
A Large-Scale Benchmark for Vietnamese Sentence Paraphrases (2025.findings-naacl)
Copied to clipboard
| Challenge: | 1.2M original–paraphrase pairs were generated using a hybrid approach to generate high-quality paraphrases. |
| Approach: | They present a high-quality Vietnamese dataset for sentence paraphrasing . they used automatic paraphrase generation and manual evaluation to ensure high quality . |
| Outcome: | The proposed dataset is the first large-scale study on Vietnamese paraphrasing . it combines automatic paraphrase generation with manual evaluation to ensure high quality . |
ViDeBERTa: A powerful pre-trained language model for Vietnamese (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing models for Vietnamese that perform well on downstream tasks, such as Question answering, are based on Transformer. |
| Approach: | They propose a pre-trained monolingual Vietnamese model with three versions . they fine-tune and evaluate the model on three important natural language downstream tasks, Part-of-speech tagging, Named-entity recognition, and Question answering. |
| Outcome: | The proposed model outperforms the existing model on three important natural language downstream tasks, Part-of-speech tagging, Named-entity recognition, and Question answering. |
PhoBERT: Pre-trained language models for Vietnamese (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Experimental results show that PhoBERT outperforms the recent best pre-trained multilingual model XLM-R in multiple Vietnamese-specific NLP tasks. |
| Approach: | They present PhoBERT with two versions, Phobert-base and PhoBRET-large, which are pre-trained for Vietnamese. |
| Outcome: | The proposed model outperforms the best pre-trained model XLM-R and improves the state-of-the-art in multiple Vietnamese-specific NLP tasks including Part-of speech tagging, Dependency parsing, Named-entity recognition and Natural language inference. |
A Vietnamese Dataset for Evaluating Machine Reading Comprehension (2020.coling-main)
Copied to clipboard
| Challenge: | despite the lack of benchmark datasets for Vietnamese, there are few studies on machine reading comprehension (MRC) . MRC is an essential core for a range of natural language processing applications such as search engines and intelligent agents. |
| Approach: | They propose to use Vietnamese Question Answering Dataset to evaluate machine reading comprehension in Vietnamese . they use over 23,000 human-generated question-answer pairs based on 5,109 Vietnamese articles . |
| Outcome: | The proposed dataset includes over 23,000 human-generated question-answer pairs based on 5,109 passages of 174 Vietnamese articles from Wikipedia. |
Multi-Dialect Vietnamese: Task, Dataset, Baseline Models and Challenges (2024.emnlp-main)
Copied to clipboard
| Challenge: | Vietnamese is a low-resource language, but each province has its own distinct pronunciation variations. |
| Approach: | They propose a dataset that captures the rich diversity of 63 provincial dialects spoken in Vietnam. |
| Outcome: | The proposed dataset captures the rich diversity of 63 provincial dialects spoken across Vietnam. |
BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing multilingual benchmarks focus primarily on language understanding tasks. |
| Approach: | They develop a multi-way multilingual benchmark that measures critical capabilities of large language models across languages. |
| Outcome: | Extensive experiments on BenchMAX reveal uneven utilization of core capabilities across languages, emphasizing the performance gaps that scaling model size alone does not resolve. |