Factual Consistency of Multilingual Pretrained Language Models (2022.findings-acl)
Copied to clipboard
| Challenge: | Recent work shows that monolingual English language models fill-in-the-blank differently for paraphrases describing the same fact. |
| Approach: | They propose a resource to analyze consistency of English language models . they find that mBERT is as inconsistent as English BERT in paraphrases . |
| Outcome: | The proposed model is as inconsistent as English BERT in English paraphrases, but it is more so for all the other 45 languages. |
Similar Papers
Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual large-scale pretrained language models store factual knowledge, but large variations are observed across languages. |
| Approach: | They propose a ranking-based consistency metric to evaluate cross-lingual consistency of factual knowledge in multilingual PLMs. |
| Outcome: | The proposed metric evaluates cross-lingual consistency of factual knowledge across languages independently from accuracy. |
Measuring and Improving Consistency in Pretrained Language Models (2021.tacl-1)
Copied to clipboard
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, Yoav Goldberg
| Challenge: | In this paper, we examine whether pretrained language models are consistent with factual knowledge. |
| Approach: | They propose a method to improve consistency of pretrained language models . consistency is a desirable property of a good language understanding model, they argue . |
| Outcome: | The proposed model improves consistency and shows that it is effective. |
Are Knowledge and Reference in Multilingual Language Models Cross-Lingually Consistent? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Cross-lingual consistency should be considered to assess cross-lingual transferability, maintain factuality of model knowledge across languages, and preserve parity of language model performance. |
| Approach: | They examine pretrained and tuned models with code-mixed coreferential statements that convey identical knowledge across languages. |
| Outcome: | The proposed model shows different levels of consistency in multilingual models, subject to language families, linguistic factors, scripts, and bottlenecks on a particular layer. |
Tracing Multilingual Factual Knowledge Acquisition in Pretraining (2025.findings-emnlp)
Copied to clipboard
Yihong Liu, Mingyang Wang, Amir Hossein Kargaran, Felicia Körner, Ercong Nie, Barbara Plank, François Yvon, Hinrich Schuetze
| Challenge: | Large Language Models are capable of recalling multilingual factual knowledge, but most studies evaluate only the final model, leaving the development of factual recall and crosslingual consistency unexplored. |
| Approach: | They trace how factual recall and crosslingual consistency evolve during pretraining, focusing on OLMo-7B as a case study. |
| Outcome: | The results show that fact frequency is the key to a better recall of multilingual facts, regardless of language, and some low-frequency facts in non-English languages can still be correctly recalled. |
Multilingual Summarization with Factual Consistency Evaluation (2023.findings-acl)
Copied to clipboard
| Challenge: | Abstractive summarization models generate factually inconsistent summaries, reducing their utility for real-world applications. |
| Approach: | They propose to use data filtering and controlled generation to detect hallucinations in machine generated summaries. |
| Outcome: | The proposed models detect factual inconsistencies in machine generated summaries, but they focus on English only. |
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Language models (LMs) capture factual knowledge by filling in the blanks of cloze-style prompts. |
| Approach: | They propose a code-switching-based method to improve the ability of multilingual LMs to access knowledge and verify its effectiveness on several benchmark languages. |
| Outcome: | The proposed method improves the ability of multilingual LMs to access knowledge and verify its effectiveness on several benchmark languages. |
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models (2021.eacl-main)
Copied to clipboard
| Challenge: | Recent work has shown that monolingual English language models can be used as knowledge bases. |
| Approach: | They use mBERT to query monolingual English language models with masked sentences to test their hypothesis. |
| Outcome: | The proposed model can be used to answer fill-in-the-blank questions in 53 languages. |
Lost in Multilinguality: Dissecting Cross-lingual Factual Inconsistency in Transformer Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Multilingual language models store factual knowledge across languages but struggle to provide consistent responses to semantically equivalent prompts in different languages. |
| Approach: | They propose a linear shortcut method that bypasses computations in the final layers . this method improves accuracy and cross-lingual consistency . |
| Outcome: | The proposed method improves prediction accuracy and cross-lingual consistency. |
Are Pretrained Multilingual Models Equally Fair across Languages? (2022.coling-1)
Copied to clipboard
| Challenge: | Pretrained multilingual language models can help bridge the digital language divide, enabling high-quality NLP models for lower-resourced languages. |
| Approach: | They propose to use a multilingual dataset to examine whether multilingual models are equally fair across languages. |
| Outcome: | The proposed model enables apples-to-apples comparison across languages of group disparities in multilingual language models. |
Comparing Explanation Faithfulness between Multilingual and Monolingual Fine-tuned Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Previous studies have investigated how different factors affect faithfulness of model explanations . |
| Approach: | They find that the larger the multilingual model, the less faithful FAs are compared to its counterpart monolingual models. |
| Outcome: | The results show that the larger the multilingual model, the less faithful the FAs are compared to its counterpart monolingual models. |