Papers with XLM-RoBERTa
Leverage Points in Modality Shifts: Comparing Language-only and Multimodal Word Representations (2023.starsem-1)
Copied to clipboard
| Challenge: | a recent study of the effect of visual grounding on language representations has given a new life to the debate around extractability and quality of semantic information in representations trained solely on textual input. |
| Approach: | They compare word embeddings from vision-and-language models to text-only models . they identify meaning properties and relations that characterize words whose embeddements are most affected by visual grounding . |
| Outcome: | The proposed model differs from text-only models on semantic representations of language . the study is the first large-scale study of the effect of visual grounding on language representations . |
Negation typology and general representation models for cross-lingual zero-shot negation scope resolution in Russian, French, and Spanish. (2021.naacl-srw)
Copied to clipboard
| Challenge: | Negation resolution remains an acute and continuously researched question in Natural Language Processing. |
| Approach: | They propose to use multilingual pre-trained general representation models to detect negation scope in languages without annotated data. |
| Outcome: | The proposed model achieves token-level F1 score between English, Spanish, French, and Russian. |
UnMASKed: Quantifying Gender Biases in Masked Language Models through Linguistically Informed Job Market Prompts (2024.eacl-srw)
Copied to clipboard
| Challenge: | Language models (LMs) often include societal biases encoded in the human-produced datasets used for their training. |
| Approach: | They evaluated six prominent language models: BERT, RoBERTa, DistilBERT, BERT- multilingual, XLM-RoBERT and DistilberT- multilinguistic. |
| Outcome: | The results show that the models generated by the models were stereotypically gendered and with a reduced bias in multilingual variants. |
N-Best ASR Transformer: Enhancing SLU Performance using Multiple ASR Hypotheses (2021.acl-short)
Copied to clipboard
| Challenge: | Spoken Language Understanding systems parse spoken utterances into semantic structures like dialog acts and slots. |
| Approach: | They propose to use concatenated N-best ASR alternatives to represent utterances . they propose to employ a simpler utteration representation with no special delimiter . |
| Outcome: | The proposed model outperforms the prior state-of-the-art model on DSTC2 dataset. |
NepBERTa: Nepali Language Model Trained in a Large Corpus (2022.aacl-short)
Copied to clipboard
| Challenge: | Nepali is a low-resource language with more than 40 million speakers worldwide. |
| Approach: | They present a BERT-based natural language understanding model trained on the most extensive monolingual Nepali corpus ever. |
| Outcome: | The proposed model performs well in Nepali-specific NLP tasks including Named-Entity Recognition, Content Classification, POS Tagging, and Sequence Pair Similarity. |
FEED PETs: Further Experimentation and Expansion on the Disambiguation of Potentially Euphemistic Terms (2023.starsem-1)
Copied to clipboard
Patrick Lee, Iyanuoluwa Shode, Alain Trujillo, Yuan Zhao, Olumide Ojo, Diana Plancarte, Anna Feldman, Jing Peng
| Challenge: | Existing work on euphemism disambiguation tasks has focused on transformers . euphorias are expressions that soften the message they convey, therefore dictionary-based approaches are ineffective . |
| Approach: | They propose to annotate PETs for vagueness and use transformers to classify PETs . they perform euphemism disambiguation experiments in three different languages . |
| Outcome: | The proposed models perform well in English euphemism disambiguation task . preliminary results will be used to launch future work . |
Saliency-based Multi-View Mixed Language Training for Zero-shot Cross-lingual Classification (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent multilingual pre-trained models have been demonstrated effective in many cross-lingual tasks. |
| Approach: | They propose a framework that leverages code-switched data with multi-view learning to fine-tune XLM-R. |
| Outcome: | The proposed model achieves state-of-the-art on zero-shot cross-lingual sentiment classification and dialogue state tracking tasks. |
Learning Compact Metrics for MT (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in machine translation and multilingual text generation have led researchers to adopt trained metrics such as COMET or BLEURT, which treat evaluation as a regression problem and use representations from multilingual pre-trained models such as XLM-RoBERTa or mBERT. |
| Approach: | They propose to use multilingual model capacity to improve model performance by transferring knowledge from one teacher to multiple students trained on related languages. |
| Outcome: | The proposed model yields 10.5% improvement over vanilla fine-tuning and reaches 92.6% of RemBERT’s performance using only a third of its parameters. |
MEDs for PETs: Multilingual Euphemism Disambiguation for Potentially Euphemistic Terms (2024.findings-eacl)
Copied to clipboard
Patrick Lee, Alain Chirino Trujillo, Diana Cuevas Plancarte, Olumide Ojo, Xinyi Liu, Iyanuoluwa Shode, Yuan Zhao, Anna Feldman, Jing Peng
| Challenge: | Euphemisms are a linguistic device used to soften or neutralize language that may otherwise be harsh or awkward to state directly. |
| Approach: | They train a multilingual transformer model to disambiguate potentially euphemistic terms in multilingual and cross-lingual settings. |
| Outcome: | The proposed model performs better than monolingual models on the disambiguation task compared to monolingual ones in multilingual and cross-lingual settings. |
Zero-Shot Dependency Parsing with Worst-Case Aware Automated Curriculum Learning (2022.acl-short)
Copied to clipboard
| Challenge: | Large multilingual pretrained language models such as mBERT and XLM-RoBERTa have been found to be effective for cross-lingual transfer of syntactic parsing models but only between related languages. |
| Approach: | They propose to use multi-task learning to dynamically optimize for parsing performance on outlier languages by using a multi-level learning approach. |
| Outcome: | The proposed method significantly outperforms uniform and size-proportional sampling in the zero-shot setting. |
Hyperpolyglot LLMs: Cross-Lingual Interpretability in Token Embeddings (2023.emnlp-main)
Copied to clipboard
| Challenge: | XLMs can support cross-lingual transfer learning with little to no additional training data. |
| Approach: | They describe a mechanism for cross-lingual transfer learning by measuring the properties of the initial token embedding layer. |
| Outcome: | The proposed model can be used to support cross-lingual transfer learning . the initial token embedding layer is expressive and interpretable . |
A Multi-layered Approach to Physical Commonsense Understanding: Creation and Evaluation of an Italian Dataset (2024.lrec-main)
Copied to clipboard
| Challenge: | Using a multilingual model, we examine the ability of large language models to perform reasoning tasks. |
| Approach: | They propose to use a multilingual model to analyze commonsense reasoning in large language models for Italian and to provide a semi-automated system to complete the annotation. |
| Outcome: | The proposed model performs at high-level classification tasks but its easoning is inconsistent and unverifiable, since it does not capture intermediate evidence. |
Improving Cross Lingual Transfer by Pretraining with Active Forgetting (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior work has shown that encoder-only LLMs show impressive cross lingual transfer of their capabilities from English to other languages. |
| Approach: | They propose a pretraining strategy that uses active forgetting to achieve similar cross lingual transfer in decoder-only LLMs. |
| Outcome: | The proposed model improves cross lingual transfer capabilities on non-English languages despite being trained on English data. |
Cross-lingual Editing in Multilingual Language Models (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing models editing techniques (METs) can efficiently update outdated LLMs without retraining. |
| Approach: | They propose a cross-lingual model editing paradigm where a fact is edited in one language and the subsequent update propagation is observed across other languages. |
| Outcome: | The proposed techniques perform well in multilingual models with knowledge stored in multiple languages. |
DIA-HARM: Dialectal Disparities in Harmful Content Detection Across 50 English Dialects (2026.acl-long)
Copied to clipboard
| Challenge: | Current disinformation detection systems are predominantly developed and evaluated on Standard American English (SAE) . however, their robustness to dialectal variation is unexplored. |
| Approach: | They propose a benchmark for evaluating disinformation detection robustness across 50 English dialects . they use multi-value's linguistically-grounded transformations to introduce D-CUBE (Dialectal Disinformation Detection Corpus) |
| Outcome: | The proposed model outperforms zero-shot LLMs in human-written dialects while AI-generated content remains stable. |
On the Language Neutrality of Pre-trained Multilingual Representations (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have focused on cross-linguality of contextual embeddings . however, they are only moderately language-neutral by default . |
| Approach: | They propose to use unsupervised centering and fitting an explicit projection on parallel data to achieve stronger language neutrality. |
| Outcome: | The proposed model outperforms existing models on XNLI and NER tasks. |
Subword Pooling Makes a Difference (2021.eacl-main)
Copied to clipboard
| Challenge: | Contextual word-representations use subword tokenization to handle large vocabularies and unknown words. |
| Approach: | They propose to use the first subword for morphological probing, POS tagging and NER to pool multiple subwords that correspond to a single word in contextual language models. |
| Outcome: | The proposed model outperforms two multilingual models on morphological probing, POS tagging and NER tasks in 9 languages. |
Transfer Learning and Distant Supervision for Multilingual Transformer Models: A Study on African Languages (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that results from high-resource languages cannot be easily transferred to realistic, low-resourced scenarios. |
| Approach: | They analyse performance of multilingual transformer models using available resources for Hausa, isiXhosa and NER and topic classification. |
| Outcome: | The proposed models can achieve with as little as 10 or 100 labeled sentences the same performance as baselines with much more supervised training data. |
Do Explicit Alignments Robustly Improve Multilingual Encoders? (2020.emnlp-main)
Copied to clipboard
| Challenge: | Explicit alignment objectives based on bitexts like Europarl and MultiUN have been shown to improve cross-lingual representations. |
| Approach: | They propose a new contrastive alignment objective that can better utilize bitexts . they propose to use a random sample of 1 million pair subset of OPUS data . |
| Outcome: | The proposed objective outperforms existing alignment objectives on a random 1 million pair subset of the OPUS dataset. |
An Empirical Study of Pre-trained Transformers for Arabic Information Extraction (2020.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual pre-trained Transformers have been shown to enable effective cross-lingual zero-shot transfer, but their performance on Arabic information extraction tasks is not well studied. |
| Approach: | They pre-train a bilingual BERT that is designed specifically for Arabic NLP and English-to-Arabic zero-shot transfer learning. |
| Outcome: | The pre-trained model significantly outperforms mBERT, XLM-RoBERTa, and AraBERT in both the supervised and zero-shot transfer settings. |
Model Selection for Cross-lingual Transfer (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing work has relied on English dev data to select among models that are fine-tuned with different learning rates, number of steps and other hyperparameters, often resulting in suboptimal choices. |
| Approach: | They propose a machine learning approach that uses the fine-tuned model’s internal representations to predict its cross-lingual capabilities. |
| Outcome: | The proposed model selects better than English validation data across twenty five languages, including eight low-resource languages, and often achieves comparable results to model selection using target language development data. |
Estimating Confidence of Predictions of Individual Classifiers and TheirEnsembles for the Genre Classification Task (2022.lrec-1)
Copied to clipboard
| Challenge: | Genre identification is a kind of non-topic text classification. genre is defined as a functional space. |
| Approach: | They propose to use SOTA to identify genres in non-topic texts . genres are functional and cannot be expressed just by some keywords . |
| Outcome: | The proposed models show that they perform better than their individual models in large datasets. |
Beware of Model Collapse! Fast and Stable Test-time Adaptation for Robust Question Answering (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) have achieved great success in question answering, but their robustness is insufficient to support their practical applications. |
| Approach: | They propose a method which regularizes the model's output and an efficient side block to reduce its inference time. |
| Outcome: | The proposed method achieves comparable or better results than previous TTA methods at a speed close to vanilla forward propagation, which is 1.8 to 4.4 speedup compared to previous methods. |
Monolingual Paraphrase Detection Corpus for Low Resource Pashto Language at Sentence Level (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing research on sentence-level paraphrase detection in Pashto has focused on English, but no work has been done on low-resource Pashtone. |
| Approach: | They propose to annotate sentences in Pashto to detect paraphrases . they will publicize a subset of 1,800 instances from their corpus, free from licensing issues. |
| Outcome: | The proposed corpus contains 6,727 sentences, encompassing 3,687 paraphrased and 3,040 non-paraphrased sentences. |
RoBERTa Low Resource Fine Tuning for Sentiment Analysis in Albanian (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent advances in the education domain have provided new opportunities for solving interesting, but difficult problems. |
| Approach: | They propose to use EduSenti to fine-tune language models for assigning sentiment to reviews of educators' performance annotated for sentiment, emotion and educational topic. |
| Outcome: | The proposed model is compared with an Albanian masked language trained model from the last XLM-RoBERTa checkpoint and shows that it is a good fit for the proposed model. |
UQA: Corpus for Urdu Question Answering (2024.lrec-main)
Copied to clipboard
| Challenge: | Urdu is a low-resource language with over 70 million native speakers . expanding the reach of NLP to languages other than English is crucial for advancing multilingual AI systems. |
| Approach: | They introduce a novel dataset for question answering and text comprehension in Urdu . they use a technique called EATS which preserves the answer spans in translated context paragraphs . |
| Outcome: | The proposed dataset preserves answer spans in translated context paragraphs. |