Papers with ALBERT
Privacy-Preserving Federated Learning for Hate Speech Detection (2025.naacl-srw)
Copied to clipboard
| Challenge: | a federated learning system with differential privacy is tailored to low-resource languages . data with fewer than 20 sentences per client struggled due to excessive noise . |
| Approach: | They propose a federated learning system with differential privacy for hate speech detection . they fine-tuned pre-trained language models to find it to be the most effective . |
| Outcome: | The proposed learning system outperforms other models in low-resource languages . balanced datasets and augmenting hateful data with non-hateful examples proved critical . |
Speedy Gonzales: A Collection of Fast Task-Specific Models for Spanish (2024.starsem-1)
Copied to clipboard
| Challenge: | Large language models (LLMs) are a common and successful approach to language and retrieval tasks. |
| Approach: | They evaluate the available large language models in Spanish and then use knowledge distillation to refine and distill them. |
| Outcome: | The proposed models are fine-tuned and distilled on knowledge distillation and are available on huggingface.co/dccuchile. |
PCEE-BERT: Accelerating BERT Inference via Patient and Confident Early Exiting (2022.findings-naacl)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) are the state-of-the-art (SOTA) models for natural language processing (NLP). |
| Approach: | They propose a patient and confident early exiting BERT (PCEE-BERT) that can work with different PLMs and popular model compression methods. |
| Outcome: | The proposed method outperforms existing models on the GLUE benchmarks and achieves different speed-up ratios. |
‘Am I the Bad One’? Predicting the Moral Judgement of the Crowd Using Pre–trained Language Models (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing studies on NLP touch upon moral contexts in text. |
| Approach: | They construct a dataset that can be used for moral judgement tasks on a popular reddit subreddit. |
| Outcome: | The proposed model passes moral judgements on posts from a popular reddit subreddit . it shows that the model can be fine tuned and improves across the datasets . |
Using Adversarial Attacks to Reveal the Statistical Bias in Machine Reading Comprehension Models (2021.acl-short)
Copied to clipboard
| Challenge: | Pre-trained language models have achieved human-level performance on many Machine Reading Comprehension (MRC) tasks, but it remains unclear whether these models truly understand language or answer questions by exploiting statistical biases in datasets. |
| Approach: | They propose a method to attack MRC models by exposing statistical biases in a RACE dataset and propose an augmented training method that can greatly reduce models’ statistical bias. |
| Outcome: | The proposed method can reduce models’ statistical biases from human-level performance to chance-level. |
A Closer Look at Linguistic Knowledge in Masked Language Models: The Case of Relative Clauses in American English (2020.coling-main)
Copied to clipboard
Marius Mosbach, Stefania Degaetano-Ortlieb, Marie-Pauline Krielke, Badr M. Abdullah, Dietrich Klakow
| Challenge: | Despite the high performance of transformer-based language models, we still lack understanding of the kind of linguistic knowledge they learn and rely on. |
| Approach: | They evaluate three transformer-based language models and test their grammatical and semantic knowledge by sentence-level probing, diagnostic cases, and masked prediction tasks. |
| Outcome: | The models capture grammatical and semantic knowledge, but they lack model-specific weaknesses especially on semantic knowledge. |
Auto-Debias: Debiasing Masked Language Models with Automated Biased Prompts (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods to mitigate human-like biases in pretrained language models are based on external corpora and require a distribution alignment loss to mitigate them. |
| Approach: | They propose an automatic method to mitigate biases in pretrained language models by searching for biased prompts such that cloze-style completions are the most different with respect to different demographic groups. |
| Outcome: | The proposed method reduces biases in pretrained language models, including gender and racial bias, and improves fairness of the models. |
CeeBERT: Cross-Domain Inference in Early Exit BERT (2024.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained Language Models suffer in inference latency due to their large size. |
| Approach: | They propose an online learning algorithm that dynamically determines early exits of samples based on the level of confidence observed at intermediate layers. |
| Outcome: | The proposed algorithm can speed up the BERT/ALBERT models by 2 - 3.1 with minimal drop in accuracy. |
Larger Probes Tell a Different Story: Extending Psycholinguistic Datasets Via In-Context Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Language model probing is often used to test specific capabilities of models, but results are limited when benchmarks are small and lack statistical power. |
| Approach: | They extend existing NEG-136 and ROLE-88 benchmarks to 750 sentence pairs and create an extended negation dataset using template-based generation. |
| Outcome: | The proposed datasets show that models perform better on the extended datasets than on the smaller benchmarks. |
BERTAC: Enhancing Transformer-based Language Models with Adversarially Pretrained Convolutional Neural Networks (2021.acl-long)
Copied to clipboard
| Challenge: | Existing models of NLP are fading away, but new ones are needed to maintain their dominance. |
| Approach: | They propose a method to pretrain a CNN using Wikipedia data and integrate it with standard TLMs. |
| Outcome: | The proposed method outperforms the original ALBERT on GLUE tasks and achieves similar performance to SOTA on open-domain QA tasks. |
Towards a Comprehensive Understanding and Accurate Evaluation of Societal Biases in Pre-Trained Transformers (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing pre-trained language models are not fully considered for societal biases . pre-training models can be useful for many NLP tasks, but they can be harmful when used at scale. |
| Approach: | They investigate gender and racial bias across pre-trained language models . they evaluate bias within pre-trainers using three metrics: WEAT, sequence likelihood, and pronoun ranking. |
| Outcome: | The proposed model fails to detect gender and racial biases in pre-trained models . the model is ineffective when word embedding, demonstrating the need for more robust bias testing in transformers. |
Life after BERT: What do Other Muppets Understand about Language? (2022.acl-long)
Copied to clipboard
| Challenge: | Existing pre-trained transformer analysis studies focus on one or two model families at a time, overlooking the variability of the architecture and pre-training objectives. |
| Approach: | They utilize oLMpics bench- mark and psycholinguistic probing datasets for a diverse set of 29 models including T5, BART, and ALBERT. |
| Outcome: | The proposed model fails to resolve compositional questions in a zero-shot fashion, suggesting that pre-training objectives are not predictive of a model’s linguistic capabilities. |
On the Interplay Between Fine-tuning and Sentence-level Probing for Linguistic Knowledge in Pre-trained Transformers (2020.findings-emnlp)
Copied to clipboard
| Challenge: | linguistic knowledge encoded in pre-trained contextual embeddings is poorly understood . fine-tuning can be used to investigate the representations of pre-train models . |
| Approach: | They propose to investigate fine-tuning of contextualized embedding models through sentence-level probing. |
| Outcome: | The proposed method improves probing accuracy for three pre-trained models. |
Models Teaching Models: Improving Model Accuracy with Slingshot Learning (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing methods to label datasets are expensive and require human labor . a semi-supervised method that augments a small dataset with labels reduces the cost of using simpler methods . |
| Approach: | They propose a semi-supervised method to augment a human-labeled dataset with labels from a teacher model to slingshot the performance of a student model. |
| Outcome: | The proposed method reduces the accuracy trade-off required to use simpler methods without disrupting their benefits. |
Robust Machine Reading Comprehension by Learning Soft labels (2020.coling-main)
Copied to clipboard
| Challenge: | Neural models have achieved great success on the task of machine reading comprehension, which are typically trained on hard labels. |
| Approach: | They propose a robust training method for machine reading comprehension models to address label sparseness problem by using three strategies to train models on soft labels. |
| Outcome: | The proposed method improves the baseline model performance and achieves state-of-the-art performance on NewsQA and QUOREF. |
Robust Bias Detection in MLMs and its Application to Human Trait Ratings (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to assess demographic bias in MLMs ignore random variability of templates and target concepts, and neglect bias quantification. |
| Approach: | They propose a systematic statistical approach to assess bias in MLMs using mixed models to account for random effects, pseudo-perplexity weights for sentences derived from templates and quantify bias using statistical effect sizes. |
| Outcome: | The proposed method matches on bias scores in magnitude and direction with small to medium effect sizes. |
Infusing Disease Knowledge into BERT for Health Question Answering, Medical Inference and Disease Name Recognition (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to augment pre-trained language models with disease knowledge are lacking. |
| Approach: | They propose a method to augment BERT-like pre-trained language models with disease knowledge. |
| Outcome: | The proposed method improves on a suite of BERT models over three tasks. |
Fight Fire with Fire: Fine-tuning Hate Detectors using Large Samples of Generated Hate Speech (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for hate speech detection are limited in size and lack of labeled datasets. |
| Approach: | They employ pretrained language models to generate large amounts of hate speech sequences from available labeled examples. |
| Outcome: | The proposed model improves generalization significantly and consistently within and across data distributions. |
IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages (2020.findings-emnlp)
Copied to clipboard
Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, Gokul N.C., Avik Bhattacharyya, Mitesh M. Khapra, Pratyush Kumar
| Challenge: | In this paper, we present NLP resources for 11 major Indian languages . distributional representations are the cornerstone of modern NLP, authors say . |
| Approach: | They introduce NLP resources for 11 major Indian languages from two major language families . monolingual corpora contains 8.8 billion tokens across all 11 languages and Indian English . they also compile a benchmark for Indian language NLU to evaluate their results . |
| Outcome: | The monolingual corpora contains 8.8 billion tokens across all 11 languages and Indian English . the pre-trained language models are based on the compact ALBERT model . |
ALBETO and DistilBETO: Lightweight Spanish Language Models (2022.lrec-1)
Copied to clipboard
| Challenge: | Recent advances in pre-trained language models have made them more popular . however, there are still limited versions of these models for other languages . |
| Approach: | They present ALBETO and DistilBETO which are versions of ALBERT and DistillBERT pre-trained exclusively on Spanish corpora. |
| Outcome: | The proposed models outperform BETO and ALBERT on Spanish datasets . the models outpace BETO on MLDoc, PAWS-X, XNLI, MLQA, SQAC and XQuAD datasets. |
Pretrained Language Model Embryology: The Birth of ALBERT (2020.emnlp-main)
Copied to clipboard
| Challenge: | Pretraining of pretrained models (LMs) has been extensively studied, but what happened during pretraining is rarely studied. |
| Approach: | They propose to use a totipotent language model to study pretraining behavior . they find that linguistic knowledge and world knowledge do not generally improve as pretraining proceeds, nor do downstream tasks’ performance. |
| Outcome: | The model learns to reconstruct and predict tokens of different parts of speech (POS) in different learning speeds during pretraining. |
Eye-Tracking Features Masking Transformer Attention in Question-Answering Tasks (2024.lrec-main)
Copied to clipboard
| Challenge: | Eye movement features are considered to be direct signals reflecting human attention distribution with a low cost to obtain, inspiring researchers to augment language models with eye-tracking (ET) data. |
| Approach: | They select first fixation duration (FFD) and total reading time (TRT) as the cognitive signals to guide Transformer attention in question-answering tasks. |
| Outcome: | The proposed models improve but compromise stability when augmenting with ET data. |
AIDER: a Robust and Topic-Independent Framework for Detecting AI-Generated Text (2025.coling-main)
Copied to clipboard
| Challenge: | Current fine-tuned detectors lack robustness against adversarial attacks and struggle with out-of-distribution topics, limiting their practical applicability. |
| Approach: | They propose a topic-independent framework for detecting AI-generated text . it leverages the ALBERT model for topic content disentanglement, enhancing transferability to unseen topics. |
| Outcome: | The proposed framework outperforms state-of-the-art methods in detecting human-written and AI-generated content under adversarial and topic-varied conditions. |
Meta Distant Transfer Learning for Pre-trained Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Notable PLMs are available for text classification tasks, but performance of PLM on downstream tasks may be limited by the availability of training set. |
| Approach: | They propose a meta-learning framework to learn the transferable knowledge across tasks using PLMs. |
| Outcome: | The proposed framework outperforms baselines on seven datasets and is task-agnostic and unbiased. |
The Architectural Bottleneck Principle (2022.emnlp-main)
Copied to clipboard
| Challenge: | a recent study examined how much information a model's representations contain . a new approach to probing is to look exactly like the component . |
| Approach: | They propose a new probing principle that aims to estimate how much information a model could extract from its representations. |
| Outcome: | The proposed probes extract syntactic information from the representations of a neural network . the proposed probe is based on the architectural bottleneck principle . |
mALBERT: Is a Compact Multilingual BERT Model Still Worth It? (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies on the ethical and ecological impact of pre-trained language models raise questions about the temporal, financial, and environmental aspects of such models. |
| Approach: | They propose to focus on smaller models, such as compact models like ALBERT, which are more ecologically virtuous than these PLMs. |
| Outcome: | The proposed model is compared with classical multilingual models and is ethically virtuous. |