Papers with BERT-large
Dynamic Masking Rate Schedules for MLM Pretraining (2024.eacl-short)
Copied to clipboard
| Challenge: | Most transformers trained with the Masked Language Modeling objective use a fixed masking rate of 15%, but wettig et al. ( 2022) found that the standard 15% masking rates are suboptimal for a variety of model settings and recommended a higher rate. |
| Approach: | They propose to dynamically schedule the masking rate throughout training to improve GLUE accuracy by 0.46% and 0.25% in BERT-base and BERT large, respectively. |
| Outcome: | The proposed method improves GLUE accuracy by up to 0.46% and 0.25% in BERT-base and BERT large compared to baselines. |
Revealing Redundant Syntax in Large Language Models through Multi-Hop Dependency Paths (2026.findings-eacl)
Copied to clipboard
| Challenge: | Prior work on attention–syntax alignment has focused on single-hop Universal Dependency edges (DPs). |
| Approach: | They extract 2–3 hop MDPs from UD-parsed English and quantify head–relation alignment with an Unlabeled Attachment Score (UAS)-style metric modified for causal masking in decoder-only models. |
| Outcome: | The authors show that head alignments are overlapped and specialized . the head alignment is measurable in large language models trained on raw text . |
FOOL ME IF YOU CAN! An Adversarial Dataset to Investigate the Robustness of LMs in Word Sense Disambiguation (2024.emnlp-main)
Copied to clipboard
Mohamad Ballout, Anne Dedert, Nohayr Abdelmoneim, Ulf Krumnack, Gunther Heidemann, Kai-Uwe Kühnberger
| Challenge: | Word sense disambiguation (WSD) is a key task in natural language processing . however, these models struggle with recognizing semantic boundaries in adversarial contexts . |
| Approach: | They propose to use a coarse-grained WSD dataset to assess model robustness . they found that some models struggled to correctly disambiguate homonyms in adversarial contexts . |
| Outcome: | The proposed dataset includes four test sets to assess the robustness of language models in WSD tasks. |
Balanced Joint Adversarial Training for Robust Intent Detection and Slot Filling (2020.coling-main)
Copied to clipboard
| Challenge: | Existing joint models for intent detection and slot filling show insufficient robustness . however, some small changes of inputs can fool the models to produce wrong predictions . |
| Approach: | They propose a joint adversarial training model that generates adversarials to attack the joint model and trains the model to defend against the adversarial examples. |
| Outcome: | The proposed model achieves significantly higher scores and improves robustness on two datasets. |