Papers by Gábor Berend
SUE: Sparsity-based Uncertainty Estimation via Sparse Dictionary Learning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to estimate uncertainty use predictive confidence, structural characteristics of representation space, or stochastic variation in model outputs. |
| Approach: | They propose a new uncertainty estimation framework based on sparse dictionary learning by identifying dictionary atoms associated with misclassified samples. |
| Outcome: | The proposed framework outperforms or matches existing methods on several NLU benchmarks and sentiment analysis benchmarks. |
Masked Latent Semantic Modeling: an Efficient Pre-training Alternative to Masked Language Modeling (2023.findings-acl)
Copied to clipboard
| Challenge: | a recent study suggests that masked language models are a useful pre-training technique for natural language processing . a study using mlms pre-trained by a team of researchers has improved performance . |
| Approach: | They propose an alternative to the classic masked language modeling paradigm . they use an unsupervised technique which uses sparse coding to make the prediction possible . |
| Outcome: | The proposed technique improves on pre-trained models compared to vanilla MLM . the proposed model returns distributions over their vocabulary peaking at plausible substitutes . |
Sparsity Makes Sense: Word Sense Disambiguation Using Sparse Contextualized Word Representations (2020.emnlp-main)
Copied to clipboard
| Challenge: | Using sparse word embeddings is highly applicable for word sense disambiguation (WSD) . |
| Approach: | They propose an overcomplete set of semantic basis vectors that allows for sparse word representations. |
| Outcome: | The proposed framework achieves an aggregated F score of 78.8 over five standard word sense disambiguating benchmark datasets. |
Changing the Basis of Contextual Representations with Explicit Semantics (2021.acl-srw)
Copied to clipboard
| Challenge: | Existing transformer-based contextual representations are opaque as their latent dimensions are not directly interpretable. |
| Approach: | They propose an algorithm where the output representation expresses human-interpretable information of each dimension. |
| Outcome: | The proposed transformations are able to predict supersense category of a word by looking for its transformed coordinate with the largest coefficient. |
Combating the Curse of Multilinguality in Cross-Lingual WSD by Aligning Sparse Contextualized Word Representations (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing approaches to handle knowledge acquisition bottlenecks in multilingual training are limited due to the curse of multilinguality. |
| Approach: | They propose to use large pre-trained monolingual language models in cross lingual zero-shot word sense disambiguation coupled with a contextualized mapping mechanism. |
| Outcome: | The proposed model improves the average F-score by nearly 6.5 points over 17 target languages. |