Papers by Michal Štefánik
Pre-trained Language Models Learn Remarkably Accurate Representations of Numbers (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work showed limited success in probing numeric values from models’ representations, indicating that these errors can be attributed to the inherent unreliability of distributionally learned embeddings in representing exact quantities. |
| Approach: | They propose a probing technique that decodes numeric values from input embeddings with near-perfect accuracy across a range of open-source LMs. |
| Outcome: | The proposed probing technique decodes numeric values from input embeddings with near-perfect accuracy across a range of open-source LMs. |
Concept-aware Data Construction Improves In-context Learning of Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent work curating in-context learners assumes that ICL emerges from vast over-parametrization or the scale of multitask training. |
| Approach: | They propose a framework for constructing training scenarios that make it beneficial for the LM to learn to utilize the analogical reasoning concepts from demonstrations. |
| Outcome: | The proposed framework makes it beneficial for the LM to learn to utilize the analogical reasoning concepts from demonstrations and fares comparably to previous in-context learners trained in large-scale multitask learning requiring magnitudes of more training data. |
Calc-X and Calcformers: Empowering Arithmetical Chain-of-Thought through Interaction with Symbolic Systems (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing language models are notoriously inclined to make factual errors in tasks requiring arithmetic computation. |
| Approach: | They propose to combine existing chain-of-thought datasets into a unified format that can be used to train and evaluate open-source calculator-using models. |
| Outcome: | The proposed model doubles the accuracy of generating correct results compared to baseline models. |
People and Places of Historical Europe: Bootstrapping Annotation Pipeline and a New Corpus of Named Entities in Late Medieval Texts (2023.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained named entity recognition models are inaccurate on modern corpora due to differences in language OCR errors. |
| Approach: | They develop a named entity recognition (NER) corpus of 3.6M sentences from medieval charters written mainly in Czech, Latin, and German. |
| Outcome: | The proposed model achieves entity-level Precision of 72.81–93.98% with 58.14–81.77% Recall on a manually-annotated test dataset. |
Language Models Learn Universal Representations of Numbers and Here’s Why You Should Care (2026.acl-long)
Copied to clipboard
Michal Štefánik, Timothee Mickus, Marek Kadlčík, Bertram Højer, Michal Spiegel, Raúl Vázquez, Aman Sinha, Josef Kuchař, Philipp Mondorf, Pontus Stenetorp
| Challenge: | Prior work has shown that large language models (LLMs) often converge to accurate input embedding for numbers, based on sinusoidal representations. |
| Approach: | They show that large language models often converge to accurate input embedding for numbers, based on sinusoidal representations. |
| Outcome: | The proposed representations are strikingly systematic, and are interchangeable in a large swathe of experimental setups. |
Self-training Language Models for Arithmetic Reasoning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work improves the reasoning capabilities of language models by scaling training data to more diverse or complex collections, but reaching further improvements becomes exceedingly expensive. |
| Approach: | They propose to use implicit feedback to improve models' reasoning capabilities by training from implicit feedback. |
| Outcome: | The proposed model can reach a correct result in +13.9% and +25.9% more cases than previous models, underlining the importance of actuality of self-training feedback. |
Can Out-of-Distribution Evaluations Uncover Reliance on Prediction Shortcuts? A Case Study in Question Answering (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work assesses models’ generalization capabilities through the lens of performance on out-of-distribution (OOD) datasets. |
| Approach: | They challenge this assumption by comparing OOD evaluations with failure modes documented in existing question-answering (QA) models. |
| Outcome: | The proposed evaluations show that the models' generalization capabilities are under-performing on out-of-distribution datasets, while others are underperforming on in-difference datasets. |
Soft Alignment Objectives for Robust Adaptation of Language Generation (2023.acl-long)
Copied to clipboard
| Challenge: | Domain adaptation is a common approach for generative language models, but it is notorious for over-specialization to the target domain, resulting in catastrophic forgetting. |
| Approach: | They propose to build training objectives on a semantic similarity of predicted tokens to the reference and avoid catastrophic forgetting of adaptation by preserving adaptation in-domain quality. |
| Outcome: | The proposed objectives mitigate catastrophic forgetting while preserving the adaptation in-domain quality while reducing computational costs. |
Adaptor: Objective-Centric Adaptation Framework for Language Models (2022.acl-demo)
Copied to clipboard
| Challenge: | Adaptor library aims to simplify complex training processes requiring customizations. |
| Approach: | They introduce Adaptor library which transposes traditional model-centric approach to objective-centric training pipeline with Objective as central abstraction. |
| Outcome: | The proposed framework simplifies training processes and improves reproducibility. |
Think Twice: Measuring the Efficiency of Eliminating Prediction Shortcuts of Question Answering Models (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing work shows that Large Language Models (LLMs) are not robust to complex language understanding tasks due to reliance on spurious correlations of training datasets. |
| Approach: | They propose a method for measuring model reliance on spurious features by exploiting chosen biases on out-of-distribution (OOD) datasets. |
| Outcome: | The proposed method shows that the reported OOD gains of debiasing methods can't be explained by mitigated reliance on biased features, suggesting that biases are shared among different QA datasets. |
Towards the Roots of the Negation Problem: A Multilingual NLI Dataset and Model Scaling Analysis (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Negations are key to determining sentence meaning, making them essential for logical reasoning. |
| Approach: | They construct and publish two new textual entailment datasets in four languages with paired examples differing in negation. |
| Outcome: | The results show that increasing the model size may improve the models’ ability to handle negations. |