Papers by Marek Kadlčík
Pre-trained Language Models Learn Remarkably Accurate Representations of Numbers (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work showed limited success in probing numeric values from models’ representations, indicating that these errors can be attributed to the inherent unreliability of distributionally learned embeddings in representing exact quantities. |
| Approach: | They propose a probing technique that decodes numeric values from input embeddings with near-perfect accuracy across a range of open-source LMs. |
| Outcome: | The proposed probing technique decodes numeric values from input embeddings with near-perfect accuracy across a range of open-source LMs. |
Concept-aware Data Construction Improves In-context Learning of Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent work curating in-context learners assumes that ICL emerges from vast over-parametrization or the scale of multitask training. |
| Approach: | They propose a framework for constructing training scenarios that make it beneficial for the LM to learn to utilize the analogical reasoning concepts from demonstrations. |
| Outcome: | The proposed framework makes it beneficial for the LM to learn to utilize the analogical reasoning concepts from demonstrations and fares comparably to previous in-context learners trained in large-scale multitask learning requiring magnitudes of more training data. |
Calc-X and Calcformers: Empowering Arithmetical Chain-of-Thought through Interaction with Symbolic Systems (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing language models are notoriously inclined to make factual errors in tasks requiring arithmetic computation. |
| Approach: | They propose to combine existing chain-of-thought datasets into a unified format that can be used to train and evaluate open-source calculator-using models. |
| Outcome: | The proposed model doubles the accuracy of generating correct results compared to baseline models. |
Language Models Learn Universal Representations of Numbers and Here’s Why You Should Care (2026.acl-long)
Copied to clipboard
Michal Štefánik, Timothee Mickus, Marek Kadlčík, Bertram Højer, Michal Spiegel, Raúl Vázquez, Aman Sinha, Josef Kuchař, Philipp Mondorf, Pontus Stenetorp
| Challenge: | Prior work has shown that large language models (LLMs) often converge to accurate input embedding for numbers, based on sinusoidal representations. |
| Approach: | They show that large language models often converge to accurate input embedding for numbers, based on sinusoidal representations. |
| Outcome: | The proposed representations are strikingly systematic, and are interchangeable in a large swathe of experimental setups. |
Self-training Language Models for Arithmetic Reasoning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work improves the reasoning capabilities of language models by scaling training data to more diverse or complex collections, but reaching further improvements becomes exceedingly expensive. |
| Approach: | They propose to use implicit feedback to improve models' reasoning capabilities by training from implicit feedback. |
| Outcome: | The proposed model can reach a correct result in +13.9% and +25.9% more cases than previous models, underlining the importance of actuality of self-training feedback. |
Can Out-of-Distribution Evaluations Uncover Reliance on Prediction Shortcuts? A Case Study in Question Answering (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work assesses models’ generalization capabilities through the lens of performance on out-of-distribution (OOD) datasets. |
| Approach: | They challenge this assumption by comparing OOD evaluations with failure modes documented in existing question-answering (QA) models. |
| Outcome: | The proposed evaluations show that the models' generalization capabilities are under-performing on out-of-distribution datasets, while others are underperforming on in-difference datasets. |
Towards the Roots of the Negation Problem: A Multilingual NLI Dataset and Model Scaling Analysis (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Negations are key to determining sentence meaning, making them essential for logical reasoning. |
| Approach: | They construct and publish two new textual entailment datasets in four languages with paired examples differing in negation. |
| Outcome: | The results show that increasing the model size may improve the models’ ability to handle negations. |