Monotonic Representation of Numeric Attributes in Language Models (2024.acl-short)
Copied to clipboard
| Challenge: | Language models (LMs) can express factual knowledge involving numeric properties such as Karl Popper was born in 1902, but how this information is encoded in the model’s internal representations is not understood well. |
| Approach: | They propose a method for finding and editing representations of numeric properties such as Karl Popper’s birth year. |
| Outcome: | The proposed method can express an increasingly late birthyear by patching activations along a “birthyear” direction. |
Similar Papers
Language Models Learn Universal Representations of Numbers and Here’s Why You Should Care (2026.acl-long)
Copied to clipboard
Michal Štefánik, Timothee Mickus, Marek Kadlčík, Bertram Højer, Michal Spiegel, Raúl Vázquez, Aman Sinha, Josef Kuchař, Philipp Mondorf, Pontus Stenetorp
| Challenge: | Prior work has shown that large language models (LLMs) often converge to accurate input embedding for numbers, based on sinusoidal representations. |
| Approach: | They show that large language models often converge to accurate input embedding for numbers, based on sinusoidal representations. |
| Outcome: | The proposed representations are strikingly systematic, and are interchangeable in a large swathe of experimental setups. |
Pre-trained Language Models Learn Remarkably Accurate Representations of Numbers (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work showed limited success in probing numeric values from models’ representations, indicating that these errors can be attributed to the inherent unreliability of distributionally learned embeddings in representing exact quantities. |
| Approach: | They propose a probing technique that decodes numeric values from input embeddings with near-perfect accuracy across a range of open-source LMs. |
| Outcome: | The proposed probing technique decodes numeric values from input embeddings with near-perfect accuracy across a range of open-source LMs. |
NumeroLogic: Number Encoding for Enhanced LLMs’ Numerical Reasoning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Language models struggle with numerical and arithmetical tasks, such as multiplying 3-digit numbers. |
| Approach: | They propose a method to include the count of digits before each number instead of “42”. |
| Outcome: | The proposed format improves the reasoning process before generating the actual number. |
How Do Language Models Acquire Character-Level Information? (2026.eacl-long)
Copied to clipboard
| Challenge: | Language models (LMs) implicitly encode character-level information, despite not being explicitly provided during training. |
| Approach: | They analyze how language models acquire character-level knowledge by comparing them to standard settings. |
| Outcome: | The results show that LMs do not treat words as opaque tokens, but instead treat them as tokens. |
Language Models Use Monotonicity to Assess NPI Licensing (2021.findings-acl)
Copied to clipboard
| Challenge: | Neural language models (LMs) have become powerful approximators of human language . fewer studies have been done on what kind of formal semantic features are encoded by LMs . |
| Approach: | They propose a series of experiments that investigate the semantic knowledge of language models . they use diagnostic classifiers, linguistic acceptability tasks and a ranking method to investigate the models' inner workings. |
| Outcome: | The proposed method can be applied to LMs trained on filtered corpora and gain stronger insights into their generalizations. |
The Geometry of Numerical Reasoning: Language Models Compare Numeric Properties in Linear Subspaces (2025.naacl-short)
Copied to clipboard
| Challenge: | Existing studies have focused on simple factual recall, but we have not explored how this is used in more complex queries. |
| Approach: | They propose to identify low-dimensional subspaces which encode numerical attributes associated with entities in comparison prompts. |
| Outcome: | The proposed model can answer numeric comparison questions using a low-dimensional subspace of theembedding space. |
Language Models Encode the Value of Numbers Linearly (2025.coling-main)
Copied to clipboard
| Challenge: | Existing studies show that large language models encode the value of numbers linearly. |
| Approach: | They construct a large language model and use linear probes to read out input numbers from hidden states. |
| Outcome: | The proposed model encodes the value of numbers linearly, and can store the outputs via simple vector additions. |
Language Models Encode Numbers Using Digit Representations in Base 10 (2025.naacl-short)
Copied to clipboard
| Challenge: | Large language models (LLMs) often make errors when handling simple numerical tasks . a natural hypothesis is that these errors stem from how LLMs represent numbers . |
| Approach: | They propose to examine how LLMs represent numbers with circular representations per digit . they propose to use digit-wise representations to shed light on errors on numerical tasks . |
| Outcome: | The proposed model is internally represented with individual circular representations per-digit in base 10 . the proposed model could be used to analyze numerical mechanisms in large language models . |
Do Language Embeddings capture Scales? (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained Language Models possess significant linguistic, common sense and factual knowledge, but are short of the capability required for general common-sense reasoning. |
| Approach: | They propose to train pretrained language models with a method of canonicalizing numbers . they address a task which is also pre-requisite for general common-sense reasoning . |
| Outcome: | The proposed model can answer questions about common sense and linguistics, but lacks the capability to answer questions on scalar attributes. |
The emergence of number and syntax units in LSTM language models (N19-1)
Copied to clipboard
| Challenge: | a recent study shows that LSTMs can capture syntax-sensitive generalizations such as long-distance number agreement. |
| Approach: | They investigate the inner mechanics of number tracking in LSTMs at the single neuron level . they find that long-distance number information is largely managed by two "number units" importantly, the behaviour of these units is partially controlled by other units to track syntactic structure . |
| Outcome: | The proposed model is based on a language model with a long-distance number agreement task. |