Laying Anchors: Semantically Priming Numerals in Language Modeling (2024.findings-naacl)
Copied to clipboard
| Challenge: | Numeracy is the comprehension of numbers, and numerals are important for comprehension. |
| Approach: | They propose methods to semantically prime numerals by generating anchors governed by the distribution of numeral in any corpus. |
| Outcome: | The proposed methods perform better on numeracy tasks for both in-domain and out-domain numerals. |
Similar Papers
Exploring Numeracy in Word Embeddings (P19-1)
Copied to clipboard
| Challenge: | Existing word embeddings are inadequate at capturing numerical properties of numbers. |
| Approach: | They propose to use word embeddings to capture numerical properties of numbers . they hope to develop methods which better capture numeric properties . |
| Outcome: | The proposed models lack the ability to capture numeric properties of numbers, the authors show . their findings provide a starting point for the development of better models . |
Learning Numeral Embedding (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing word embedding methods do not learn numeral embedds well because numerals are limited in number and their appearances in training corpora are highly scarce. |
| Approach: | They propose two numeral embedding methods that can handle the out-of-vocabulary problem for numerals. |
| Outcome: | The proposed methods can handle the out-of-vocabulary problem for numerals. |
Methods for Numeracy-Preserving Word Embeddings (2020.emnlp-main)
Copied to clipboard
Dhanasekar Sundararaman, Shijing Si, Vivek Subramanian, Guoyin Wang, Devamanyu Hazarika, Lawrence Carin
| Challenge: | Word embedding models capture semantic relationships between words but fail to capture numerical properties associated with numbers. |
| Approach: | They propose a method to assign and learn embeddings for numbers using word embedders. |
| Outcome: | The proposed model outperforms pre-trained word embedding models across multiple examples of two tasks. |
Do NLP Models Know Numbers? Probing Numeracy in Embeddings (D19-1)
Copied to clipboard
| Challenge: | Existing models cannot capture numeracy, but they can be useful for complex reasoning tasks. |
| Approach: | They investigate numerical reasoning capabilities of a question-answering model . they probe token embedding methods on synthetic list maximum, number decoding, and addition tasks. |
| Outcome: | The proposed model excels on questions that require numerical reasoning, i.e., it already captures numeracy. |
Pre-trained Language Models Learn Remarkably Accurate Representations of Numbers (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work showed limited success in probing numeric values from models’ representations, indicating that these errors can be attributed to the inherent unreliability of distributionally learned embeddings in representing exact quantities. |
| Approach: | They propose a probing technique that decodes numeric values from input embeddings with near-perfect accuracy across a range of open-source LMs. |
| Outcome: | The proposed probing technique decodes numeric values from input embeddings with near-perfect accuracy across a range of open-source LMs. |
Representing Numbers in NLP: a Survey and a Vision (2021.naacl-main)
Copied to clipboard
| Challenge: | Numeracy is an essential skill for language understanding since numbers are often interspersed in text. |
| Approach: | They propose a comprehensive taxonomy of tasks and methods to represent numbers in text . they synthesize best practices for representing numbers in texts and articulate a vision for holistic numeracy . |
| Outcome: | The proposed model synthesizes best practices for representing numbers in text . it argues that the model is more effective than other approaches . |
Numeracy for Language Models: Evaluating and Improving their Ability to Predict Numbers (P18-1)
Copied to clipboard
| Challenge: | Numeracy is the ability to understand and work with numbers. |
| Approach: | They propose a neural architecture that uses a continuous probability density function to model numerals from an open vocabulary using hierarchical models. |
| Outcome: | The proposed model reduces errors by 18% and 54% on clinical and scientific datasets compared to the second best model for each dataset . |
How to Leverage Digit Embeddings to Represent Numbers? (2025.coling-main)
Copied to clipboard
| Challenge: | Existing numerical reasoning models struggle to understand numbers, despite simple generalisations. |
| Approach: | They propose to use mathematical priors to compute digit embeddings and explicitly incorporate them into transformer models by adding a special token to the input embedded digits or introducing an additional loss function to enhance correct predictions. |
| Outcome: | The proposed method is compatible with any pretrained model and easy to implement. |
Language Models Learn Universal Representations of Numbers and Here’s Why You Should Care (2026.acl-long)
Copied to clipboard
Michal Štefánik, Timothee Mickus, Marek Kadlčík, Bertram Højer, Michal Spiegel, Raúl Vázquez, Aman Sinha, Josef Kuchař, Philipp Mondorf, Pontus Stenetorp
| Challenge: | Prior work has shown that large language models (LLMs) often converge to accurate input embedding for numbers, based on sinusoidal representations. |
| Approach: | They show that large language models often converge to accurate input embedding for numbers, based on sinusoidal representations. |
| Outcome: | The proposed representations are strikingly systematic, and are interchangeable in a large swathe of experimental setups. |
Numeracy enhances the Literacy of Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Specialized number representations have shown improvements on numerical reasoning tasks like arithmetic word problems and masked number prediction. |
| Approach: | They propose to use six different number encoders to improve masked word prediction by avoiding conflating nominal and ordinal number occurrences. |
| Outcome: | The proposed representations improve masked word prediction accuracy and generalize to contexts without annotated numbers. |