Papers by Avijit Thawani
Representing Numbers in NLP: a Survey and a Vision (2021.naacl-main)
Copied to clipboard
| Challenge: | Numeracy is an essential skill for language understanding since numbers are often interspersed in text. |
| Approach: | They propose a comprehensive taxonomy of tasks and methods to represent numbers in text . they synthesize best practices for representing numbers in texts and articulate a vision for holistic numeracy . |
| Outcome: | The proposed model synthesizes best practices for representing numbers in text . it argues that the model is more effective than other approaches . |
Numeracy enhances the Literacy of Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Specialized number representations have shown improvements on numerical reasoning tasks like arithmetic word problems and masked number prediction. |
| Approach: | They propose to use six different number encoders to improve masked word prediction by avoiding conflating nominal and ordinal number occurrences. |
| Outcome: | The proposed representations improve masked word prediction accuracy and generalize to contexts without annotated numbers. |
Learn Your Tokens: Word-Pooled Tokenization for Language Modeling (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Language models typically tokenize text into subwords, using a deterministic, hand-engineered heuristic of combining characters into longer surface-level strings such as ‘ing’ or whole words. |
| Approach: | They propose a 'learn your tokens' scheme which pooles bytes/characters into word representations and decodes individual characters/bytes per word in parallel. |
| Outcome: | The proposed tokenizer outperforms subword models and byte/character models over the word boundary and outperformed on rare words by a factor of 30! |