Papers by Thomas Hofmann
On the Effect of (Near) Duplicate Subwords in Language Modelling (2024.findings-acl)
Copied to clipboard
| Challenge: | Tokenisation is a core part of language models but can lead to less efficient training because it removes character-level information. |
| Approach: | They propose to use a tokenisation method to split a character sequence into subwords which are assigned random indices before being served to the LM. |
| Outcome: | The proposed model can generalise across duplicated subwords, but this incurs extra cost and is less data efficient. |
Autoregressive Text Generation Beyond Feedback Loops (D19-1)
Copied to clipboard
| Challenge: | Autoregressive feedback exposes the evolution of the hidden state trajectory to potential biases from well-known train-test discrepancies. |
| Approach: | They combine a latent state space model with a CRF observation model to investigate the state evolution of a hidden state trajectory. |
| Outcome: | The proposed model performs better on unconditional sentence generation compared to baselines while avoiding some prototypical failure modes. |
Local and Global Decoding in Text Generation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Text generation relies heavily on decoding algorithms that sample strings from a language model distribution. |
| Approach: | They propose to introduce globally-normalised versions of traditional decoding methods and propose an independent Metropolis-Hastings algorithm to approximate sampling from globally-averaged distributions without explicitly computing them. |
| Outcome: | The proposed method approximates the distributions without explicitly computing them. |
Decoding a Neural Retriever’s Latent Space for Query Suggestion (2022.emnlp-main)
Copied to clipboard
Leonard Adolphs, Michelle Chen Huebscher, Christian Buck, Sertan Girgin, Olivier Bachem, Massimiliano Ciaramita, Thomas Hofmann
| Challenge: | Neural retrieval models have replaced bag-of-words methods for document retrieval . however, they lack the interpretability of bag-off-word models . |
| Approach: | They train a query decoder that generates a meaningful query from a latent representation of a neural search engine. |
| Outcome: | The proposed model outperforms both query reformulation and PRF information retrieval baselines. |
Causal Estimation of Tokenisation Bias (2025.acl-long)
Copied to clipboard
| Challenge: | Modern language models define probabilities over character-strings, but in practice, it does . Ideally, the choice of the tokeniser should not affect the probability assigned to the underlying character- string. |
| Approach: | They quantify a type of tokenisation bias by framing it as a causal effect and estimating it using the regression discontinuity design. |
| Outcome: | The proposed model can estimate tokenisation bias by comparing subwords around arbitrary cutoff points. |