Papers by Tiago Pimentel

Modeling the Unigram Distribution (2021.findings-acl)

Challenge:	Existing studies have found that higher quality language models provide more powerful predictors of human reading behavior, but empirical support for the QP hypothesis is mixed.
Approach:	They propose to test the quality–power hypothesis by using surprisal language models to test their ability to predict eye tracking data.
Outcome:	The proposed model is based on a set of language models with a 'quality-power' hypothesis.

Copied to clipboard

Irene Nikkarinen, Tiago Pimentel, Damián Blasi, Ryan Cotterell

Challenge:	a novel model for estimating the unigram distribution is proposed for a language . the model is based on the word type and word type distributions, but does not consider contextual information.
Approach:	They propose a neuralization of Goldwater's (2011) model for estimating the unigram distribution in a language.
Outcome:	The proposed model outperforms character-level models in estimating the unigram distribution in a language.

On the probability–quality paradox in language generation (2022.acl-short)

Copied to clipboard

Clara Meister, Gian Wiher, Tiago Pimentel, Ryan Cotterell

Text Generation

Challenge:	a recent study suggests that mode-seeking decoding methods produce unnatural language . a higher probability text is perceived as more human-like, but this is not the case with stochastic decoding .
Approach:	They propose to explain why higher-probability text is perceived as more human-like . they argue that human-type language should contain an amount of information close to entropy .
Outcome:	The proposed model produces lower-probability text, which is perceived as more human-like . elucidation of this hypothesis supports the hypothesis .

On the Effect of (Near) Duplicate Subwords in Language Modelling (2024.findings-acl)

Copied to clipboard

Anton Schäfer, Thomas Hofmann, Imanol Schlag, Tiago Pimentel

A Tale of a Probe and a Parser (2020.acl-main)

Challenge:	Tokenisation is a core part of language models but can lead to less efficient training because it removes character-level information.
Approach:	They propose to use a tokenisation method to split a character sequence into subwords which are assigned random indices before being served to the LM.
Outcome:	The proposed model can generalise across duplicated subwords, but this incurs extra cost and is less data efficient.

Copied to clipboard

Rowan Hall Maudslay, Josef Valvoda, Tiago Pimentel, Adina Williams, Ryan Cotterell

Investigating Critical Period Effects in Language Acquisition through Neural Language Models (2025.tacl-1)

Challenge:	researchers train supervised models to extract linguistic structure from output of another model . supervised model can be used to perform tasks such as part-of-speech tags or dependency trees .
Approach:	They compare a structural probe to a more traditional parser with a lightweight parameterisation.
Outcome:	The structural probe outperforms a traditional parser on seven of nine languages . the researchers found that the model outperformed the parsers by 11.1 points .

Copied to clipboard

Ionut Constantinescu, Tiago Pimentel, Ryan Cotterell, Alex Warstadt

Probing for the Usage of Grammatical Number (2022.acl-long)

Challenge:	Scholars of human development have long debated whether these phenomena are predetermined by innately encoded developmental changes in the maturing brain or natural consequences of increased experience.
Approach:	They use language models to test whether CP effects are peculiar to humans . they find that LMs do not show CP when L2 exposure is delayed . scholars have long debated whether innate maturation changes predetermine CP .
Outcome:	The proposed model does not show CP effects when the age of exposure of L2 is delayed.

Copied to clipboard

Karim Lasri, Tiago Pimentel, Alessandro Lenci, Thierry Poibeau, Ryan Cotterell

Convergence and Divergence of Language Models under Different Random Seeds (2025.emnlp-main)

Challenge:	Pre-trained language models can be used to perform a wide array of NLP tasks, but their encoding is still a mystery.
Approach:	They propose a usage-based probing setup to find an encoding that the model actually uses, and propose 'a use-based approach' they propose to use a behavioral task to remove the linguistic property, and to identify which encodes are used to transfer information from a noun to its head verb.
Outcome:	The proposed encodings are based on a behavioral task which cannot be solved without the linguistic property.

Copied to clipboard

Finlay Fehlauer, Kyle Mahowald, Tiago Pimentel

Finding Concept-specific Biases in Form–Meaning Associations (2021.naacl-main)

Challenge:	a large body of work has examined the training dynamics of language models.
Approach:	They investigate the convergence of language models (LMs) trained under different random seeds . they find that larger models reconverge faster in later training stages, while smaller models never actually reconverge.
Outcome:	The proposed model size and training checkpoints influence convergence of language models under different seeds.

Copied to clipboard

Tiago Pimentel, Brian Roark, Søren Wichmann, Ryan Cotterell, Damián Blasi

Multilinguality

Challenge:	Existing methods to detect cross-linguistic associations are not effective, but their effects are minor.
Approach:	They propose a method to measure cross-linguistic associations by controlling for the influence of language family and geographic proximity within a large concept-aligned, cross-lingual lexicon.
Outcome:	The proposed method shows that it is small, but it is unsurprisingly small (less than 0.5% on average).

Revisiting the Optimality of Word Lengths (2023.emnlp-main)

Copied to clipboard

Tiago Pimentel, Clara Meister, Ethan Wilcox, Kyle Mahowald, Ryan Cotterell

Challenge:	Existing theories of communication cost are based on the length of an utterance, but they are not based upon the frequency of the utterant.
Approach:	They propose a method to minimize CCH's cost by comparing language lengths to surprisal's expectation and variance-to-mean ratios.
Outcome:	The proposed method does not minimize CCH’s cost, but rather a lower bound, which we term CCH-lower.

Phonotactic Complexity and Its Trade-offs (2020.tacl-1)

Copied to clipboard

Tiago Pimentel, Brian Roark, Ryan Cotterell

How (Non-)Optimal is the Lexicon? (2021.naacl-main)

Challenge:	Existing measures of linguistic complexity are relatively coarse-see, for example, Moran and Blasi (2014) and 2 below for reviews.
Approach:	They propose to measure bits per phoneme using the negative log-probability of a word in a language model and a collection of 1016 basic concept words across 106 languages.
Outcome:	The proposed measure allows a cross-linguistic comparison of phonotactic complexity across languages.

Copied to clipboard

Tiago Pimentel, Irene Nikkarinen, Kyle Mahowald, Ryan Cotterell, Damián Blasi

Challenge:	lexical meanings are mapped to wordforms by usage pressures and constraints on sequences of symbols.
Approach:	They propose a coding-theoretic view of the lexicon and a novel generative statistical model to quantify its compressibility under various constraints.
Outcome:	The proposed model shows that (compositional) morphology and graphotactics can account for most of the complexity of natural codes—as measured by code length.

The time scale of redundancy between prosody and linguistic context (2025.acl-long)

Copied to clipboard

Tamar I Regev, Chiebuka Ohams, Shaylee Xie, Lukas Wolf, Evelina Fedorenko, Alex Warstadt, Ethan Wilcox, Tiago Pimentel

Challenge:	Prior work has shown that the information carried by prosodic features is substantially redundant with that carried by the surrounding words.
Approach:	They examine the time scale of this relationship, studying how it varies with the length of past and future contexts.
Outcome:	The results show that prosody features show some redundancy with future words, but only with a short scale of 1-2 words, consistent with reports of incremental short-term planning in language production.

Locally Typical Sampling (2023.tacl-1)

Copied to clipboard

Clara Meister, Tiago Pimentel, Gian Wiher, Ryan Cotterell

Responsible & Trustworthy NLP

Challenge:	a discrepancy in probabilistic language generators has puzzled the language generation community for years .
Approach:	They propose a method for local typical sampling to enforce a conditional entropy criterion for probabilistic models.
Outcome:	The proposed method can provide new insights into why high-probability texts can be dull or repetitive.

Revisiting the Uniform Information Density Hypothesis (2021.emnlp-main)

Copied to clipboard

Clara Meister, Tiago Pimentel, Patrick Haller, Lena Jäger, Ryan Cotterell, Roger Levy

Challenge:	The uniform information density hypothesis posits a preference among language users for utterances structured such that information is distributed uniformly across a signal.
Approach:	They propose to test the hypothesis by using reading time and acceptability data to examine the effect of surprisal on language comprehension and acceptabilities.
Outcome:	The proposed hypothesis makes predictions about language comprehension and linguistic acceptability .

How to Compute the Probability of a Word (2024.emnlp-main)

Copied to clipboard

Tiago Pimentel, Clara Meister

Local and Global Decoding in Text Generation (2024.findings-emnlp)

Challenge:	Language models estimate a probability distribution over strings in a natural language . many recent linguistic studies have been incorrectly computing word probabilities .
Approach:	They propose to use the correct method to compute word probabilities . they highlight issues when relying on models that use end-of-word tokenisers .
Outcome:	Empirically, correcting the widespread bug affects measured outcomes in sentences and lexical optimisation analyses.

Copied to clipboard

Daniel Gareev, Thomas Hofmann, Ezhilmathi Krishnasamy, Tiago Pimentel

Pareto Probing: Trading Off Accuracy for Complexity (2020.emnlp-main)

Challenge:	Text generation relies heavily on decoding algorithms that sample strings from a language model distribution.
Approach:	They propose to introduce globally-normalised versions of traditional decoding methods and propose an independent Metropolis-Hastings algorithm to approximate sampling from globally-averaged distributions without explicitly computing them.
Outcome:	The proposed method approximates the distributions without explicitly computing them.

Copied to clipboard

Tiago Pimentel, Naomi Saphra, Adina Williams, Ryan Cotterell

Representation Learning

Challenge:	Neural networks are a pillar of modern NLP systems, but their inner workings are poorly understood.
Approach:	They propose a probe metric that reflects the trade-off between probe complexity and performance: the Pareto hypervolume.
Outcome:	The proposed probe metric conforms to accepted rankings among contextual representations, and is more complex than other probe tasks.

On the Intersection of Context-Free and Regular Languages (2023.eacl-main)

Copied to clipboard

Clemente Pasti, Andreas Opedal, Tiago Pimentel, Tim Vieira, Jason Eisner, Ryan Cotterell

Linguistic Theories

Challenge:	We generalize the Bar-Hillel intersection construction so that the given WFSA may contain -arcs.
Approach:	They propose a construction that generalizes the Bar- Hillel in the case the desired automaton has -arcs and generalize the weighted extension so that the given WFSA may contain arcs.
Outcome:	The proposed construction can encode the structure of both the input automaton and grammar while retaining the asymptotic size of the original construction.

Meaning to Form: Measuring Systematicity as Information (P19-1)

Copied to clipboard

Tiago Pimentel, Arya D. McCarthy, Damian Blasi, Brian Roark, Ryan Cotterell

Challenge:	A longstanding debate in semiotics centers on the relationship between linguistic signs and their corresponding semantics: is there an arbitrary relationship between word forms and their meaning, or does some systematic phenomenon pervade?
Approach:	They propose to quantify the systematicity of the sign using mutual information and recurrent neural networks to examine 106 languages.
Outcome:	The proposed model reduces entropy in a word form conditioned on its semantic representation and recovers English examples of systematic affixes.

Do Generalisation Results Generalise? (2026.findings-acl)

Copied to clipboard

Matteo Boglioni, Andrea Sgobbi, Gabriel Tavernini, Francesco Rita, Marius Mosbach, Tiago Pimentel

Challenge:	Existing studies evaluating generalisation performance on large language models focuses on a single out-of-distribution dataset .
Approach:	They examine whether OOD generalisation results generalise across multiple OOD testsets throughout a finetuning run and then evaluate the partial correlation of results .
Outcome:	The proposed model achieves high scores on multiple OOD testsets, regressing out in-domain performance.

Disambiguatory Signals are Stronger in Word-initial Positions (2021.eacl-main)

Copied to clipboard

Tiago Pimentel, Ryan Cotterell, Brian Roark

Challenge:	lexical studies of word processing and lexicon access provide evidence of preferred nature of word-initial versus word-final segments . conjecture that languages have evolved to provide more information earlier in words is based on existing methods .
Approach:	They propose to use a new method to assess the informativeness of word-initial versus word-final segments.
Outcome:	The proposed measures avoid the confounds found in existing methods.

Naturalistic Causal Probing for Morpho-Syntax (2023.tacl-1)

Copied to clipboard

Afra Amini, Tiago Pimentel, Clara Meister, Ryan Cotterell

A Cross-Linguistic Pressure for Uniform Information Density in Word Order (2023.tacl-1)

Challenge:	Existing methods for probing are limited and lack understanding of their limitations and weaknesses.
Approach:	They propose a strategy for input-level intervention on naturalistic sentences . they use morpho-syntactic features of a sentence to intervene on the rest of the sentence .
Outcome:	The proposed approach allows for input-level intervention on naturalistic sentences while keeping the rest of the sentence unchanged.

Copied to clipboard

Thomas Hikaru Clark, Clara Meister, Tiago Pimentel, Michael Hahn, Ryan Cotterell, Richard Futrell, Roger Levy

Challenge:	a recent study has compared real and counterfactual word orders, but one functional pressure has been overlooked . a study of 10 typologically diverse languages shows that real word orders have greater uniformity than reverse word orders .
Approach:	They propose to test whether a pressure for UID may have influenced word order patterns cross-linguistically.
Outcome:	The proposed model shows that real orders have greater uniformity than reverse orders among SVO languages.

A Corpus for Large-Scale Phonetic Typology (2020.acl-main)

Copied to clipboard

Elizabeth Salesky, Eleanor Chodroff, Tiago Pimentel, Matthew Wiesner, Ryan Cotterell, Alan W Black, Jason Eisner

Using Information Theory to Characterize Prosodic Typology: The Case of Tone, Pitch-Accent and Stress-Accent (2025.acl-long)

Challenge:	Existing multilingual speech corpora have limited data in many languages . existing corpus is limited to a small number of languages with available data .
Approach:	They propose a large-scale phonetic typology corpus with phoneme-level labels and phoneme alignments in 690 readings spanning 635 languages.
Outcome:	The proposed corpus covers 635 languages and includes acoustic-phonetic measures of vowels and sibilants.

Copied to clipboard

Ethan Wilcox, Cui Ding, Giovanni Acampa, Tiago Pimentel, Alex Warstadt, Tamar I Regev

Quantifying the redundancy between prosody and text (2023.emnlp-main)

Challenge:	lexical identity and prosody are well-studied parameters of linguistic variation, but they are difficult to predict in tonal languages.
Approach:	They propose to characterize the relationship between lexical identity and prosody using information theory to estimate mutual information between the text and pitch curves.
Outcome:	The proposed hypothesis supports perspectives that view linguistic typology as gradient, rather than categorical.

Copied to clipboard

Lukas Wolf, Tiago Pimentel, Evelina Fedorenko, Ryan Cotterell, Alex Warstadt, Ethan Wilcox, Tamar Regev

Predicting Declension Class from Form and Meaning (2020.acl-main)

Challenge:	Existing studies suggest partial redundancy between prosody and linguistic information.
Approach:	They use large language models to estimate how much information is redundant between prosody and the words themselves.
Outcome:	The proposed model can predict prosodic features across prosodic features, including intensity, duration, pauses, and pitch contours.

Copied to clipboard

Adina Williams, Tiago Pimentel, Hagen Blix, Arya D. McCarthy, Eleanor Chodroff, Ryan Cotterell

Challenge:	phonological form and meaning of nouns can provide imperfect clues, but they can also be indicative of grammatical gender.
Approach:	They propose a method to measure how much information can be gleamed from knowing the form and/or meaning of nouns.
Outcome:	The proposed method provides additional quantitative support for a classic linguistic finding that form and meaning are relevant for the classification of nouns into declensions.

A surprisal–duration trade-off across and within the world’s languages (2021.emnlp-main)

Copied to clipboard

Tiago Pimentel, Clara Meister, Elizabeth Salesky, Simone Teufel, Damián Blasi, Ryan Cotterell

Challenge:	Throughout human evolution, countless languages have evolved, each with unique features.
Approach:	They analysed a corpus of 600 languages to find strong evidence for a surprisal–duration trade-off between languages and languages.
Outcome:	The proposed model shows that phones are produced faster in languages where they are less surprising and vice versa.

Tokenisation is NP-Complete (2025.acl-long)

Copied to clipboard

Philip Whittington, Gregor Bachmann, Tiago Pimentel

What Do Prosody and Text Convey? Characterizing How Meaningful Information is Distributed Across Multiple Channels (2026.acl-long)

Challenge:	Tokenisation is the first step required to use a language model.
Approach:	They propose to find tokenisers which maximise the compression of a text by either finding a vocabulary directly or selecting a sequence of merge operations.
Outcome:	The proposed tokenisation methods are able to find tokenisers that maximise compression of a text.

Copied to clipboard

Aditya Yadavalli, Tiago Pimentel, Tamar I Regev, Ethan Gotlieb Wilcox, Alex Warstadt

Multimodality

Challenge:	Prosody—the melody of speech—conveys critical information often not captured by the words or text of a message.
Approach:	They propose an information-theoretic approach to quantify how much is conveyed by prosody that is not recoverable from text alone.
Outcome:	The proposed framework can quantify how much is conveyed by prosody that is not recoverable from text alone and crucially, what prosody conveys.

A Bayesian Framework for Information-Theoretic Probing (2021.emnlp-main)

Copied to clipboard

Tiago Pimentel, Ryan Cotterell

Challenge:	a recent paper suggests that probing should be seen as approximating a mutual information.
Approach:	They propose a Bayesian mutual information framework that probes probing representations from the perspective of Bayes' agents.
Outcome:	The proposed framework allows for more intuitive results in scenarios with finite data.

Causal Estimation of Tokenisation Bias (2025.acl-long)

Copied to clipboard

Pietro Lesci, Clara Meister, Thomas Hofmann, Andreas Vlachos, Tiago Pimentel

Clara Meister, Mario Giulianelli, Tiago Pimentel

Challenge:	Modern language models define probabilities over character-strings, but in practice, it does . Ideally, the choice of the tokeniser should not affect the probability assigned to the underlying character- string.
Approach:	They quantify a type of tokenisation bias by framing it as a causal effect and estimating it using the regression discontinuity design.
Outcome:	The proposed model can estimate tokenisation bias by comparing subwords around arbitrary cutoff points.

Copied to clipboard

Linguistics & Cognitive NLP

Challenge:	Existing studies have shown that surprisal theory ignores the possibility of similarity between words and treats them as distinct entities.
Approach:	They propose a new measure of comprehension effort called information value that accounts for communicative equivalences between possible continuations.
Outcome:	The proposed measure of comprehension effort is based on the diversity index of the diversity of communicative units.

Information-Theoretic Probing for Linguistic Structure (2020.acl-main)

Copied to clipboard

Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, Ryan Cotterell

Explainability & Interpretability in NLP

Challenge:	Neural networks are the backbone of modern stateof-the-art natural language processing systems.
Approach:	They propose an information-theoretic operationalization of probing as estimating mutual information that contradicts received wisdom . they evaluate on a set of ten typologically diverse languages often underrepresented in NLP research—plus English—totalling eleven languages.
Outcome:	The proposed model outperforms existing models on ten typologically diverse languages and English on 11 languages.

The Architectural Bottleneck Principle (2022.emnlp-main)

Copied to clipboard

Tiago Pimentel, Josef Valvoda, Niklas Stoehr, Ryan Cotterell

UniMorph 4.0: Universal Morphology (2022.lrec-1)

Challenge:	a recent study examined how much information a model's representations contain . a new approach to probing is to look exactly like the component .
Approach:	They propose a new probing principle that aims to estimate how much information a model could extract from its representations.
Outcome:	The proposed probes extract syntactic information from the representations of a neural network . the proposed probe is based on the architectural bottleneck principle .

Copied to clipboard

Khuyagbaatar Batsuren, Omer Goldman, Salam Khalifa, Nizar Habash, Witold Kieraś, Gábor Bella, Brian Leonard, Garrett Nicolai, Kyle Gorman, Yustinus Ghanggo Ate, Maria Ryskina, Sabrina Mielke, Elena Budianskaya, Charbel El-Khaissi, Tiago Pimentel, Michael Gasser, William Abbott Lane, Mohit Raj, Matt Coler, Jaime Rafael Montoya Samame, Delio Siticonatzi Camaiteri, Esaú Zumaeta Rojas, Didier López Francis, Arturo Oncevay, Juan López Bautista, Gema Celeste Silva Villegas, Lucas Torroba Hennigen, Adam Ek, David Guriel, Peter Dirix, Jean-Philippe Bernardy, Andrey Scherbakov, Aziyana Bayyr-ool, Antonios Anastasopoulos, Roberto Zariquiey, Karina Sheifer, Sofya Ganieva, Hilaria Cruz, Ritván Karahóǧa, Stella Markantonatou, George Pavlidis, Matvey Plugaryov, Elena Klyachko, Ali Salehi, Candy Angulo, Jatayu Baxi, Andrew Krizhanovsky, Natalia Krizhanovskaya, Elizabeth Salesky, Clara Vania, Sardana Ivanova, Jennifer White, Rowan Hall Maudslay, Josef Valvoda, Ran Zmigrod, Paula Czarnowska, Irene Nikkarinen, Aelita Salchak, Brijesh Bhatt, Christopher Straughn, Zoey Liu, Jonathan North Washington, Yuval Pinter, Duygu Ataman, Marcin Wolinski, Totok Suhardijanto, Anna Yablonskaya, Niklas Stoehr, Hossep Dolatian, Zahroh Nuriah, Shyam Ratan, Francis M. Tyers, Edoardo M. Ponti, Grant Aiton, Aryaman Arora, Richard J. Hatcher, Ritesh Kumar, Jeremiah Young, Daria Rodionova, Anastasia Yemelina, Taras Andrushko, Igor Marchenko, Polina Mashkovtseva, Alexandra Serova, Emily Prud’hommeaux, Maria Nepomniashchaya, Fausto Giunchiglia, Eleanor Chodroff, Mans Hulden, Miikka Silfverberg, Arya D. McCarthy, David Yarowsky, Ryan Cotterell, Reut Tsarfaty, Ekaterina Vylomova

An Exploration of Left-Corner Transformations (2023.emnlp-main)

Challenge:	The Universal Morphology project provides broad-coverage instantiated morphological inflection tables for hundreds of diverse languages.
Approach:	They propose a language-independent feature schema for rich morphological annotation and a type-level resource of annotated data in diverse languages realizing that schema.
Outcome:	The proposed schema has added 66 new languages, including 24 endangered languages.

Copied to clipboard

Andreas Opedal, Eleftheria Tsipidi, Tiago Pimentel, Ryan Cotterell, Tim Vieira

What About the Precedent: An Information-Theoretic Analysis of Common Law (2021.naacl-main)

Challenge:	Fig. 1 shows how left-corner transformations remove left recursion from context-free grammars . a common use case is to remove left-recursed grammars, which is necessary for top-down parsing .
Approach:	They generalize left-corner transformations to support semiring-weighted production rules . they also provide finer-grained control over which left corners may be moved .
Outcome:	The proposed transformation removes left recursion from context-free grammars . it also provides finer-grained control over which left corners may be moved .

Copied to clipboard

Josef Valvoda, Tiago Pimentel, Niklas Stoehr, Ryan Cotterell, Simone Teufel

Challenge:	In common law, the outcome of a new case is determined mostly by precedent cases, rather than by existing statutes.
Approach:	They propose to model the argumentation of precedent cases and compare them to a case out-come classification task to determine how the precedent influences the outcome of a new case.
Outcome:	The proposed method compared arguments of two longstanding jurisprudential views on the European Court of Human Rights (ECtHR) and the precedent cases.

Generating Text from Language Models (2023.acl-tutorials)

Copied to clipboard

Afra Amini, Ryan Cotterell, John Hewitt, Luca Malagutti, Clara Meister, Tiago Pimentel

On Homophony and Rényi Entropy (2021.emnlp-main)

Challenge:	a growing percentage of natural language processing tasks focus on the generation of text from probabilistic language models.
Approach:	They will provide a centralized discussion of critical considerations when choosing how to generate from a language model.
Outcome:	This tutorial will provide a centralized discussion of critical considerations when choosing how to generate from a language model.

Copied to clipboard

Tiago Pimentel, Clara Meister, Simone Teufel, Ryan Cotterell

Linguistic Theories

Challenge:	Recent theories of language optimality have tried to justify its prevalence, arguing that homophony enables the reuse of efficient wordforms and is thus beneficial for languages.
Approach:	They propose a new information-theoretic quantification of a language’s homophony: the sample Rényi entropy.
Outcome:	The proposed method is more nuanced than either Piantadosi et al.'s or Trott and Bergen's results.

Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation (2023.findings-acl)

Copied to clipboard

Marius Mosbach, Tiago Pimentel, Shauli Ravfogel, Dietrich Klakow, Yanai Elazar

Responsible & Trustworthy NLP

Challenge:	Recent studies show that in-context learning and few-shot fine-tuning can generalize well out-of-domain.
Approach:	They compare few-shot fine-tuning and in-context learning for task adaptation . they find that both approaches generalize similarly, but exhibit large variation .
Outcome:	The proposed methods outperform in-context learning and few-shot fine-tuning with OPT models of different sizes.

A Natural Bias for Language Generation Models (2023.acl-short)

Copied to clipboard

Clara Meister, Wojciech Stokowiec, Tiago Pimentel, Lei Yu, Laura Rimell, Adhiguna Kuncoro

Text Generation

Challenge:	a standard probabilistic model for language generation has likely not yet learnt many semantic or syntactic rules of natural language, making it difficult to estimate the probability distribution over next tokens.
Approach:	They propose to initialise bias terms in a model's final linear layer with the log-unigram distribution and use it to output the unigram frequency statistics as prior knowledge.
Outcome:	The proposed method improves learning efficiency and improves overall performance.

A Non-Linear Structural Probe (2021.naacl-main)

Copied to clipboard

Jennifer C. White, Tiago Pimentel, Naomi Saphra, Ryan Cotterell

On the Efficacy of Sampling Adapters (2023.acl-long)

Challenge:	Probing is a method of investigating the encoding of knowledge in contextual representations.
Approach:	They propose to kernelize a metric and develop a non-linear variant with an identical number of parameters by using a kernel-based probe.
Outcome:	The proposed probe learns only linear transformations and achieves statistically significant performance improvement over baseline in all languages.

Copied to clipboard

Clara Meister, Tiago Pimentel, Luca Malagutti, Ethan Wilcox, Ryan Cotterell

Green & Sustainable NLP

Challenge:	Using sampling adapters can improve the quality of the generated text.
Approach:	They propose a framework for understanding sampling adapters and propose 'sampling adapters' they argue that the shift enforced by them can be viewed as a trade-off between precision and recall .
Outcome:	The proposed framework can be used to improve the quality of language models by modifying their distributions to improve their precision and recall.

Speakers Fill Lexical Semantic Gaps with Context (2020.emnlp-main)

Copied to clipboard

Tiago Pimentel, Rowan Hall Maudslay, Damian Blasi, Ryan Cotterell