Papers by Marco Baroni
Prediction Hubs are Context-Informed Frequent Tokens in LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Hubness is a tendency for a few points to be among the nearest neighbours of a disproportionate number of other points. |
| Approach: | They show that only large-scale representation comparisons are not characterized by hubness . they show that hubs are the result of context-modulated frequent tokens . |
| Outcome: | The results show that the comparison between context and unembedding vectors does not result in hubness . the findings suggest that hubness is not a negative property that needs to be mitigated when LLMs are being used for next token prediction. |
On the Distribution of Deep Clausal Embeddings: A Large Cross-linguistic Study (P19-1)
Copied to clipboard
| Challenge: | Empirical evidence on the prevalence and limits of embeddings has been based on either laboratory setups or corpus data of relatively limited size. |
| Approach: | They use large, dependency-parsed corpora to capture clausal embedding through dependency graphs and assess their distribution. |
| Outcome: | The results show that there is no evidence for hard constraints on embedding depth . they also show that sentences with many embeddable clauses do not display a bias towards less deep embedded sentences. |
CNNs found to jump around more skillfully than RNNs: Compositional Generalization in Seq2seq Convolutional Networks (P19-1)
Copied to clipboard
| Challenge: | Recent deep neural network successes rekindled debates on their natural language processing abilities. |
| Approach: | They propose to test the ability of sequence-to-sequence networks to perform systematic, compositional generalization of linguistic rules. |
| Outcome: | The proposed dataset shows that convolutional networks perform better on compositional generalization tasks than RNNs. |
Word-order Biases in Deep-agent Emergent Communication (P19-1)
Copied to clipboard
| Challenge: | a recent study examines the "natural" word-order constraints that constrain neural networks . we train models to communicate about paths in a simple gridworld . |
| Approach: | They propose to inoculate a notion of "effort" into neural networks to make their linguistic behavior more human-like. |
| Outcome: | The proposed models show a strong tendency to avoid redundancy and minimize long-distance dependencies. |
Unnatural language processing: How do language models handle machine-generated prompts? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Language model prompt optimization research has shown that semantically and grammatically well-formed manually crafted prompts are outperformed by automatically generated token sequences with no apparent meaning or syntactic structure. |
| Approach: | They propose to use machine-generated prompts to probe how models respond to input that is not composed of natural language expressions. |
| Outcome: | The proposed model outperforms human-crafted prompts on a target zero-shot task. |
Compositionality and Generalization In Emergent Languages (2020.acl-main)
Copied to clipboard
| Challenge: | a new study examines whether emergent languages possess compositionality . compositionality is a core concept in linguistics, but linguists' definitions assume full knowledge of primitive expressions and their combination rules. |
| Approach: | They propose to use compositionality to combine expressions according to systematic rules to refer to composite concepts. |
| Outcome: | The proposed language has compositionality, but it is not generalized, the authors show . they show that the more compositional a language is, the more easily it will be picked up by new learners . |
EGG: a toolkit for research on Emergence of lanGuage in Games (D19-3)
Copied to clipboard
| Challenge: | Existing approaches to simulating language emergence among deep neural agents are challenging due to the discrete nature of communication. |
| Approach: | They propose a toolkit that greatly simplifies the implementation of emergent-language communication games. |
| Outcome: | The proposed toolkit simplifies the implementation of emergent-language communication games. |
Colorless Green Recurrent Networks Dream Hierarchically (N18-1)
Copied to clipboard
| Challenge: | Recurrent neural networks (RNNs) can induce non-trivial properties of language. |
| Approach: | They investigate whether RNNs can track hierarchical syntactic structure . they include nonsensical sentences where RNN cannot rely on semantic cues . |
| Outcome: | The proposed models can predict long-distance agreement in nonsensical sentences in Italian and English. |
MemoryPrompt: A Light Wrapper to Improve Context Tracking in Pre-trained Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Transformer-based language models (LMs) track contextual information through large, hard-coded input windows. |
| Approach: | They propose a leaner approach where a pre-trained LM is augmented with a small auxiliary recurrent network that passes information to the LM by prefixing its regular input with . vectors. |
| Outcome: | The proposed method outperforms larger LMs with full input history on a long-distance dialogue dataset and does not suffer catastrophic forgetting when adapted to new tasks. |
Communication breakdown: On the low mutual intelligibility between human and neural captioning (2022.emnlp-main)
Copied to clipboard
| Challenge: | 0-shot performance of a neural caption-based image retriever is higher when fed captions from a human-produced caption generator . despite the fact that the caption generator does not take the set of distractor images into account, this performance is only marginally above chance level. |
| Approach: | They compare the 0-shot performance of a neural caption-based image retriever with captions from a human-produced captioner. |
| Outcome: | The proposed model performs better when given human-produced captions or neural captions . the best pre-trained model perform better when fed captions produced by an out-of-the-box model . |
Miss Tools and Mr Fruit: Emergent Communication in Agents Learning about Object Affordances (P19-1)
Copied to clipboard
| Challenge: | Recent research studies communication emergence in communities of deep network agents assigned a joint task . authors propose a game meeting many desiderata for a natural communication environment . |
| Approach: | They propose a task capturing aspects of the human environment and human conversation . they propose 'game' meeting many desiderata for a natural communication environment . |
| Outcome: | The proposed task captures aspects of human environment and human conversation, but the agents develop multiple idiolects, resulting in a common language. |
What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties (P18-1)
Copied to clipboard
| Challenge: | a lack of understanding of the properties of sentence embeddings is limiting the use of the techniques. |
| Approach: | They propose 10 probing tasks designed to capture simple linguistic features of sentences . they use three different encoders to train embeddings in eight different ways . |
| Outcome: | The proposed tasks capture key linguistic features of sentences, but they are difficult to infer from them. |
Bridging Information-Theoretic and Geometric Compression in Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Current language models (LMs) encode training data into finitely many variables that allow generalization to infinitely many grammatical utterances. |
| Approach: | They propose to analyze compression in language models from geometric and information-theoretic perspectives. |
| Outcome: | The proposed model can model human language in a relatively small dimension. |
How agents see things: On visual representations in an emergent language game (D18-1)
Copied to clipboard
| Challenge: | Existing studies focus on the agents’ symbol usage, rather than on their representation of visual input. |
| Approach: | They propose to use visual representations of objects to create language-like communication systems by integrating them with the visual input of a game. |
| Outcome: | The proposed model and setup of Lazaridou et al. (2017) show that the representations of the agents' symbols do not capture the conceptual properties of the objects depicted in the input images. |
The emergence of number and syntax units in LSTM language models (N19-1)
Copied to clipboard
| Challenge: | a recent study shows that LSTMs can capture syntax-sensitive generalizations such as long-distance number agreement. |
| Approach: | They investigate the inner mechanics of number tracking in LSTMs at the single neuron level . they find that long-distance number information is largely managed by two "number units" importantly, the behaviour of these units is partially controlled by other units to track syntactic structure . |
| Outcome: | The proposed model is based on a language model with a long-distance number agreement task. |
Emergent Language-Based Coordination In Deep Multi-Agent Systems (2022.emnlp-tutorials)
Copied to clipboard
| Challenge: | Pre-trained deep networks are the standard building blocks of modern AI applications. |
| Approach: | This tutorial will introduce deep net emergent communication and discuss current shortcomings . participants will implement and analyze two emergentic communication setups from the literature . |
| Outcome: | The presentation will cover various topics from the present and recent past, as well as discussing current shortcomings and suggest future directions. |