Papers by Lucas Charpentier
More room for language: Investigating the effect of retrieval on language models (2024.naacl-short)
Copied to clipboard
| Challenge: | Retrieval-augmented language models are a promising alternative to standard pretraining, but little attention has been put into understanding what this type of training scheme does to the underlying language model when analyzed as a standalone -separated from the overall retrieval pipeline. |
| Approach: | They propose an ‘ideal retrieval’ methodology to study these models in a fully controllable setting and propose a retrieval augmentation methodology to examine their effects. |
| Outcome: | The proposed model saves substantially less world knowledge in their weights, but is worse at comprehending global context. |
Compositional Generalization with Grounded Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for combining language models with knowledge graphs struggle with generalization to sequences of unseen lengths and novel combinations of seen base components. |
| Approach: | They propose a procedure for generating natural language questions paired with knowledge graphs that targets different aspects of compositionality and avoids grounding models in information already encoded in their weights. |
| Outcome: | The proposed method fails to generalize to unseen lengths and to novel combinations of seen base components. |