Papers by Louis Martin
Efficient Large Scale Language Modeling with Mixtures of Experts (2022.emnlp-main)
Copied to clipboard
Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, Giridharan Anantharaman, Xian Li, Shuohui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O’Horo, Jeffrey Wang, Luke Zettlemoyer, Mona Diab, Zornitsa Kozareva, Veselin Stoyanov
| Challenge: | Mixture of Experts layers (MoEs) enable efficient scaling of language models . large autoregressive language models such as GPT-3 can be adapted to a wide range of tasks . |
| Approach: | They propose to use Mixture of Experts layers to enable efficient scaling of language models . they find that MoEs are substantially more compute efficient than dense models compared to MoE models - but only when they are more modestly trained . |
| Outcome: | The proposed model outperforms dense models in a wide range of tasks and domains. |
ASSET: A Dataset for Tuning and Evaluation of Sentence Simplification Models with Multiple Rewriting Transformations (2020.acl-main)
Copied to clipboard
| Challenge: | Existing models for sentence simplification are focused on a single transformation, such as lexical paraphrasing or splitting. |
| Approach: | They propose a dataset for assessing sentence simplification in English using a crowdsourced multi-reference corpus. |
| Outcome: | The proposed dataset shows that it captures characteristics of simplicity better than other datasets. |
MUSS: Multilingual Unsupervised Sentence Simplification by Mining Paraphrases (2022.lrec-1)
Copied to clipboard
| Challenge: | MUSS trains strong models using sentence-level paraphrase data instead of labeled simplification data. |
| Approach: | They propose a multilingual unsupervised sentence simplification system that does not require labeled simplification data. |
| Outcome: | The proposed model outperforms the previous best supervised models on English, French, and Spanish benchmarks despite not using labeled simplification data. |
CamemBERT: a Tasty French Language Model (2020.acl-main)
Copied to clipboard
Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, Benoît Sagot
| Challenge: | Pretrained language models are now ubiquitous in Natural Language Processing, but their use in other languages is limited. |
| Approach: | They propose to train monolingual Transformer-based model for other languages using web crawled data instead of Wikipedia data and a relatively small web crawl dataset leads to better results. |
| Outcome: | The proposed model performs as well as those obtained using larger datasets. |
Effective Long-Context Scaling of Foundation Models (2024.naacl-long)
Copied to clipboard
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, Hao Ma
| Challenge: | Large language models (LLMs) are rapidly deployed and continue to evolve through scaling. |
| Approach: | They propose a method to train strong long-context LLMs that are capable of utilizing massive context windows of up to 32,000 tokens. |
| Outcome: | The proposed model can surpass gpt-3.5-turbo-16k's overall performance on long-context benchmarks with a cost-effective instruction tuning procedure that is free of expensive annotations. |
Polar Ducks and Where to Find Them: Enhancing Entity Linking with Duck Typing and Polar Box Embeddings (2023.emnlp-main)
Copied to clipboard
Mattia Atzeni, Mikhail Plekhanov, Frederic Dreyer, Nora Kassner, Simone Merello, Louis Martin, Nicola Cancedda
| Challenge: | Entity linking methods based on dense retrieval are often not efficient in large-scale applications as they are sensitive to the structure of the embedding space. |
| Approach: | They propose a method to infuse structural information into the space of entity representations by using prior knowledge of entity types. |
| Outcome: | The proposed method outperforms other type-aware approaches and matches generative models with 18 times more parameters. |
EASSE: Easier Automatic Sentence Simplification Evaluation (D19-3)
Copied to clipboard
| Challenge: | EASSE provides access to a broad range of evaluation resources including standard automatic metrics, word-level accuracy scores and reference-independent quality estimation features. |
| Approach: | They propose to provide a Python package that provides access to automatic evaluation and comparison of Sentence Simplification (SS) systems. |
| Outcome: | The proposed tool allows comparison and understanding of the performance of Sentence Simplification (SS) systems. |
Controllable Sentence Simplification (2020.lrec-1)
Copied to clipboard
| Challenge: | Text simplification is often considered an all-purpose generic task where the same simplifications are suitable for all but multiple audiences can benefit from simplified text in different ways. |
| Approach: | They propose a controllable simplification model that provides explicit control on simplification systems based on Sequence-to-Sequence models. |
| Outcome: | The proposed model outperforms standard models on simplification benchmarks. |