Papers by Mostafa Elhoushi
Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls (2025.emnlp-main)
Copied to clipboard
Feiyang Kang, Newsha Ardalani, Michael Kuchnik, Youssef Emad, Mostafa Elhoushi, Shubhabrata Sengupta, Shang-Wen Li, Ramya Raghavendra, Ruoxi Jia, Carole-Jean Wu
| Challenge: | a large-scale empirical study compares natural web data, diverse synthetic types, and mixtures of natural and synthetic data. |
| Approach: | They conduct a large-scale empirical study on large-volume LLMs using a unified protocol and scaling laws. |
| Outcome: | The proposed method is faster than pre-training on natural web data, the authors show . their results are consistent with previous studies on rephrased text and textbooks . |
LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding (2024.acl-long)
Copied to clipboard
Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, Ahmed Aly, Beidi Chen, Carole-Jean Wu
| Challenge: | Large Language Models (LLMs) have been deployed to many applications, yet their high compute and memory requirements lead to high financial and energy costs when deployed to GPU servers. |
| Approach: | They propose an end-to-end solution to speed-up inference of large language models . they apply layer dropout, and show that it increases the accuracy of early exit at earlier layers without adding any auxiliary layers or modules to the model. |
| Outcome: | The proposed method shows speedups of up to 2.16x on summarization for CNN/DM documents, 1.82x on coding, and 2.0x on TOPv2 semantic parsing task. |
Calibrating Beyond English: Language Diversity for Better Quantized Multilingual LLMs (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing quantization methods typically use small, English-only calibration sets . however, their impact on multilingual models remains underexplored . |
| Approach: | They evaluate eight calibration settings across two quantizers on data from 10 different languages. |
| Outcome: | The results show that tailoring calibration sets to the evaluation language yields the largest improvements for individual languages, underscoring the importance of linguistic alignment. |