Papers by Mohammadreza Armandpour
TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretraining (2025.acl-long)
Copied to clipboard
Jeffrey Li, Mohammadreza Armandpour, Seyed Iman Mirzadeh, Sachin Mehta, Vaishaal Shankar, Raviteja Vemulapalli, Samy Bengio, Oncel Tuzel, Mehrdad Farajtabar, Hadi Pouransari, Fartash Faghri
| Challenge: | Large language models (LLMs) trained on historical web data inevitably become outdated. |
| Approach: | They propose a web-scale dataset for time-continual pretraining of LLMs derived from 114 dumps of Common Crawl (CC) they also design time-stratified evaluations to assess how well various continual learning methods adapt to new data while retaining past knowledge. |
| Outcome: | The proposed benchmarks show that autoregressive meta-schedules combined with a fixed-ratio replay of older data can achieve comparable held-out loss to re-training from scratch, while requiring significantly less computation (2.6x). |