Papers by Brandon Norick
Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset (2025.acl-long)
Copied to clipboard
Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro
| Challenge: | Recent Common Crawl datasets remove 90% of data, limiting their suitability for long token horizon training. |
| Approach: | They propose to combine classifier ensembling, synthetic data rephrasing and heuristic filters to achieve better trade-offs between accuracy and data quantity. |
| Outcome: | The proposed model-based filtering improves MMLU by 5.6 over DCLM for 15T tokens . the full 6.3T token dataset matches DCLM on MMLO, but contains four times more unique real tokens than DCLM . |