Based on the available sources, the largest public dataset by token count appears to be Red Pajama 2—which was announced as having around 30 trillion tokens [17]. Other large datasets include Nemotron‑CC (6.3 trillion tokens) [4], the Common Corpus (over 2 trillion tokens) [9], Dolma (3 trillion tokens) [5], Zyda (1.3 trillion tokens) [3], and MINT‑1T (one trillion tokens) [34]. However, of these, the reported 30 trillion tokens of Red Pajama 2 makes it the largest dataset in terms of number of tokens.

Zyphra Introduces Zyda Dataset: A 1.3 Trillion Token Dataset for Open Language Modeling – Every Intel

what is the largest dataset in terms of number of tokens today?

Follow Up Recommendations

Related Content You May Like