Pandipedia entry
100
what is the largest dataset in terms of number of tokens today?
Based on the available sources, the largest public dataset by token count appears to be Red Pajama 2—which was announced as having around 30 trillion tokens[5]. Other large datasets include Nemotron‑CC (6.3 trillion tokens)[2], the Common Corpus (over 2 trillion tokens)[4], Dolma (3 trillion tokens)[3], Zyda (1.3 trillion tokens)[1], and MINT‑1T (one trillion tokens)[6]. However, of these, the reported 30 trillion tokens of Red Pajama 2 makes it the largest dataset in terms of number of tokens.
Save this answer
Create your account to keep this answer and continue from it later.
Sorry, Pandi could not find an answer.
Let's look at alternatives:
- Modify the query.
- Start a new thread.
- Remove sources (if manually added).
