LAION, The Pile, and more datasets
What's actually used to train these LLMs? A brief look at some of the datasets involved.
Stable Diffusion was trained on a dataset called LAION-5B ("Large-scale Artificial Intelligence Open Network"), which is comprised of 5.85 billion image-text pairs crawled from the internet. The actual crawled data comes from Common Crawl.
3.15 billion pages contained in 380 TB. OpenAI's GPT-3 was, in part, trained by the data in Common Crawl. It is a non-profit founded by Gil Elbaz in 2011 (Elbaz founded Applied Semantics, which was acquired by Google in 2003 for $102mm and later became AdSense).
A set of 22 smaller datasets was used to train GPT-J.
A filtered subset of Common Crawl
"Books3" a collection of ebooks downloaded from Bibliotik
OpenWebText2 – scraped URLs from Reddit with a score of 3 or higher
BookCorpus2 – a set of 18k books from "somewhere"
The book corpuses used aren't specified in the GPT-3 paper. Most likely because they are from gray hat sources like Bibliotik.
Books1 (most likely either Gutenberg)
Books2 (most likely BookCorpus?)