In a recent interview, George Hotz claimed that GPT-4 is just an eight-way mixture model of 220B parameters. It could be a Mixture of Experts (MoE) model. That estimates GPT-4 at about 1.2 trillion parameters (8 x 220 billion).
thanks for the callout! would appreciate [signal boost](https://twitter.com/swyx/status/1671183813190504448) on the source podcast :)
Updated!
thanks for the callout! would appreciate [signal boost](https://twitter.com/swyx/status/1671183813190504448) on the source podcast :)
Updated!