Why didn't we get GPT-2 in 2005?

bearsyankees

also the internet wasn't as robust so less training data ofc

fuzzfactor

>There was no grand plan that led to LLMs being created. People just did a bunch of things step-by-step. With so much uncertainty, progress is more “little hops” rather than grand journeys—technology is more “evolved” and less “designed”. This pretty much sums up the whole article, and it makes perfect sense. So much was going to happen anyway that you really do have to look away from the technology to answer why. I would say GPTs didn't come earlier because sama didn't have enough high-dollar contacts yet who would extend their confidence to him. And then what you get is little hops done big because that's whats already familiar to the big money owners. So LLMs are first out of the gate to reach impressive capabilities, but something completely unfamiliar that may be superior due to purposeful design including fundamental principles will have to lag behind until a very big change would become possible.

Panzerschrek

Running GPT-2 after it's trained wasn't as computationally-easy as nowadays. So, you not only need the best supercomputer in the world to train it, but also a datacenter with similar computational power to run enough instances to make it useful.