Remix.run Logo
HarHarVeryFunny 2 hours ago

When OpenAI was founded, the mission was to develop AI, but nobody (anywhere) knew how to do AI, so OpenAI did ML research on games instead, which is what DeepMind was doing (with Google's perceived AI/ML dominance being the raison d'etre for OpenAI, and Google having just bought DeepMind). This was the era when Karpathy was at OpenAI.

Around the time Karpathy left, Ilya Sutskever, another OpenAI founder, started playing with Google's new "Transformer" architecture, which was the beginning of the "GPT" series, GPT-1, GPT-2 and eventually ChatGPT (GPT 3.5 + RLHF). In retrospect OpenAI's early Transformer experiments and GPT-1 was the inflection point that moved OpenAI from a company that wanted to build AI, as soon as anyone else did, to one that was actually doing so, although I think it would be revisionist for anyone to claim they knew what they were doing at the time. The early GPT-1 and GPT-2 papers read more like "wow, this is a bit unexpected, look at all of the things it can do!".