I’m surprised there’s no mention of creativity and outside the box thinking. Listening to this podcast I was wondering if we could train the LLM with knowledge cutoff right before transformers, and ask it to come up with an ML method for LLMs. I’m quite sure none of today’s models would be able to (obviously without access to internet search)