Remix.run Logo
Remix clone Hacker News
new | show | ask | jobsGithub
▲ godelski a day ago

  > This is also taking his comments about transformers scaling out of context
There have been significant improvements in architectural changes, but since "pre-training" isn't being openly published people just lazily attribute it to "scale". Scale of what? Scale

  > AGI will require architecture changes not simply more data.
As Sutton claimed, all along[0]. People read what they wanted to read. Unfortunately, they also push against researchers who were trying to explore other architectures and methods. I honestly believe that has held us back. There was always room for people to explore different paths. The problem was, and has always been, preventing or discouraging others from exploring new paths. It's ironic given that one of the most common reasons a paper is rejected is for "lack of novelty"[1]. But if someone does something novel then the reviewer simply compares it to the existing state of the art, which is an unfair comparison[2].

It's silly since people want to move fast, but we're really not good at recognizing what has actually allowed us to move forward in the past.

[0] He speaks about Bitter Lesson directly at 9:57 but the earlier part gives better context: https://www.youtube.com/watch?v=21EYKqUsPfg

[1] An essentially meaningless statement. Near meaningless because it is not objective.

[2] You compare a novel technique against a technique that has had hundreds or thousands of researchers optimizing over several years. That's not even remotely fair.

▲tensor 7 hours ago | parent [-]

100%. You can both commercialize transformers while still exploring new paths. Having one of the world's top researchers doing productization work is a huge waste of their time.

You also see this whenever news about novel but small models comes out. People are instantly dismissive. "Oh it's small, who cares?" Of course it's small, why waste resources testing it on a larger scale before making sure it works well on a small scale? Would people rather news not come out until it's SOTA model?