| This idea has not failed to pan out at all. I work for a startup that is exactly what GP described, and am set for life because of how wildly successful it is. Notably, we are successful, in a genuine sense of the word: we bootstrapped from running tiny models to larger and larger models on our own slowly improving fleet of GPUs, and now have millions in revenue without a single dime of outside investment. Conversely, you cannot call taking on ~1 trillion in debt and purchase commitments to scale "success". OpenAI and Anthropic are underwater financially. To be precise, they're in the Mariana Trench. |
| |
| ▲ | klipt 2 hours ago | parent | next [-] | | Perhaps an analogy to Moore's law? Bitter lesson #1: don't waste time optimizing code when a faster processor is around the corner. What countered it: Moore's law stopped working. Bitter lesson #2 similarly relies on scaling laws that might have diminishing returns wrt model runtime vs intelligence. Runtime matters for turnaround on the problem you're solving. | | |
| ▲ | wild_egg an hour ago | parent [-] | | Moore's Law has nothing to do with processors getting faster. Dennard scaling stopped working but Moore just slowed somewhat, not stopped. |
| |
| ▲ | applfanboysbgon 2 hours ago | parent | prev | next [-] | | This is a misunderstanding of either the bitter lesson or what was being claimed, on multiple accounts. Firstly, the bitter lesson is merely about human expertise-tuned algorithms vs. throwing raw compute at a domain. But, notably, it is still domain-specific. No matter how much compute you throw at training an LLM, it is never going to beat a Chess engine at Chess. If you give a Chess engine 1,000,000 compute units and a general-purpose LLM 1,000,000 compute units, the Chess engine is obviously superior at Chess; ergo, there is value in throwing compute units into training models for specific tasks. This is true for within several orders of magnitude of compute, in fact. It's also true that if you give the Chess engine 1000 compute units it'll still beat the all-purpose model with 1,000,000 units, so actually there's a lot of value in training for specific tasks. Secondly, the bitter lesson is predicated on compute being cheap. There was a period where a hand-tuned algorithm informed by human expertise would outperform a raw alpha-beta search at Chess. Then compute got cheaper, and DeepBlue ascended to the top. Compute is now expensive again relative to the tasks being performed. We are absolutely still in a period where human expertise in training LLMs will outperform a naive approach with more raw compute. | | |
| ▲ | CamperBob2 2 hours ago | parent [-] | | I don't know much about chess engines; do they still use hand-tuned algorithms, or are they more like AlphaZero, where they learn through self-play to beat any/all possible human contenders? I don't believe DeepBlue was automated to that extent, but it may have been. In the latter case, the chess example would tend to support the Bitter Lesson, rather than refute it. I would also be VERY slow to claim that general-purpose models will never be competitive at chess. It wasn't so long ago that transformers couldn't add two-digit numbers reliably without resorting to tool use. They are now as good at "mental arithmetic" as any human savant. It wouldn't surprise me at all to see someone come up with a model that just happens to be really, really good at leveraging the portions of its general training data having to do with chess. In fact you could argue that AGI demands such a model, if we are to assume that LLMs are a guidepost in that direction. | | |
| ▲ | dmoy 17 minutes ago | parent [-] | | I don't know anything about the last 8 years of chess engines, but yea maybe 8-10 years ago AlphaZero shit all over e.g. stockfish. |
|
| |
| ▲ | HDThoreaun 2 hours ago | parent | prev [-] | | The issue is that GP is misusing the bitter lesson. Yes, search + learn tends to be more effective than human rules based strategies, but that's not what's being considered here. The original claim is effectively that AGI isn't needed for most tasks and more value can be created by using search + learn to solve specific problems instead of applying general models to every problem. Then GP commented a non sequitur |
|
