| |
| ▲ | nickysielicki 42 minutes ago | parent | next [-] | | The Bitter Lesson is very popular right now. It seems true right now. It’s having its moment right now. That doesn’t actually mean it’s axiomatically true. Commenter below gets it absolutely correct: stockfish, which runs on your 5 year old phone, is dramatically better at chess than Fable. Like, so much better that it’s not even remotely comparable. The theory of the Bitter Lesson, and it’s only a theory, is that LLMs could eventually outperform stockfish. It’s not true today and it remains to be seen whether it will ever be true. For now, specialized models are absolutely better at specialized tasks. | | |
| ▲ | Animats a minute ago | parent | next [-] | | Good point. Dumb AIs are needed for customer service. Most of that industry is still at "press 1 for sales, 2 for billing..." and needs something that will run locally on a 1U server. | |
| ▲ | Evidlo 13 minutes ago | parent | prev | next [-] | | This seems really backwards. The Bitter Lesson is all about large data-based approaches vs hand-crafted ones, it doesn't say anything about language models not trained specifically for chess. I can't find the comment you're referring to, but the latest versions of stockfish are based on neural networks trained on millions of games, so if anything the Bitter Lesson turned out true here. | |
| ▲ | PEe9bB7D 7 minutes ago | parent | prev [-] | | Maybe depends on how you ask it? Directly, or let it write a chess program? I think the latter can yield way better results. |
| |
| ▲ | ZainRiz 27 minutes ago | parent | prev | next [-] | | I'd respectfully push back on the framing here. If you look at value as purely the LLM output, then there's a valid argument that the best frontier models will always be better than fine tuned specialists. (I'm not convinced personally, but it's a defensible claim) But that misses two dimensions:
1. The cost of acquiring that output
2. What is actually "good enough" for that specialist domain Not every output needs to be the best to produce value. And as specialist models increase in cost, their cost/value proposition goes down. At some point, there's a threshold where cheaper, fine tuned models are "good enough" at the task and also substantially cheaper than the expert models. That's where fine tuning helps. Personally, I became a believer in fine tuning after fine tuning a 1B Qwen model as a second pass over my local voice transcription app, achieving excellent accuracy at ~zero token cost and waaaay lower latency than if I'd invoked my Claude subscription under the hood. | |
| ▲ | srcreigh 29 minutes ago | parent | prev | next [-] | | No. The bitter lesson is about capabilities. GP is talking about efficiency. GP isn’t suggesting that focused narrow model(s) will be more capable than large model, but that many small focused models can have sufficient capability while being more optimal. Also, the bitter lesson is just wrong. The bitter lesson is about hand tuned AI vs computational general methods. However in truth today’s AI uses both. We have general compute heavy models which require narrow expert instructions (eg tools internet docs). LLMs would not be as good without expertly written context, and expert context without LLMs aren’t as good either. | |
| ▲ | applfanboysbgon 2 hours ago | parent | prev | next [-] | | This idea has not failed to pan out at all. I work for a startup that is exactly what GP described, and am set for life because of how wildly successful it is. Notably, we are successful, in a genuine sense of the word: we bootstrapped from running tiny models to larger and larger models on our own slowly improving fleet of GPUs, and now have millions in revenue without a single dime of outside investment. Conversely, you cannot call taking on ~1 trillion in debt and purchase commitments to scale "success". OpenAI and Anthropic are underwater financially. To be precise, they're in the Mariana Trench. | | |
| ▲ | wild_egg 2 hours ago | parent | next [-] | | Wait, you actually found a viable counter to The Bitter Lesson? Please say more | | |
| ▲ | klipt an hour ago | parent | next [-] | | Perhaps an analogy to Moore's law? Bitter lesson #1: don't waste time optimizing code when a faster processor is around the corner. What countered it: Moore's law stopped working. Bitter lesson #2 similarly relies on scaling laws that might have diminishing returns wrt model runtime vs intelligence. Runtime matters for turnaround on the problem you're solving. | | |
| ▲ | wild_egg an hour ago | parent [-] | | Moore's Law has nothing to do with processors getting faster. Dennard scaling stopped working but Moore just slowed somewhat, not stopped. |
| |
| ▲ | applfanboysbgon an hour ago | parent | prev | next [-] | | This is a misunderstanding of either the bitter lesson or what was being claimed, on multiple accounts. Firstly, the bitter lesson is merely about human expertise-tuned algorithms vs. throwing raw compute at a domain. But, notably, it is still domain-specific. No matter how much compute you throw at training an LLM, it is never going to beat a Chess engine at Chess. If you give a Chess engine 1,000,000 compute units and a general-purpose LLM 1,000,000 compute units, the Chess engine is obviously superior at Chess; ergo, there is value in throwing compute units into training models for specific tasks. This is true for within several orders of magnitude of compute, in fact. It's also true that if you give the Chess engine 1000 compute units it'll still beat the all-purpose model with 1,000,000 units, so actually there's a lot of value in training for specific tasks. Secondly, the bitter lesson is predicated on compute being cheap. There was a period where a hand-tuned algorithm informed by human expertise would outperform a raw alpha-beta search at Chess. Then compute got cheaper, and DeepBlue ascended to the top. Compute is now expensive again relative to the tasks being performed. We are absolutely still in a period where human expertise in training LLMs will outperform a naive approach with more raw compute. | | |
| ▲ | CamperBob2 an hour ago | parent [-] | | I don't know much about chess engines; do they still use hand-tuned algorithms, or are they more like AlphaZero, where they learn through self-play to beat any/all possible human contenders? I don't believe DeepBlue was automated to that extent, but it may have been. In the latter case, the chess example would tend to support the Bitter Lesson, rather than refute it. I would also be VERY slow to claim that general-purpose models will never be competitive at chess. It wasn't so long ago that transformers couldn't add two-digit numbers reliably without resorting to tool use. They are now as good at "mental arithmetic" as any human savant. It wouldn't surprise me at all to see someone come up with a model that just happens to be really, really good at leveraging the portions of its general training data having to do with chess. In fact you could argue that AGI demands such a model, if we are to assume that LLMs are a guidepost in that direction. |
| |
| ▲ | HDThoreaun an hour ago | parent | prev [-] | | The issue is that GP is misusing the bitter lesson. Yes, search + learn tends to be more effective than human rules based strategies, but that's not what's being considered here. The original claim is effectively that AGI isn't needed for most tasks and more value can be created by using search + learn to solve specific problems instead of applying general models to every problem. Then GP commented a non sequitur |
| |
| ▲ | z3t4 an hour ago | parent | prev [-] | | Do you have a website? |
| |
| ▲ | CamperBob2 an hour ago | parent | prev [-] | | Suggesting otherwise has become an extraordinary claim requiring extraordinary evidence. VibeThinker 3B constitutes extraordinary evidence, IMO. The first such evidence I've seen myself. Very small model, very low literacy, almost no world knowledge, but it is as good at math and logical reasoning as models a hundred times larger. The Bitter Lesson is a valid and trenchant observation about how about we got here, but I think it's a mistake to assume it tells us very much about where we're going. Too much has changed recently and is still doing so. | | |
| ▲ | spockz 10 minutes ago | parent | next [-] | | So theoretically, if you give that model the means to find information, ascertain the quality of said information, it could still reason its way to an proper answer? Is this whole thing than maybe a read vs write optimisation again? Spent more time and effort training more knowledge into the model upfront and get it out in a single question instead of training a small model and needing more steps to answer the same question? | |
| ▲ | algo_trader 14 minutes ago | parent | prev [-] | | > VibeThinker 3B constitutes extraordinary evidence.. math and logical reasoning Any similar model aimed at coding? A >10B model for mass spawning/swarming and reporting back to a larger model | | |
|
|
