| ▲ | gmueckl 4 hours ago |
| The result is hardly a clean room implementation. It was rather a brute force attempt to decompress fuzzily stored knowledge contained within the network and it required close steering (using a big suite of tests) to get a reasonable approximation to the desired output. The compression and storage happened during the LLM training. Prove this statement wrong. |
|
| ▲ | libraryofbabel 3 hours ago | parent | next [-] |
| Nobody disputes that the LLM was drawing on knowledge in its training data. Obviously it was! But you'll need to be a bit more specific with your critique, because there is a whole spectrum of interpretations, from "it just decompressed fuzzily-stored code verbatim from the internet" (obviously wrong, since the Rust-based C compiler it wrote doesn't exist on the internet) all the way to "it used general knowledge from its training about compiler architecture and x86 and the C language." Your post is phrased like it's a two sentence slam-dunk refutation of Anthropic's claims. I don't think it is, and I'm not even clear on what you're claiming precisely except that LLMs use knowledge acquired during training, which we all agree on here. |
| |
| ▲ | nicoburns an hour ago | parent [-] | | "clean room" usually means "without looking at the source code" of other similar projects. But presumably the AIs training data would have included GCC, Clang, and probably a dozen other C compilers. | | |
| ▲ | signatoremo 18 minutes ago | parent [-] | | Suppose you the human are working on a clean room implementation of C compiler, how do you go about doing it? Will you need to know about: a) the C language, and b) the inner working of a compiler? How did you acquire that knowledge? |
|
|
|
| ▲ | NitpickLawyer 4 hours ago | parent | prev | next [-] |
| > Prove this statement wrong. If all it takes is "trained on the Internet" and "decompress stored knowledge", then surely gpt3, 3.5, 4, 4.1, 4o, o1, o3, o4, 5, 5.1, 5.x should have been able to do it, right? Claude 2, 3, 4, 4.1, 4.5? Surely. |
| |
| ▲ | shakna 2 hours ago | parent | next [-] | | Well, "Reimplement the c4 compiler - C in four functions" is absolutely something older models can do. Because most are trained, on that quite small product - its 20kb. But reimplementing that isn't impressive, because its not a clean room implementation if you trained on that data, to make the model that regurgitates the effort. | | |
| ▲ | signatoremo 15 minutes ago | parent [-] | | > Well, "Reimplement the c4 compiler - C in four functions" is absolutely something older models can do. Are you sure about that? Do you have some examples? The older Claude models can’t do it according to TFA. |
| |
| ▲ | gmueckl 2 hours ago | parent | prev | next [-] | | This comparison is only meaningful with comparable numbers of parameters and context window tokens. And then it would mainly test the efficiency and accuracy of the information encoding. I would argue that this is the main improvement over all model generations. | |
| ▲ | hn_acc1 2 hours ago | parent | prev | next [-] | | Are you really asking for "all the previous versions were implemented so poorly they couldn't even do this simple, basic LLM task"? | | |
| ▲ | Philpax an hour ago | parent [-] | | Please look at the source code and tell me how this is a "simple, basic LLM task". |
| |
| ▲ | geraneum 4 hours ago | parent | prev [-] | | Perhaps 4.5 could also do it? We don’t know really until we try. I don’t trust the marketing material as much. The fact that the previous version (smaller versions) couldn’t or could do it does not really disprove that claim. |
|
|
| ▲ | Marha01 4 hours ago | parent | prev | next [-] |
| Even with 1 TB of weights (probable size of the largest state of the art models), the network is far too small to contain any significant part of the internet as compressed data, unless you really stretch the definition of data compression. |
| |
| ▲ | jesse__ 3 hours ago | parent | next [-] | | This sounds very wrong to me. Take the C4 training dataset for example. The uncompressed, uncleaned, size of the dataset is ~6TB, and contains an exhaustive English language scrape of the public internet from 2019. The cleaned (still uncompressed) dataset is significantly less than 1TB. I could go on, but, I think it's already pretty obvious that 1TB is more than enough storage to represent a significant portion of the internet. | | |
| ▲ | FeepingCreature 2 hours ago | parent [-] | | This would imply that the English internet is not much bigger than 20x the English Wikipedia. That seems implausible. | | |
| ▲ | jesse__ an hour ago | parent [-] | | > That seems implausible. Why, exactly? Refuting facts with "I doubt it, bro" isn't exactly a productive contribution to the conversation.. |
|
| |
| ▲ | kgeist 2 hours ago | parent | prev | next [-] | | A lot of the internet is duplicate data, low quality content, SEO spam etc. I wouldn't be surprised if 1 TB is a significant portion of the high-quality, information-dense part of the internet. | | | |
| ▲ | gmueckl 3 hours ago | parent | prev [-] | | This is obviously wrong. There is a bunch of knowledge embedded in those weights, and some of it can be recalled verbatim. So, by virtue of this recall alone, training is a form of lossy data compression. |
|
|
| ▲ | 0xCMP 3 hours ago | parent | prev | next [-] |
| I challenge anyone to try building a C compiler without a big suite of tests. Zig is the most recent attempt and they had an extensive test suite. I don't see how that is disqualifying. If you're testing a model I think it's reasonable that "clean room" have an exception for the model itself. They kept it offline and gave it a sandbox to avoid letting it find the answers for itself. Yes the compression and storage happened during the training. Before it still didn't work; now it does much better. |
| |
| ▲ | hn_acc1 2 hours ago | parent [-] | | The point is - for a NEW project, no one has an extensive test suite. And if an extensive test suite exists, it's probably because the product that uses it also exists, already. If it could translate the C++ standard INTO an extensive test suite that actually captures most corner cases, and doesn't generate false positives - again, without internet access and without using gcc as an oracle, etc? |
|
|
| ▲ | brutalc 4 hours ago | parent | prev [-] |
| No one needs to prove you wrong. That’s just personal insecurity trying to justify ones own worth. |
| |
