| ▲ | qoez 12 hours ago |
| Interesting reply from an openai insider:
https://x.com/unixpickle/status/1925795730150527191 |
|
| ▲ | epr 11 hours ago | parent | next [-] |
| Actually no, it's not interesting at all. Vague dismissal of an outsider is a pretty standard response by insecure academic types. It could have been interesting and/or helpful to the conversation if they went into specifics or explained anything at all. Since none of that's provided, it's "OpenAI insider" vs John Carmack AND Richard Sutton. I know who I would bet on. |
| |
| ▲ | handsclean 7 hours ago | parent | next [-] | | It seems that you’ve only read the first part of the message. X sometimes aggressively truncates content with no indication it’s done so. I’m not sure this is complete, but I’ve recovered this much: > I read through these slides and felt like I was transported back to 2018. > Having been in this spot years ago, thinking about what John & team are thinking about, I can't help but feel like they will learn the same lesson I did the hard way. > The lesson: on a fundamental level, solutions to these games are low-dimensional. No matter how hard you hit them with from-scratch training, tiny models will work about as well as big ones. Why? Because there's just not that many bits to learn. > If there's not that many bits to learn, then researcher input becomes non-negligible. > "I found a trick that makes score go up!" -- yeah, you just hard-coded 100+ bits of information; a winning solution is probably only like 1000 bits. You see progress, but it's not the AI's. > In this simplified RL setting, you don't see anything close to general intelligence. The neural networks aren't even that important. > You won't see _real_ learning until you absorb a ton of bits into the model. The only way I really know to do this is with generative modeling. > A classic example: why is frame stacking just as good as RNNs? John mentioned this in his slides. Shouldn't a better, more general architecture work better? > YES, it should! But it doesn't, because these environments don't heavily encourage real intelligence. | |
| ▲ | lairv 9 hours ago | parent | prev | next [-] | | Alex Nichol worked on "Gotta Learn Fast" in 2018 which Carmack mentions in his talk, he also worked on foundational deep learning methods like CLIP, DDPM, GLIDE, etc. Reducing him to a "seething openai insider" seems a bit unfair | |
| ▲ | ActivePattern 10 hours ago | parent | prev | next [-] | | It's a OpenAI researcher that's worked on some of their most successful projects, and I think the criticism in his X thread is very clear. Systems that can learn to play Atari efficiently are exploiting the fact that the solutions to each game are simple to encode (compared to real world problems). Furthermore, you can nudge them towards those solutions using tricks that don't generalize to the real world. | | |
| ▲ | dgb23 7 hours ago | parent [-] | | That sounds like an extremely useful insight that makes this kind of research even more valuable. |
| |
| ▲ | quadrature 8 hours ago | parent | prev | next [-] | | Do you have an X account, if you're not logged in you'll only see the first post in the thread. | | | |
| ▲ | kadushka 10 hours ago | parent | prev | next [-] | | He did go into specifics and explained his point. Or have you only read his first post? | |
| ▲ | MattRix 8 hours ago | parent | prev [-] | | It’s not vague, did you only see the first tweet or the entire thread? |
|
|
| ▲ | johnb231 3 hours ago | parent | prev | next [-] |
| Carmack replied to that https://x.com/ID_AA_Carmack/status/1925973500327591979 |
|
| ▲ | lancekey 5 hours ago | parent | prev | next [-] |
| I think some replies here are reading the full twitter thread, while others (not logged in?) see only the first tweet. The first tweet alone does come off as a dismissal with no insight. |
| |
|
| ▲ | jjulius 10 hours ago | parent | prev | next [-] |
| I appreciate how they don't tell us what lesson they learned. |
| |
| ▲ | dcre 9 hours ago | parent [-] | | It is a thread. You may have only seen the first tweet because Twitter is a user-hostile trash fire. “The lesson: on a fundamental level, solutions to these games are low-dimensional. No matter how hard you hit them with from-scratch training, tiny models will work about as well as big ones. Why? Because there's just not that many bits to learn.” https://unrollnow.com/status/1925795730150527191 | | |
| ▲ | jjulius 9 hours ago | parent [-] | | Thank you for clarifying. I don't have a Twitter account, and the linked tweet genuinely looks like a standalone object. Mea culpa. | | |
|
|
|
| ▲ | zeroq 11 hours ago | parent | prev | next [-] |
| >> "they will learn the same lesson I did"
Which is what? Don't trust Altman? x) |
|
| ▲ | alexey-salmin 7 hours ago | parent | prev | next [-] |
| Each of these games is low-dimensional and require not the "intelligence" but more like "reflexes", I tend to agree. However making a system that can beat an unknown game does require generalization. If not real a intelligence (whatever that means) but at the level of say "a wolf". Whether it can arise from RL alone is not certain, but it's there somewhere. |
|
| ▲ | andy_ppp 12 hours ago | parent | prev | next [-] |
| My bet is on Carmack. |
| |
| ▲ | WithinReason 9 hours ago | parent | next [-] | | "Graphics Carmack" is a genius but that doesn't mean that "AI Carmack" is too. | | |
| ▲ | MrLeap 9 hours ago | parent | next [-] | | I wouldn't bet against him. "The Bitter Lesson" may imply an advantage to someone who historically has been at the tip of the spear for squeezing the most juice out of GPU hosted parallel computation. Graphics rendering and AI live on the same pyramid of technology. A pyramid with a lot of bricks with the initials "JC" carved into them, as it turns out. | | |
| ▲ | mhh__ 4 hours ago | parent | next [-] | | I would be long carmack in the sense that I think he will have good judgement and taste running a business but I really don't see anything in common between AI and graphics. Maybe someone better at aphorisms than me can say it better but I really don't see it. There are definitely mid-level low hanging fruits that would look like the kinds of things he did in graphics but the game just seems completely different. | |
| ▲ | KerrAvon 2 hours ago | parent | prev | next [-] | | I think people would do well to read about Philo Farnsworth in this context. | |
| ▲ | kadushka 8 hours ago | parent | prev [-] | | Only if computation is the bottleneck. GPT-4.5 shows it’s not. |
| |
| ▲ | cheschire 6 hours ago | parent | prev | next [-] | | Carmack is always a genius, but like most people he requires luck, and like most people, the house always wins. Poor Armadillo Aerospace. | |
| ▲ | dumdedum123 3 hours ago | parent | prev | next [-] | | Exactly. I know him and like him. He is a genius programmer for sure BUT people forget that the last successful product that he released was Doom 3 over 20 years ago. Armadillo was a failure and Oculus went nowhere. He's also admitted he doesn't have much of math chops, which you need if you want to make a dent in AI. (Although the same could have been said of 3D graphics when he did Wolfenstein and Doom, so perhaps he'll surprise us) I wish him well TBH | |
| ▲ | mrguyorama 2 hours ago | parent | prev [-] | | What has "Graphics Carmack" actually done since about 2001? So, his initial tech was "Adaptive tile refresh" in Commander Keen, used to give it console style pixel-level scrolling. Turns out, they actually hampered themselves in Commander Keen 1 by not understanding the actual tech, and implemented "The Jolt", a feature that was not necessary. The actual hardware implemented scrolling the same way that consoles like the NES did, and did not need "the jolt", nor the limitations it imposed. Then, Doom and Quake was mostly him writing really good optimizations of existing, known and documented algorithms and 3D techniques, usually by recognizing what assumptions they could make, what portions of the algorithm didn't need to be recalculated when, etc. Very talented at the time, but in the software development industry, making a good implementation of existing algorithms that utilize your specific requirements is called doing your job. This is still the height of his relative technical output IMO. Fast Inverse Square Root was not invented by him, but was floating around in industry for a while. He still gets kudos for knowing about it and using it. "Carmack's reverse" is a technique for doing stencil shadows that was a minor (but extremely clever) modification to the "standard" documented way of doing shadow buffers. There is evidence of the actual technique from a decade before Carmack put it in Doom 3 and it was outright patented by two different people the year before. There is no evidence that Carmack "stole" or anything this technique, it was independent discovery, but was clearly also just a topic in the industry at the time. "Megatextures" from Rage didn't really go anywhere. Did Carmack actually contribute anything to VR rendering while at Oculus? People treat him like this programming god and I just don't understand. He was well read, had a good (maybe too good) work ethic, and was very talented at writing 386 era assembly code. These are all laudable, but doesn't in my mind imply that he's some sort of 10X programmer who could revolutionize random industries that he isn't familiar with. 3D graphics math isn't exactly difficult. | | |
| |
| ▲ | cmpxchg8b 4 hours ago | parent | prev | next [-] | | Appeal to authority is a logical fallacy. People often fall into the trap of thinking that because they are highly intelligent and an expert in one domain that this makes them an expert in one or more other domains. You see this all the time. | | |
| ▲ | mrandish an hour ago | parent | next [-] | | > People often fall into the trap of thinking that because they are highly intelligent and an expert in one domain that this makes them an expert in one or more other domains. While this is certainly true, I'm not aware of any evidence that Carmack thinks this way about himself. I think he's been successful enough that's he's personally 'post-economic' and is choosing to spend his time working on unsolved hard problems he thinks are extremely interesting and potentially tractable. In fact, he's actively sought out domain experts to work with him and accelerate his learning. | |
| ▲ | edanm 3 hours ago | parent | prev | next [-] | | Expecting an expert in one thing to also be pretty good at other domains, especially when they're relatively related, isn't a fallacy. | |
| ▲ | rurp 3 hours ago | parent | prev [-] | | Bayesian reasoning isn't a fallacy. A known expert in one domain is often correct about things in a related one. The post didn't claim that Carmack is right, just that that he's who they would bet on to be right, which seems perfectly reasonable to me. |
| |
| ▲ | ramesh31 9 hours ago | parent | prev | next [-] | | What has he shipped in the last 20 years? Oculus is one thing, but that was firmly within his wheelhouse of graphics optimization. Abrash and co. handled the hardware side of things. Carmack is a genius no doubt. But genius is the result of intense focused practice above and beyond anyone else in a particular area. Trying
to extend that to other domains has been the downfall of so many others like him. | | |
| ▲ | alexey-salmin 3 hours ago | parent | next [-] | | Ever since Romero departed the id Software had shipped *checks notes* Quake II, Quake III, Doom 3 and Quake 4. Funnily enough Romero himself didn't ship much either. IMO it's one of the most iconic "duo breakups". The whole is greater than the sum of the parts. | | | |
| ▲ | 8 hours ago | parent | prev [-] | | [deleted] |
| |
| ▲ | speed_spread 11 hours ago | parent | prev [-] | | I suspect Carmack in the Dancehall with the BFG. |
|
|
| ▲ | cmiles74 10 hours ago | parent | prev | next [-] |
| From a marketing perspective, this strikes me as a very predictable response. |
|
| ▲ | 12 hours ago | parent | prev | next [-] |
| [deleted] |
|
| ▲ | roflcopter69 12 hours ago | parent | prev [-] |
| Funny, I was just commenting something similar here, see https://news.ycombinator.com/item?id=44071614 And I say this while most certainly not being as knowledgeable as this openai insider. So it even I can see this, then it's kinda bad, isn't it? |
| |