| ▲ | sigmoid10 a day ago |
| Sooo... it can play Pokemon. Feels like they had to throw that in after Google IO yesterday. But the real question is now can it beat the game including the Elite Four and the Champion. That was pretty impressive for the new Gemini model. |
|
| ▲ | minimaxir a day ago | parent | next [-] |
| That Google IO slide was somewhat misleading as the maintainer of Gemini Plays Pokemon had a much better agentic harness that was constantly iterated upon throughout the runtime (e.g. the maintainer had to give specific instructions on how to use Strength to get past Victory Road), unlike Claude Plays Pokemon. The Elite Four/Champion was a non-issue in comparison especially when you have a lv. 81 Blastoise. |
| |
| ▲ | fourier456 a day ago | parent [-] | | Okay, wait though like I want to know the full transcript because that actually is a better / softer benchmark if you measure in terms of the necessary human input. |
|
|
| ▲ | archon1410 a day ago | parent | prev | next [-] |
| Claude Plays Pokemon was the original concept and inspiration behind "Gemini Plays Pokemon". Gemini arguably only did better because it had access to a much better agent harness and was being actively developed during the run. See: https://www.lesswrong.com/posts/7mqp8uRnnPdbBzJZE/is-gemini-... |
| |
|
| ▲ | throwaway314155 a day ago | parent | prev | next [-] |
| Gemini can beat the game? |
| |
| ▲ | mxwsn a day ago | parent | next [-] | | Gemini has beat it already, but using a different and notably more helpful harness. The creator has said they think harness design is the most important factor right now, and that the results don't mean much for comparing Claude to Gemini. | | |
| ▲ | throwaway314155 a day ago | parent [-] | | Way offtopic to TFA now, but isn't using an improved harness a bit like saying "I'm going to hardcore as many priors as possible into this thing so it succeeds regardless of its ability to strategize, plan and execute? | | |
| ▲ | silvr a day ago | parent | next [-] | | While true to a degree, I think this is largely wrong. Wouldn't it still count as a "harness" if we provided these LLMs with full robotic control of two humanoid arms, so that it could hold a Gameboy and play the game that way? I don't think the lack of that level of human-ness takes away from the demonstration of long-context reasoning that the GPP stream showed. Claude got stuck reasoning its way through one of the more complex puzzle areas. Gemini took a while on it also, but made it through. I don't that difference can be fully attributed up to the harnesses. Obviously, the best thing to do would be to run a SxS in the same harness of the two models. Maybe that will happen? | | |
| ▲ | throwaway314155 a day ago | parent [-] | | I can appreciate that the model is likely still highly capable with a good harness. Still, I think this is more in line with ideas from say, speed running (or hell even reinforcement learning) where you want to prove something profound is possible and to do so before others do, you need to accumulate a series of "tricks" (refining exploits/hacking rewards) in order to achieve the goal. but if you use too many tricks you're no longer proving something as profound as originally claimed. In speed running this tends to splinter into multiple categories. Basically, the gane being conpleted by gemini was in an inferior category (however minuscule) of experiment. I get it though. People demanded these types of changes in the CPP twitch chat, because the pain of watching the model fail in slow motion is simply too much. |
| |
| ▲ | samrus a day ago | parent | prev | next [-] | | it is. the benchmark was somewhat cheated, from the perspective of finding out how the model adjusts and plans within a dynamic reactive environment | |
| ▲ | 11101010001100 a day ago | parent | prev [-] | | They asked gemini to come up with another word for cheating and it came up with 'harness'. |
|
| |
| ▲ | klohto a day ago | parent | prev [-] | | 2 weeks ago |
|
|
| ▲ | hansmayer a day ago | parent | prev [-] |
| Right, but on the other hand... how is it even useful? Let's say it can beat the game, so what? So it can (kind of) summarise or write my emails - which is something I neither want nor need, they produce mountains of sloppy code, which I would have to end up fixing, and finally they can play a game? Where is the killer app? The gaming approach was exactly the premise of the original AI efforts in the 1960s, that teaching computers to play chess and other 'brainy' games will somehow lead to development of real AI. It ended as we know in the AI nuclear winter. |
| |
| ▲ | samrus a day ago | parent | next [-] | | from a foundational research perspective, the pokemon benchmark is one of the most important ones. these models are trained on a static task, text generation, which is to say the state they are operating in does not change as they operate. but now that they are out we are implicitly demanding they do dynamic tasks like coding, navigation, operating in a market, or playing games. this are tasks where your state changes as you operate an example would be that as these models predict the next word, the ground truth of any further words doesnt change. if it misinterprets the word bank in the sentence "i went to the bank" as a river bank rather than a financial bank, the later ground truth wont change, if it was talking about the visit to the financial bank before, it will still be talking about that regardless of the model's misinterpretation. But if a model takes a wrong turn on the road, or makes a weird buy in the stock market, the environment will react and change and suddenly, what it should have done as the n+1th move before isnt the right move anymore, it needs to figure out a route of the freeway first, or deal with the FOMO bullrush it caused by mistakenly buying alot of stock we need to push against these limits to set the stage for the next evolution of AI, RL based models that are trained in dynamic reactive environments in the first place | | |
| ▲ | hansmayer a day ago | parent [-] | | Honestly I have no idea what is this supposed to mean, and the high verbosity of whatever it is trying to prove is not helping it. To repeat: We already tried making computers play games. Ever heard of Deep Blue, and ever heard of it again since the early 2000s? | | |
| ▲ | lechatonnoir a day ago | parent | next [-] | | Here's a summary for you: llm trained to do few step thing. pokemon test whether llm can do many step thing. many step thing very important. | | |
| ▲ | hansmayer 21 hours ago | parent [-] | | Are you showing off how the the extensive LLM usage impaired your writing and speaking capabilities? | | |
| |
| ▲ | Rudybega a day ago | parent | prev [-] | | The state space for actions in Pokemon is hilariously, unbelievably larger than the state space for chess. Older chess algorithms mostly used Brute Force (things like minimax) and the number of actions needed to determine a reward (winning or losing) was way lower (chess ends in many, many, many fewer moves than Pokemon). Successfully navigating through Pokemon to accomplish a goal (beating the game) requires a completely different approach, one that much more accurately mirrors the way you navigate and goal set in real world environments. That's why it's an important and interesting test of AI performance. | | |
| ▲ | hansmayer 21 hours ago | parent [-] | | Thats all wishful thinking, with no direct relation to the actual use cases. Are you going to use it to play games for you? Here is a much more reliable test: Would you blindly copy and paste the code the GenAI spits out at you? Or blindly trust the recommendations it makes about your terraform code ? Unless you are a complete beginner, you would not, because it sometimes generates downright the opposite of what you asked it to do. It is because the tool is guessing the outputs and not really knowing what it all means. It just "knows" what character sequences are most likely (probability-wise) to follow the previous sequence. Thats all there is to it. There is no big magic, no oracle having knowledge you dont etc. So unless you tell me you are ready to blindly use whatever the GenAI playing pokemon tells you to do, I am sorry, but you are just fooling yourself. And in the case you are ready to blindly follow it - I sure hope you are ready for a life of an Eloi? | | |
| ▲ | Rudybega 7 hours ago | parent [-] | | All of that is totally unrelated to the point I'm trying to make. Pokemon is interesting because it's a test of whether these models can solve long time horizon tasks. That's it. | | |
| ▲ | hansmayer 6 hours ago | parent [-] | | Ok, well now that you phrase it clearly like that, it makes much more sense, so it's a test of being able to keep a relatively long context-length. Another incremental improvement I suppose. |
|
|
|
|
| |
| ▲ | j_maffe a day ago | parent | prev | next [-] | | > Where is the killer app? My man, ChatGPT is the sixth most visited website in the world right now. | | |
| ▲ | hansmayer a day ago | parent [-] | | But I did not ask "what was the sixth most visited website in the world right now?", did I? I asked what was the killer app here. I am afraid vague and un-related KPIs will not help here, otherwise we may as well compare ChatGPT and PornHub based on the number of visits, as you seem to suggest. | | |
| ▲ | signatoremo a day ago | parent | next [-] | | Are you saying PornHub isn’t a killer app? | | | |
| ▲ | j_maffe a day ago | parent | prev [-] | | If the now default go-to source for quick questions and formulating text isn't a killer app in your eyes, I don't know what is. | | |
| ▲ | Jensson a day ago | parent | next [-] | | Not killer enough to warrant trillions of dollars valuation that the VC money are looking for here. | | |
| ▲ | hansmayer 21 hours ago | parent [-] | | The VCs have already burnt on the order of 200B USD, to generate about 10B operating income, within the total "industry" (source: The Information). It will be interesting when some of them start asking about the returns. |
| |
| ▲ | hansmayer 21 hours ago | parent | prev [-] | | I already know how to read, write and think by myself, so no - that is not a killer app. Especially when it produces wrong answers with an authoritative attitude. | | |
| ▲ | j_maffe 18 hours ago | parent [-] | | Then it's not a killer app for you. Doesn't stop it from being so for many others. |
|
|
|
| |
| ▲ | minimaxir a day ago | parent | prev | next [-] | | It's a fun benchmark, like simonw's pelican riding a bike. Sometimes fun is the best metric. | |
| ▲ | lechatonnoir a day ago | parent | prev [-] | | This is a weirdly cherry-picked example. The gaming approach was also the premise of DeepMind's AI efforts in 2016, which was nine years ago. Regardless of what you think about the utility of text (code), video, audio, and image generation, surely you think that their progress on the protein-folding problem and weather prediction have been useful to society? What counts as a killer app to you? Can you name one? | | |
| ▲ | hansmayer 19 hours ago | parent | next [-] | | Well the example came from their own press-release, so who cherry-picked it? Why should I name the next killer app ? Isnt that something that we just recognise the moment it shows up, like we did with www and e-commerce? Its not something a comittee staffed by a bunch of MBAs defines ahead of the time, as is currently the case with the use-cases that are being pushed into our faces every day. I would applaud and cheer if their efforts were focused on scientific problems that you mentioned. Unfortunately for us, this is not what the bean-counters heading all major tech corps see as useful. Do you honestly think any one of them has the benefit of society at heart? No, they want to make money by selling you bullshit products like e-mail summarising and such. Perhaps in the process also to get rid of software developers altogether as well. Then once we as the society lose the ability to do anything on our own, relying on these bullshit machines they gain not only in terms of being able to entshittify their products and squeeze that extra buck, but also opens a "world of possibilities" (for the rich) in terms of societal control. But sure, at least you will still have your, what is it now, two-day delivery from Amazon and a handholding tool to help you speak, write and do anything meaningful as a human being. | |
| ▲ | rxtexit 14 hours ago | parent | prev [-] | | The whole idea of a "killer app" is stupid. It is a dismissive rhetorical device to prove a wrong point on an internet forum such as this that has nothing to do with reality. | | |
|
|
