Update: Seems quite literally they have bugs on the hardware I'm trying to run this with:

From Poolside CEO Eiso Kant on Twitter:

> Learning we have some bugs on the RTX6000. We’re on it. Team has worked non stop last days and it’s getting late for a lot of the inference folks, so might be until tomorrow till we have a solution. - https://x.com/eisokant/status/2079693050796785720

Update2: I'm now running poolside/Laguna-S-2.1-NVFP4 with vLLM 0.23.1rc1.dev1378+gd6dbdb9b0 (FlashInfer 0.6.14) and seeing slightly better results in regards to the looping. I can't see any specific changes that would affect this though, strangely enough.

▲ embedding-shape 11 hours ago | parent | next [-]

Update3, summarized from RTX6kPRO Discord:

> The BF16 checkpoint doesn't exhibit any of the complaints [...] with our current quants, the models tend to choose the wrong logits sometimes [...] why we'll need a requant [...] We're not aware of any bugs in any runtimes themselves [...] We have two remaining things that we're trying to tackle: some people are reporting thinking being too hard to trigger (^^) , and others are saying it thinks too much. We've seen much more of the latter internally

▲ Lwerewolf 11 hours ago | parent [-]

Deleted earlier, didn't see you post, pasting here:

/* Just started testing with the gguf (with gpu offload, m5 max 128gb), q4_k_m, running seemingly well. Speed is initially slightly faster than antirez/ds4 - decode tok/s in the 30s, prefill ~400-ish. Expected, given the slightly smaller size. Looks to be working fine, but too early to tell. Definitely likes to "think". */

Anyways, guessing that discord might be focusing on the nvfp4 stuff. I've noticed spelling mistakes in the thinking traces, tool calls have been fine so far.

▲ Lwerewolf 7 hours ago | parent [-]

Well, just ran said gguf on the GeneralsX codebase with a pretty open-ended "Explain this codebase to me, and the general game loop." prompt, and...

  Let me also look at the GameLogic::update() to see the rest of the update flow, especially the object update loop.

  Actually, I think I have enough information now. Let me also check the GameLogic::UPDATE to understand the full update flow.

...repeating forever.

Since they mentioned that they're working on new quants, guess I'll wait. From earlier tests on work stuff, it's definitely capable.

▲

Lwerewolf 5 hours ago | parent [-]

Right:

https://huggingface.co/poolside/Laguna-S-2.1-FP8/discussions...

Wait time.

	▲	embedding-shape 5 hours ago \| parent [-]
		More updates from the Discord (also mostly about the NVFP4): > we've got an NVFP4 checkpoint that has a KD of 0.135 relative to the BF16 checkpoint (for W4A4, on some GSM8K prompts). [...] We're still seeing some looping on W4A4 (although much less than before), whereas for W4A16 we've not been able to make it loop. Given that W4A4 is still a little broken, we won't make this an official release (we have one more idea planned to fix that). [...] The RC1 for NVFP4 is reachable as poolside/Laguna-S-2.1-NVFP4 @ RC1 Seems to be available already indeed: https://huggingface.co/poolside/Laguna-S-2.1-NVFP4/tree/RC1

▲ 11 hours ago | parent | prev [-]

[deleted]