Author here:

Yeah, the encoder and decoder stuff is explicit, but the internal structure in generated during training. I don't think the big labs were doing this back when I did the research; no one was back in '24.

I just didn't get round to publishing for years, because I have a day job.

By the way, it still works! I tested it earlier this year on Qen3.6 and you still see improvements, so either a) no one actually paid attention, or b) it has more room to scale.

▲

mike_hearn 13 hours ago | parent [-]

I think you're right that the idea of looping layers is unique to you, congrats and thanks for writing those great blog posts (I read them for the first time a few days ago!). But the idea that the thinking is happening in an abstract space via neural circuits in the middle layers I feel was one that I was reading about in 2024 at least, as Anthropic have been doing this kind of research for a long time. Maybe I'm misremembering though!

My impression from reading the literature is that there are a gazillion interesting ideas and findings published that nobody is picking up in production models. The big labs are researcher constrained, there just aren't enough hours in the day to keep up with the literature and integrate all the interesting ideas found there. So it's not surprising that your trick still works. It'd be even less surprising to discover nobody at these labs has read your blogs, or they have but never found time to experiment with them. Or, they tried, but there is no set of loops that improves some metrics without harming others - I would expect neural circuits to be misaligned across the middle layers so looping layers for one task would put a fault line in circuits for other tasks.

Then they have to trade off the extra GPU capacity needed to do the extra layers, and so on.

	▲	tedd4u 12 hours ago \| parent [-]
		The Ouro looping results are interesting [1] and they are focused more on the improved reasoning from looping middle layers rather than the parameter efficiency aspect. They train 1.4 and 2.6B parameter models with 7T tokens. The training includes learning how many times to loop on any given token (there’s an early exit module). My guess as to why (as far as we know) looping is not in frontier models yet is that, at frontier training run scale, it’s probably going to require a lot of trial and error and at-scale research. While currently they already probably have a list of dozens or hundreds of of promising ideas that don’t complicate things as much. In the other hand, Ouro’s looping technique shows ability to compete well with models with 3x parameters which seems attention-getting to me. If there’s another 3x to be had down that path. It’s order of magnitude opportunity. Btw there is a great related work section in the paper. [1] https://ouro-llm.github.io/