Remix.run Logo
kelseyfrog 5 hours ago

I'm literally zero concerned.

Looped transformers replace n-different self attention layers into one layer that gets executed m-times usually until a stopping condition is met. My personal intuition is that it just leaves another degree of freedom in the way QKV weights can be packed so that it's slightly more efficient.

You have to take a step back and examine the context in which the post is written. The LW/EA community is just a little obsessed with AI safety - it's easy to construct hypothetical events where A(G/S)I exterminates humanity that function as a technological version of Pascal's Wager.

One of the AI safety interests is AI explainability - the thought here that reading an AI's 'thoughts' will help us design safer models as well as detect models that go 'rogue' or are malevolently plotting against humans.

That's where the fear of looped transformers comes from. Is the residual stream that looped transformers iterate on a potential hiding place for plotting AI?

In my opinion, no more so than the residual stream of existing transformers. It changes zero.

dist-epoch 5 hours ago | parent [-]

You could imagine large number of loops, thousands. But you are constrained by the width of the residual stream since you loop over one token. But then you can imagine the model learning to sub-divide it to pack even more info into it.

kelseyfrog 5 hours ago | parent [-]

I can imagine a lot of things. However there is a packing limit for QKV weights that sets the ceiling on how much this occurs, and it's quite low - think 1.3-1.8x. The limiting factor here isn't the number of iterations, it's model size, same as it ever was.