| ▲ | Latent Lattice Language Models(shivamduggal4.github.io) | |
| 30 points by E-Reverance 5 hours ago | 4 comments | ||
| ▲ | augment_me 3 hours ago | parent | next [-] | |
Seems like just an extension of Short-conv in the linear attention methods where we instead carry the EMA of the entire sequence instead of nearest 4 tokens. It would make sense in my opinion to do the evals based on the reason for needing Short-conv in the architectures that use it in the first place, where it was added to support exact retrieval/needle-in-haystack when we compress the attention state. It's odd to make this arch extension and not test beyond a 2k context window, or with the needle-in-haystack benchmark. In my head this should fail because the EMA cannot carry this granular information | ||
| ▲ | royal__ 3 hours ago | parent | prev | next [-] | |
How significant is a .98 decrease in perplexity. I'm a researcher in this field and I'm having trouble interpreting the implication of this. | ||
| ▲ | tkslucas 3 hours ago | parent | prev | next [-] | |
Interesting approach to scale compute, never thought of using the per layer state like that | ||
| ▲ | bionhoward 2 hours ago | parent | prev [-] | |
I am so excited for RNNs to be cool again | ||