Remix.run Logo
▲ rhdunn an hour ago

Do we know that that is the case?

We know the architectures of the open weight models as they include architecture diagrams, support in projects like llama.cpp, etc.

Companies like Anthropic and OpenAI don't release information about their models (aside from general things like context window size, whether they support reasoning, etc.). A lot of what we know is speculation. From press releases Anthropic have stated that they are getting Claude to assist with designing new iterations of the models, but no specific details.

The open weight models and other arXiv papers/research are iterating on and experimenting with the base/core transformer architecture:

- RoPE/NoPE/etc. for context window scaling;

- dense vs mixture of experts models;

- Mamba-based State Space Models (falcon models, etc.);

- reasoning, combining transformers with a residual stream, Attention Residuals (Kimi), etc.

- activation functions: ReLU; SwiGLU; etc.

- etc.

▲bobmarleybiceps 40 minutes ago | parent [-]

I think a lot of model architecture changes don't move the needle ~that much and are basically done to improve runtime performance and scaling. - There's tons of cases of ml researchers, tech companies, and labs that propose new and improved models that turn out to not really be significant quality improvements (if at all) once the community starts experimenting with them.

it does seem like (IMO) the biggest leaps have basically been perf & scaling + RL on high quality data + harnesses.

▲versteegen 3 minutes ago | parent [-]

Yes, but "improving runtime performance and scaling": does directly translate into huge quality improvement too, because then you can you a much larger or deeper model, train it for much longer, and have a larger context.