Remix.run Logo
▲ derefr an hour ago

In general, one could assert without knowing any details that:

1. the models likely have orders-of-magnitude different backend costs to run (e.g. each warmed instance of the model reserving one slice of one GPU, vs one full GPU, vs layers distributed across multiple GPUs) -- as otherwise it wouldn't be worth explicitly distinguishing these models and putting different token spends on them (they'd instead just be automatically routed between); and

2. there's at least one particular specialized problem domain, not captured by this benchmark yet still relevant to external customers (so not just "it's better at training data evaluation/synthesis"), where these models are more-competent-enough to make their higher per-token costs "worth it" to those external customers.

I won't hazard a guess at what that problem domain is; I'm sure it'll be deduced after people spend a few weeks playing with it.