Remix.run Logo
▲ datadrivenangel 2 hours ago

Does not say what type of model or size they were serving, but one of the traces mentions Maple, which is a 20b model. Still, getting ~70+ tok/s is quite good.

▲ilaksh 3 minutes ago | parent | next [-]

Not really. It's only 1B active, and he was probably using the 5 gb quant. Also he said it was over 200 tokens per second.

But that still not a breakthrough or anything for such a small active set of parameters that is quantized

▲risyachka 2 hours ago | parent | prev | next [-]

Fpga is software-driven, model can verify its output easily.

Zero surprise it did it

▲datadrivenangel 2 hours ago | parent | prev [-]

it only cost them $10,000 in Claude API usage. :)

▲hmate9 2 hours ago | parent [-]

You get that much usage in a monthly $200 subscription.

▲dvh an hour ago | parent | next [-]

So who's paying the cost?

▲stronglikedan 32 minutes ago | parent | next [-]

Everyone else that doesn't use the entirety of the plan they pay for.

▲nyc_data_geek 28 minutes ago | parent | prev | next [-]

Retail and society as ANT shares get pulled into index and wealth funds

▲ralph84 39 minutes ago | parent | prev | next [-]

API pricing vs. individual plan pricing is just price discrimination. They're both positive gross margin.

▲surgical_fire 32 minutes ago | parent [-]

People believe in some wild stuff.

▲exe34 an hour ago | parent | prev [-]

Corporate customers

▲dofm an hour ago | parent | prev [-]

And a kitten.