> So I simultaneously can tell you that its smart people really thinking about every facet of the problem, and I can't tell you much more than that.

"we do 1970s mainframe style timesharing"

there, that was easy

▲ kstrauser 7 days ago | parent | next [-]

For real. Say it takes 1 machine 5 seconds to reply, and that a machine can only possibly form 1 reply at a time (which I doubt, but for argument).

If the requests were regularly spaced, and they certainly won’t be, but for the sake of argument, then 1 machine could serve 17,000 requests per day, or 120,000 per week. At that rate, you’d need about 5,600 machines to serve 700M requests. That’s a lot to me, but not to someone who owns a data center.

Yes, those 700M users will issue more than 1 query per week and they won’t be evenly spaced. However, I’d bet most of those queries will take well under 1 second to answer, and I’d also bet each machine can handle more than one at a time.

It’s a large problem, to be sure, but that seems tractable.

	▲	mh- 7 days ago \| parent [-]
		Yes. And batched inference is a thing, where intelligent grouping/bin packing and routing of requests happens. I expect a good amount of "secret sauce" is at this layer. Here's an entry-level link I found quickly on Google, OP: https://medium.com/@wearegap/a-brief-introduction-to-optimiz...

▲ brookst 7 days ago | parent | prev | next [-]

But that’s not accurate. There are all sorts of tricks around KV cache where different users will have the same first X bytes because they share system prompts, caching entire inputs / outputs when the context and user data is identical, and more.

Not sure if you were just joking or really believe that, but for other peoples’ sake, it’s wildly wrong.

▲ kossTKR 5 days ago | parent [-]

Really? So the system recognises someone asked the same question and serves the same answer? And who on earth shares the exact same context?

I mean i get the idea but sounds so incredibly rare it would mean absolutely nothing optimisation wise.

▲ brookst 4 days ago | parent | next [-]

Yes. It is not incredibly rare, it's incredibly common. A huge percentage of queries to retail LLMs are things like "hello" and "what can you do", with static system prompts that make the total context identical.

It's worth maybe a 3% reduction in GPU usage. So call it a half billion dollars a year or so, for a medium to large service.

	▲	throwaway2037 2 days ago \| parent [-]
		`> It's worth maybe a 3% reduction in GPU usage. So call it a half billion dollars a year or so, for a medium to large service.` So if 3% is 500M, then annual spend is ~16.6B. That is medium sized these days?

▲ fc417fc802 5 days ago | parent | prev [-]

Even if that were the case you wouldn't be wrong. Adding caching and deduplication (and clever routing and sharding, and ...) on top of timesharing doesn't somehow make it not timesharing anymore. The core observation about the raw numbers still applies.

▲ claytongulick 6 days ago | parent | prev | next [-]

I'm pretty sure that's not right.

They're definitely running cluster knoppix.

:-)

▲ rootsudo 7 days ago | parent | prev [-]

Makes perfect sense, completely understand now!