Would it be that many? Asked AI to do some rough calculation, and it spit that:

Making 50 SOTA AI requests per day ≈ running a 10W LED bulb for about 2.5 hours per day

Given I usually have 2-3 lights on all day in the house, that's like 1500 LLM requests per day (which sounds quite more than I do).

So even a month worth of requests for building some software doesn't sound that much. Having a local beefy traditional build server compling or running tests for 4 hours a day would be like ~7,600 requests/day

▲

shakna 3 hours ago | parent | next [-]

> Making 50 SOTA AI requests per day ≈ running a 10W LED bulb for about 2.5 hours per day

This seems remarkably far from what we know. I mean, just to run the data centre aircon will be an order of magnitude greater than that.

▲

anonzzzies 5 hours ago | parent | prev [-]

Is that true? Because that's indeed FAR less than I thought. That would definitely make me worry a lot less about energy consumption (not that I would go and consume more but not feeling guilty I guess).

▲

derekdahmer 2 hours ago | parent [-]

A H100 uses about 1000W including networking gear and can generate 80-150 t/s for a 70B model like llama.

So back of the napkin, for a decently sized 1000 token response you’re talking about 8s/3600s*1000 = 2wh which even in California is about $0.001 of electricity.

	▲	pshc 2 hours ago \| parent [-]
		With batched parallel requests this scales down further. Even a MacBook M3 on battery power can do inference quickly and efficiently. Large scale training is the power hog.