The actual constraint is how long people are willing to wait for results.

If the results are expected to be really good, people will wait a seriously long time.

That’s why engineers move on to the next feature as soon as the thing is working - people simply don’t care if it could be faster, as long as it’s not too slow.

It doesn’t matter what’s technically possible- in fact, a computer that works too fast might be viewed as suspicious. Taking a while to give a result is a kind of proof of work.

> people simply don't care

I don't think that's right, even for laypeople. It's just that the pain of things that take 5 seconds when they could take 50 ms is subtle and can be discounted or ignored until you are doing a hundred things in a row that take 5 seconds instead of 50 ms. And if you don't know that it should be doable in 50 ms then you don't necessarily know you should be complaining about that pain.

> It doesn’t matter what’s technically possible- in fact, a computer that works too fast might be viewed as suspicious. Taking a while to give a result is a kind of proof of work.

In recent times I found myself falling for this preconception when a LLM starts to spit text just a couple of seconds after a complex request.