By the way why don't these LLM interfaces come with a pause button?

And a "prune here" button.

It often happens that the interesting information is in the first paragraph or so, and the remainder is all just the LLM not knowing when to stop. This is super annoying as a conversation then ends up being 90% noise.

▲

postalcoder 3 hours ago | parent [-]

Pruning an assistant's response like that would break prompt caching.

Prompt caching is probably the single most important thing that people building harnesses think about and yet it's mind share in end users is virtually zero. If you had to think of all the weirdest, most seemingly baffling design decisions in an AI product, the answer to "why" is probably "to not break prompt caching".

	▲	amelius 31 minutes ago \| parent [-]
		Maybe so, but pruning is still a useful feature. If it hurts performance that much, maybe pruning could just hide the text leaving the cache intact?

▲

stainablesteel 5 hours ago | parent | prev [-]

i imagine they're doing superman level distributed compute across multiple clouds somewhere and cared more about delivering the final result of that than having the ability to pause. which is probably possible, but would require way more work than would be worthwhile. they probably thought the ability to stop and resubmit would be an adequate substitute.

▲

amelius 5 hours ago | parent [-]

These models are autoregressive so I doubt they are running them across multiple clouds. And besides, a pause button is useful from a user's pov.

▲

stainablesteel 4 hours ago | parent [-]

i'm not sure it is, what's so useful about it?

	▲	amelius 2 hours ago \| parent [-]
		Like I said in another comment: It often happens that the interesting information is in the first paragraph or so, and the remainder is all just the LLM not knowing when to stop. This is super annoying as a conversation then ends up being 90% noise.