Token use exploded a week or two ago. I worked for a few hours and hit my 5 hour limit. Did only some simple bug fix. Somethings definitely broken.

This is the #1 thing i HATE with LLMs. They are not predictable like a compiler is.

▲

cromka a day ago | parent | next [-]

It's not LLMs! Its their providers! If you self-hosted, you'd observe the slow down but you'd get same quality.

▲

jatora 17 hours ago | parent [-]

Weird! So you're claiming local models somehow are deterministic? Amazing.

▲

cogman10 15 hours ago | parent | next [-]

All models are deterministic.

There is random input to make the models evaluate differently every time, but without that (or if you fix it) then the model will always give the same output with the same input.

There's no somehow to it, it's actually surprisingly hard to make a computer do something in a non-deterministic fashion.

	▲	jatora 9 hours ago \| parent [-]
		I was being sarcastic. LLM output is not deterministic (at least not in any meaningful way beyond the fact that everything in the universe is deterministic).

▲

cromka 4 hours ago | parent | prev [-]

No, that's not what I am claiming.

Damn, do I hate such manipulating fallacies.

▲

KeplerBoy a day ago | parent | prev | next [-]

It's probably not token usage, but a flexible 5 hour limit. There's a reason why they only state it in percentages.

	▲	lukan a day ago \| parent [-]
		Nope, the issue was not the 5 h limit, but that the 5 hour limit was gone in some minutes and that this burned trough 50% of my weeks fable limit. Before I could do roughly one full 5h session (fable till max) per day.

▲

monster_truck a day ago | parent | prev [-]

They're actually extremely predictable. What you are experiencing is the provider you are entrusting with your time, effort, and money fucking you because they need the capacity to train the next one. So without any real notice or indication other than things vaguely seeming broken in ways everyone seems to be experiencing differently, they're serving up some shitty quant of what you already paid to use.

▲

singularity2001 a day ago | parent [-]

Extremely predictable is an exaggeration. I was teaching machine learning and we needed half a dozen lines to make even simple networks deterministic. LLMs are not really predictable, even if we want to, they will sometimes behave almost chaotic depending on the input.

▲

techmunky 18 hours ago | parent | next [-]

My understanding of LLMs are that they are giant functions and generate output deterministically. However, the way these providers setup and route traffic means that your traffic lands on different boxes where each box has different inputs that run alongside your prompt -- essentially creating "non-deterministic" behavior. Basically it's BS to make LLMs look sentient. Each model could be burned onto an ASIC if you really wanted to be locked into that model and get incredible token speeds. If I'm wrong I would like to be corrected, thanks.

	▲	singularity2001 14 hours ago \| parent [-]
		I replied to your neighbor https://news.ycombinator.com/item?id=50432889

▲

bmacho a day ago | parent | prev | next [-]

> we needed half a dozen lines to make

6 lines per model? Can't you put that 6 lines in a function/macro/decorator/library so you only have to call it?

▲

monster_truck a day ago | parent | prev [-]

I'm not sure how you're arriving at "if you use the same seed with the same version of the software on the same hardware with the same drivers you get the same result is an exaggeration", it's wild that you're a teacher and you seemingly do not know this.

	▲	hgoel 20 hours ago \| parent \| next [-]
		Order of operations affects the result in floating point, even moreso at lower precision. GPUs are massively parallel devices with many different sources of latency, so order of operations is not guaranteed for things like dot products, where many threads contribute to the result. Consequently, you can get non-deterministic outputs even though nothing else has changed.
	▲	singularity2001 14 hours ago \| parent \| prev [-]
		[dead]