That's not inherent, that's a consequence of performance optimizations. It's absolutely a choice to run those matrix calculations in a way that fails to have predictable execution ordering. It's just that the speed benefits to allowing that are considerable.

You can make it trivially deterministic by running single threaded on a cpu, but it's becomes too slow for practical applications if you do that.

▲

efficax 9 hours ago | parent [-]

well sure, but i mean realistically speaking, we cannot step debug an llm's output to find out what happened given the way we currently execute inference

	▲	solenoid0937 9 minutes ago \| parent \| next [-]
		> we cannot step debug an llm's output to find out what happened We absolutely can with mechanistic interpretability & companies like Anthropic, OpenAI, Meta, and Google do precisely this do debug their models.
	▲	embedding-shape 7 hours ago \| parent \| prev [-]
		Depends on who "we" are, what you're talking about is a thing for inference providers doing batched inference and similar stuff. If you run one inference requests locally, you can actually step-by-step debug LLM output, just there is a ton of steps. But there is nothing "inherently random" or non-deterministic involved here, just optimization strategies for the large inference servers that makes it "impossible".