At the end of the page is this paragraph:

Another, more subtle justification for the Butlerian Jihad is also found in Frank Herbert's original novels, specifically Heidegger's thesis that the use of technology trains humans to think like machines. The problem is that machines are deterministic; thus, training people to be machines is self-limiting.

I wonder whether Heidegger would consider the probabilistic nature of LLMs as having a positive effect on human thought.

LLM's are probabilistic but with a very limited range. They are tuned to be predictable enough to serve as a consumer product, so most ideas you get from them will revolve around the same concepts repeatedly.

LLM force humans to think like machines far more than just writing code.

Writing code is a creative activity with large number of degrees of freedom and opportunities to express yourself. Button pushing and approving code changes is a deeply limiting activity that alienates you from your work.

Although LLMs are not deterministic in normal operation (because of the temperature values used) they are still strongly biased in their outputs. Bias in their outputs is literally how they function.

So you get a different type of conformism where people who interact with LLMs a lot are going to be influenced towards the median, as encoded in those token probability look up tables.

The effect is basically the same as what Heidegger and Illich were worried about, and I think we're seeing it happen already everywhere.

A solution(?) is to try to be your true weird authentic self when you act in the world, rather than bending your words and behaviours to the machine and LLM probability shape.

Ok this fascinating. I happen to have some Heidegger files, and went a-greppin. I'd say he's really talking about "inaminate objectness", as opposed to our human animate subjectness. Reasons:

1. In Science and Reflection, he says that both the 'completely calculated' nature of classical mechanics and the 'statistical character' of quantum mechanics are two representation of an underlying 'objectness of material nature'.

2. In Discourse on Thinking, he's far more concerned with the instrumental, utilitarian nature of machines than literal determinism, a concept he (somewhat confusingly/uniquely) connects to computation/'computational thinking', as opposed to the inherently-human capability for 'meditative thinking'. I personally would say this bit is refuted by LLMs in the end, but because they have intuitive rationality, a capability that seemed impossible to artificially design to the scientists of the time -- not because they involve dice.

3. In What is Thinking?, he gives the closest connection of this dichotomy to one of his better-known main topics, namely [handi]craft as a special subset of work: 'what maintains and sustains even this handicraft is not the mere manipulation of tools, but the relatedness to wood.' In this sense machines would be merely manipulating tools, stochastic or otherwise.

4. Overall I get the sense from skimming that lecture that his concern is a little mischaracterized, and in a kind of beautiful way that backs up my "Dune is still about AI" thesis described in a top-level comment somewhere below: his concern is about any society that could build complex machines (a 'technological' one), not the behavioral aspects of the machines themselves. So that concern applies equally to Star Trek (AGI), 1984 (no AGI), and our current post-singularity society (it's complicated/potentially ""just"" guessing).

All of this fits very well into his central thesis, which is continuing the Kantian & Hegelian tradition of examining how humans might be moral, free, and spiritual despite being meat computers of a largely-unknowable nature. In Heideggerland, this is expressed best in dramatic, nigh-unreadable passages like:

  Temporality already maintains itself ecstatically within the horizons of its ecstases; and in temporalizing itself, it comes back to those entities which are encountered in the "there".
  With Dasein's factical existence, entities within-the-world are already encountered too.
  The fact that such entities are discovered along with Dasein's own "there" of existence, is not left to Dasein's discretion.
  Only what it discovers and discloses on occasion, in what direction it does so, how and how far it does so—only these are matters for Dasein's freedom, even if always within the limitations of its thrownness.