Yes, "An LLM is just a new higher level programming language", sure. A new programming language with no tractable guarantees about the behavior of any particular program, including in practice no guarantees that the same source code (in this new programming language) will reliably produce the same behavior. This is very different from traditional programming languages and how we can reason about them.

(Yes, one can write C programs with undefined behavior, but C does also have many well-defined properties which allow people to reason about well-defined C programs logically reliably.)

(And the behavior of any given C implementation is completely defined.)