I don't think that the concept of "real reasoning" vs simulated or fake reasoning makes any sense... LLM reasoning can be regarded as a subset of human reasoning, and a more useful comparison would be not real vs fake but rather what is missing from LLM reasoning that would need to be added (likely in a completely new architecture - not an LLM/transformer) to make it more human-like and capable.

Human reasoning, and cortical function in general, would also appear to be prediction based, but there are many differences to LLMs, starting with the fact that we learn continuously and incrementally from our own experience and prediction failures and successes. Human reasoning is basically chained what-if prediction, based on predictive outcomes of individual steps that we have learnt, either in terms of general knowledge or domain-specific problem solving steps that we have learnt.

Perhaps there is not so much difference between what a human does and an LLM does in, say, tackling a math problem when the RL-trained reasoning-LLM chains together a sequence of reasoning steps that worked before...

Where the difference come in, is in how the LLM learned those steps in the first place, and what happens when its reasoning fails. In humans these are essentially the same thing - we learn by predicting and giving it a go, and learn from prediction failure (sensory/etc feedback) to update our context-specific predictions for next time. If we reach a reasoning/predictive impasse - we've tried everything that comes to mind and everything fails, then our innate traits of curiosity and boredom (maybe more?) come to play and we will explore the problem and learn and try again. Curiosity and exploration can of course lead to gain of knowledge from things like imitation and active pursuit (or receipt) of knowledge from sources other then personal experimentation.

The LLM of course has no ability to learn (outside of in-context learning - a poor substitute), so is essentially limited in capability to what it has been pre-trained on, and pre-training is never going to be the solution to a world full of infinite ever-changing variety.

So, rather than say that an LLM isn't doing "real" reasoning, it seems more productive to acknowledge that prediction is the basis of reasoning, but that the LLM (or rather a future cognitive architecture - not a pass-thru stack of transformer layers!) needs many additional capabilities such as continual/incremental learning, innate traits such as curiosity to expose itself to learning situations, and other necessary cognitive apparatus such as working memory, cognitive iteration/looping (cf thalamo-cortical loop), etc.