I think we can talk about intelligence without introducing consciousness.

So there are two questions

1. Is LLMs really showing intelligent behavior. 2. Is that real intelligence.

I think even to the first question, the answer is no. Because really intelligent behavior that can solve complex math problems should be also be able to do basic things like counting, without any extra mechanisms. Intelligent behavior would understand low level things, and building understanding of higher level things using them. When an entity shows "understanding of higher level things, without showing an understanding of some of the low level things involved, then it becomes clear that the entity is not intelligent.

So this should be proof enough to identify that LLMs does not in fact show intelligent behavior. It is just some trick. An illusion. So we can reach this conclusions even without consider the implementation and training of LLMs, which We would only have to consider it to answer the second question, which was not required because it failed the first one.