it might know all this is my point and it can reason about it regardless.

The point is that it has limits in what tools it can use by virtue of being a finite program. Those limits imply that there are some math problems that it cannot solve. The LLM can know about this (and frontier LLMs likely do), but that doesn't mean it will be able to solve the math problems.

To put another way, if it were true that some fixed LLM could solve every math problem, it would implement Halt, which is impossible.