How is that any different from Googling something and believing any of the highly-SEO optimized results that pollute the front page?

That's the point, nobody really believes there is an intelligence generating Google results. It is a best-effort based engine. However, people have this belief that ChatGPT has somehow some intelligent engine generating results, which is incorrect. It is only generating statistically good results; if it is true or false depends on what the person using it will do with the results. If it is poetry, for example, it is always true. If it is how to find the cure for cancer it will with very high probability be false. But if you're writing a novel about a scientist finding a cure for cancer, then that same response will be great.

Search engines didn't need $500B and growing in CAPEX.