I feel like the discussion here is missing the point. It doesn't matter if the AI overview is correct or not, it doesn't matter if you can turn it off or not. People are using it instead of visiting actual websites. Google has copied the entire World Wide Web into their LLM and now people not using the web anymore! We have bemoaned the fact that Facebook and Twitter replaced most of the web for most people, but now it's not even those, it's a single LLM owned and controlled by a single corporation.

Is there an appreciable difference between a company that controls what information is surfaced via pagerank and one that does so via LLM?

Remember the past scandals with google up/downranking various things? This isn't a new problem. Wrt how the average person gets information google doesn't really have more control because people aren't clicking through as much.