I think the downside would mainly be complexity. As soon as you do that, you have to develop what that intermediate representation is and how it gets stored. But moreover, you'd need to develop workarounds for the fact that all external code infrastructure (version control, editors, command line tools) is built for text.

I can't see a crazy huge downside from a python point of view, but seems like a much bigger upside than flexible formatting would be needed to justify breaking from all of that stuff.

I was thinking just plain python - let's say Black formatted code is the default and we commit only that. Then on the visualization side, the IDE can format it for whatever we want.

Actually, this could be a really easy feature for the IDE and could work already easily.