> Take a list of files for example. you can run ls -l in a terminal and it will attempt a crude text based formatting of the output. or you can run a full gui filemanager.

ls's output (and a few other programs') are human facing. They're not meant to be machine readable.

> what if there was a command that would produce ls output as a table data structure, run in a sort of terminal that knows how to format a table nicely without the ls command having to include any formatting instructions? this terminal would have support for all sorts of data types and structures to display all of them nicely.

Because that's just complexity for its own sake. What's simpler is to just create a shell alias for a particular view you want. The shell is for interactive usage, with some capabilities for automation. If something better is needed, you write a real program for it.