There was a movement towards working with syntax trees directly and treating source code as a generated serialization of those syntax trees about 20-25 years ago. This probably started with refactoring as it was pioneered in the nineties. Things like Visual Age actually stored code in a database instead of on the file system. Later intentional programming (Charles Simonyi was pushing that) tried to also do things with this. And of course model driven development was a thing around the same time.

Refactorings (when done right) are syntax tree transformations that preserve things like referential integrity, etc. that ensure code does the same thing before and after applying a refactoring.

A rename becomes trivial if you are simply working on the symbol directly. For that to work with file based source trees, you need to parse the whole thing, keep track of where symbols are referred in files, rename the symbol and then update all the places in the source tree. That stuff becomes a lot easier when the code representation isn't a bunch of files but the syntax tree. The symbol just gets a different name. Anything that uses the symbol will still use the same symbol.

People like editing files of course and that has resulted in a lot of friction developing richer tools that don't store text but something that preserves more structure. The fact that we're still going on about formatting issues a quarter century later maybe shows that this is something to revisit. For many languages and editors, robust symbol renames are still somewhat science fiction. And that's just the most basic refactoring.

▲

zokier 4 days ago | parent [-]

Meh.

> That stuff becomes a lot easier when the code representation isn't a bunch of files but the syntax tree

You are just mixing abstraction layers here. That syntax tree still needs to be stored in file(s) somehow, and nothing prevents having syntax tree aware (or smarter) tooling operating on human readable files. Basically deserializing AST and parsing source code are the same thing. The storage format really isn't that significant factor here.

So what is needed is better tools rather than fiddling with storage format. Microsofts Roslyn is obvious example, but plenty of modern compilers are moving in the direction of exposing APIs to interact with the codebase.

	▲	jillesvangurp 3 days ago \| parent [-]
		> That syntax tree still needs to be stored in file(s) somehow Sure, but there are less flaky ways than spreading a syntax tree across files. Visual Age actually used a database for this back in the day. Smalltalk did similar things by storing code in an image file that contained both byte code and method definitions. You could export source code if you wanted. But wouldn't do that while developing typically. That's not an approach that caught on. But it has some advantages. What you are describing is what Eclipse did with Java. Eclipse was the successor to Visual Age. The Eclipse incremental compiler for Java updated an internal data structure for the IDE. It could do neat things as partial compilation to enable running tests even in the presence of some compile errors. It also was really fast. By the time you stopped typing, it would have already compiled your code. Running the tests was similarly fast. The problem of syncing a tree of source files with an AST is just a bit hard. Intellij never came close to this and has always had lots of trouble keeping its internal caches coherent. There's even a top level "invalidate caches" option in the File menu (still there, I checked. Right next to the Repair IDE option). They were off by 2-3 orders of magnitude. Seconds (at best) instead of milliseconds. I still miss Eclipse's speed every day I use Intellij. Some compilers are taking some steps to supporting more advanced IDEs. But there aren't a lot of those beyond what Jetbrains provides. VS Studio Code support varies between different languages. But mostly it's very limited on this front. The Rust compiler is one of those. Though I don't know the current state of that. Mostly it's not well known for its blazing performance (the compiler). I'm not sure if Jetbrains leverages many of those features in its Rust IDE (I'm not a Rust developer).