Does anyone else hate reading AI summaries of code? Code can be pithy, but at least its terse compared to prose. When you add how verbose LLMs can be, I often end up reading a paragraph to explain a few lines. Or the opposite happens where the summary skips important edge cases or criteria. "You're right, X also does Y. I missed that in my initial analysis," is much too common of a phrase.

I like the idea of using LLMs to transform code into something more readable, and vice versa. I am not sure if meandering paragraphs and linear lists are the best targets.

I wonder if a productive avenue might be "doing aspect-oriented programming in reverse": asking the LLM not to "summarize" the code per se, but rather to "clarify" it by transforming it into what a programming blog post would call a "toy example" of what the code is doing, by stripping out all the (non-semantic) error-handling, logging, metrics incrementing, etc — all the things that you might treat as their own "aspects" under AOP.

AI summaries are great for learning which parts of the codebase are load bearing.

The "language model" aspect shows IMO, at least for someone who grew up with the material of 90s and 2000s where we describe things more in mechanical, engineering terms. When gpt4 landed it was somehow amazing to see the output, but yeah nowadays I keep skimming through the explanation. It's like an intermediate dev who has nothing else to do but create long sentences to describe what could be simpler.

I don't like the default explanations. But I prompt for small code snippets with explanations of a problem and the solution. This is in the context of extending features, fixing bugs, reviewing new code etc. I still skim the code, but it's nice to have a somewhat thoughtful overview of the key points like database schema, API spec, algorithm or abstraction. It makes it easier to skim a large diff without feeling lost. It also quite often catches some weird choices that might slip through it not carefully reasoning about the code. For me it is hard to understand code I didn't write myself so I have landed on this workflow.

I've found that I don't mind getting a lot of text back from an LLM, when I was the one who prompted it. I can easily enough let my eyes flit around in the text and figure out what I need to, and I expect that putting all that text in the context window will help with the rest of the conversation.

It's when that text gets copied and pasted into a blog, or a PR, etc. that it really galls.

There is a reason Caveman (https://github.com/JuliusBrussee/caveman) is as popular as it is

I wish LLM summary of code was put aside the actual code, like the Jasmine documentation[0].

[0]: https://jasmine.github.io/tutorials/your_first_suite

I'm a bit confused by your use of 'pithy, but at least its terse'. Don't pithy and terse mostly mean similar things? Here it seems like you're contrasting them. Not trying to be pedantic, apologies if it comes off that way.

I use AI explainers a lot in my own projects and have skills that give outputs I like. I'll even pipe the output into NotebookLM to make a video sometimes. This is actually a decent way to grok complex subsystems (YMMV).

What I don't care for is the pretty-yet-verbose HTML outputs that I see from some teams. They can be way too long and it's often the case that the engineer who made it didn't read the whole thing.

Right, the value is not so clear of having multiple people wade through lengthy LLM responses. Part of the value that a human in the loop adds is figuring out what's correct and important in the response and passing on a more polished result.

Your AI summary is raw output, like uncooked mince meat.

Ask for a rewriting pass! Get it to edit for word count, remove redundancies, remove run-on sentences, and change sentence structure to use active voice.

You wouldn't ship your first draft - why let AI do it?

We need a new specification language for summaries. I think just a (pseudo?-)programming language’s API (modules, interfaces, method signatures), but more concise and descriptive than Java, maybe Haskell or OCaml minus quirks (…the academics were this close to an ideal syntax, maybe they should’ve gave up those languages’ LR parsers…)

Ask for a walkthrough of the highlights (with '...' omissions so that the most important parts are quickly visible) instead of a verbal summary.

You're absolutely right!

I hate fighting with the AI to get a good summary. Reading the good summary is great. Getting to it is the challenge.

First attempt: mountains of meta conversation and not answering the original question.

Second attempt: shorter and more concise, cuts out some of the nonsense, but terribly written.

Third attempt: goes off the rails, misunderstands what you are asking and tries pushing the work onto you.

Fourth attempt: finally something that's concise, reasonably well written, to the point and passable.

Yes this is why I ask the agent to summarize the code as a comic of a lizard explaining it to a cat.

[dead]