Thank you for your response!

I agree that spending time on inference or compute every time for the same LLM task is wasteful and the results would be less desirable.

But I don't think the thought experiment should end with that. We can continue to engineer and problem solve the shortcomings of the approach, IMHO.

You provided a good example of an optimization - tool creation.

Trying to keep my mind maximially open - one could think of a "design time" performance at runtime - where the user interacting with the system is describing what they want the first time, and the system is assembling the tool (much like we do now with AI assisted coding, but perhaps without even seeing the code).

Once that piece of the system is working it is persisted so no more inference is required, as essentially code - a tool, that saves time. I am thinking of this as essentially memoizing a function body- i.e. generating and persisting the code.

There could even be some process overseeing the generated code/tool to make sure the quality meets some standard and providing automated iteration, testing, etc if needed.

A big problem is if the LLM never converges to the "right" solution on it's on (e.g. the right tool to generate the HTML from the SQL query, without any hallucination). But, I am willing to momentarily punt on that problem as being more to do with the determinism problem and the quality of the result. The issue isn't per se the non-deterministic results of an LLM anyway, it's the quality of the result fit for purpose for the use case.

I think it's difficult but possible to go further with the thought experiment. A system that "builds itself" at runtime, but persists what it builds, based on user interaction and prompting when the result is satisfactory...

I remember one of the first computer science things I learned- the program that could print out it's own source code. Even then we were believing that systems could build themselves and grow themselves.

So my ask would be to look beyond the initial challenge of the first time costs of generating the tool/code and solve that by persisting a suitable result.

What challenge or problem comes next in this idea?