I've been working on inverting the control theory for the agent loop. Instead of the user initiating everything, the agent runs automatically in the background and calls the user for feedback as part of tool use. The end game for me is to get rid of the chat interface altogether and move back toward async email and other messaging channels. The chatbot UI as a means of driving the business always felt like a temporary stepping stone / clever demo.

I think there are 10-100x productivity gains lurking in here. It is very expensive for a human to reserialize their mental state into a prompt each time a task needs working on. An agent can do this ~instantly and with high frequency 24/7. The higher the rate of evaluation the less change has to be dealt with between any two iterations. So, the likelihood that a given iteration needs human help goes down as you increase the rate of evaluation per unit of wall clock time. Tighter and faster control loops tend to require less severe corrective measures than slow and sloppy ones.

This is the most plausible reason for so many tokens in the future. I can actually see a million tokens per second making sense. I have a pretty good idea how I'd approach this if I actually had access to this kind of infrastructure. 1Mtok/s is baby tier in terms of raw information theory. The politics of employing a system like this are far more terrifying to me than any technological aspects. Humans really like having control over things, even when that control is pure downside for the business.