Our newest TPUs: 8tx and 8i Max, built for the agentic era

pierreb-aiva a day ago | parent | next [-]

> Because AI agents are stateful, they frequently pause for seconds or hours while waiting on a database lookup, an external software tool, or human feedback. To make our accelerators more power-efficient during those pauses, our new Agent Compute Engine software automatically suspends idle agent sessions and shifts working memory off the chip in milliseconds, cutting operational infrastructure costs

Nice. I wonder how they manage to move kv cache from working memory to disk so quickly to enable speed ups when pauses are a few seconds at most

	▲	jononor 20 hours ago \| parent [-]
		Off chip does not have to mean to disk? I would assume a staged approach. First offload to CPU RAM, and then to disk (if not used within a few seconds), and then to network storage (so it can maybe be resumed by other machines). And some request routing that makes use of cache affinity based on session/user.

▲

gniv a day ago | parent | prev | next [-]

Seems like a big deal, but there's nothing about availability. They might say more during the broadcast later today.

▲

xnx a day ago | parent | prev [-]

More bad news for Nvidia today.