For the fine-motor commands: or, the model can write the code to generate them on the fly. It seems to work, in my very limited experiment.

As for memory: my approach is to give the robot a python repl and, basically, a file system - the LLM can write modules, poke at the robot via interactive python, etc.

Basically, the LLM becomes a robot programmer, writing code in real-time.