General models can run robots. They are very bad at it. It is much more efficient to have a stack of models with 'dumber' but faster models taking care of lower level functions and higher level ones that are much slower taking care of high level planning. Consciousness isn't super fast, but it is very smart. Your 'lizard brain' (to use an overstretched metaphor) and reflexive system are relatively stupid but far faster and condense a whole raft of inputs to something your consciousness can deal with. For robots it isn't all that different. Sensors and actuators require millisecond level responses, that's something you're not going to get out of an LLM on the other side of a communications link.

▲

mrtksn 7 hours ago | parent | next [-]

What models actually run? Do they generate the PWM signals/their widths to drive a servo or something? Or is it more like "joystick commands" generator?

▲

jacquesm 6 hours ago | parent [-]

More like high level motion controller, so 'servo 23 1700 us' or for better grade servos 'servo 23 pos 145400', possibly concurrent with motion commands to other actuators and possibly synchronized, possibly using other kinds of protocols (G-codes).

Motion controllers are pretty impressive these days. PWM generation is too time critical to leave to high level processors, usually this is done in hardware if you want it to be guaranteed glitch free but there are also 'bit banging' servo controllers out there.

	▲	mrtksn 5 hours ago \| parent [-]
		Interesting, so how does the inference work here? are they aiming for a fixed length intervals like 10 times/second generating the positions or is it more like reactive, i.e when the request to hold a cup from a higher level comes then this is treated as a promt, the sensors are the context and when the inference finished the pulses are sent according to the output?

▲

gopalv 7 hours ago | parent | prev | next [-]

> Sensors and actuators require millisecond level responses, that's something you're not going to get out of an LLM on the other side of a communications link.

We've gotten to slightly a frame behind for the models I'm using to play video games - 36ms to get the next action.

The idea is probably not have a frontier model doing this, but to make a frontier model build these out of its own brain (well, directly edit the weights instead of RL tuning).

My team did call these sort of slices of a main brain a "Reflex agent" in similar fashion, since they do not reason only respond.

	▲	dwattttt 6 hours ago \| parent [-]
		The faster you are, the more systems you're able to control directly. At ~30ms per "choice"/output, you can choose what lower level motors/servos/systems drive towards, but you can't generate the signals that drive them. You're looking at 1kHz and maybe higher there.

▲

baq 7 hours ago | parent | prev | next [-]

> General models can run robots. They are very bad at it.

Reminder frontier labs are 6 months ahead of public models internally, if not more

	▲	jacquesm 7 hours ago \| parent [-]
		That does not make them orders of magnitude faster.

▲

raincole 4 hours ago | parent | prev | next [-]

General models can {insert something general models will be very good at}. They are very bad at it.

Every single time.

▲

cobychapple 4 hours ago | parent | prev [-]

nothing says reflexes like getting a 429 rate limit halfway down the stairs.