"Make the button red" probably doesn't need an LLM at all.
One tends to use LLMs for everything in practice. It‘s inconvenient to switch mode of operation