The problem is, the robot has to know what I want it to do without me having to dictate it.

That's the beauty of human interaction, it can't be massively truncated down to just even finger pointing.