The right tool would likely be some conventional autopilot software; if you want AI cred you could train a Neural Network which maps some kind of path to the control features of the drone. LLMs are language models -- good for language, but not good for spacial reasoning or navigation or many of the other things you need to pilot a drone.

So you are suggesting building a full featured package that is nontrivial compared to this fun excitement?

Vision models do a pretty decent job with spatial reasoning. It’s not there yet but you’re dismissing some interesting work going on.