Sounds like it might not have your precise location in context?

which is more or less the crux of the problem android auto's gemini implementation. It sometimes does not even know that it is in a car. The ghome implementation is also like this.