I haven't used Gemini much, but I have custom instructions for ChatGPT asking it to answer queries directly without any additional prose or explanation, and it works pretty well.

It works to cut down on verbosity, but verbosity is also how it thinks. You could be lobotomizing your responses