Perhaps tools trained into the model rather than exposed through prompting would mitigate the performance hit (but might affect model quality?).

This is where you start to fine-tune the weights, you can get pretty great results when it comes to specific tool calls with the right data.