Just adding to your suggestion.

For those new to llama.cpp, the llama-server component running in "router mode" is fantastic; you place your common configuration in a defaults section of an ini file, and then model specific configuration in separate sections of the ini file. You can switch between models, on the fly, using the API, and the built-in web front-end for llama-server supports model switching as a drop down menu. Llama-server can just always be running, in the background (you can set an idle time for it to automatically unload weights to free vram/ram, and a maximum simultaneously running model count which will evict models on an LRU basis before loading new ones, if the count is exceeded).

llama-bench is another great tool included in llama.cpp, where you can provide ranges of options to test, and just let it run through all the variations that you requested, then take the best performing and add those settings to the ini file above.

There are a lot of optional tunables that can really improve performance with llama-server. E.g., typical average tok/sec generation speed improved over 7X for Gemma4 31B, on my laptop, with tuned batch-size / ubatch-size and enabling both spec-type draft-mtp and ngram-mod (with some custom settings on both). I.e., even though you don't need to do this for llama-server to be useful, it can be worth your while to spend a little time optimizing settings for your favorite models especially if your hardware is marginal.