There is an open PR for VLLM to do this via DefussionGemma

https://github.com/vllm-project/vllm/pull/57250