interesting approach. But training ctc head on top of gemma's audio encoder will most likely produce better results and faster inference

That's close to what this is: the reader is a CTC head on the audio encoder, but its output words are EmbeddingGemma's own text embeddings for ~229k words, instead of a separately learned vocabulary. I also tried a plain letter-level CTC head on the encoder. It did a bit better on clean audiobooks (7.0 vs 8.2 WER at the time) but much worse on harder audio like a lecture (16.5 vs 10.8), mostly from misspellings.