This is so right. We training Whisper Large model on 20,000 audio samples specific to a domain and it ended up reducing the ASR by 5% while improving WER of the finetuned domain by 0.5%.

Instead we ended up with no finetuning. We give audio snippet to 2 AsR models, take 3 best transcriptions and ask the LLm to pick the best based on the context. That produced significantly higher accuracy in how an agent understands the users.

Deep Fusion is best, when words and phrase patterns in the domain are known. Deep Fusion means to hint the Whisper decoder about the next possible words using LLM-in-the-loop.

Can you go into more detail on this? I’ve been putting together the data pipeline for an ASR fine tune but your approach sounds more interesting.