Pre-processing with a vocal extraction model (bs-rofomer or similar) helps a lot with the hallucinations, especially with poor quality sources.

I'm working with fairly "clean" audio (voice only) and still see ridiculous hallucinations.