One week ago I was on the hunt for an open source model that can do diatization and I had to literally give up because I could not find any easy to use setup.

I don't know if that will change, but right now only the Voxtral Mini Transcribe V2 supports diarization and it's not open-weight. The Voxtral Realtime model doesn't support diarization, but is open-weight.

WhisperX ?