Diarization is done separately to ASR anyway (it's usually a separate run, after the ASR).