Remix.run Logo
▲ spudlyo an hour ago

How does this compare to using a model like facebook/wav2vec2-base960h for scoring at a 20ms per frame, and then something like torchaudio.functional.forced_align to map those frames onto the transcript words?

I get pretty accurate results this way, along with some nudging heuristics based on the expected leading or trailing word sibilance.

For multilingual work, facebook/mms-1b-all (trained on ~1100 languages) works extremely well, but is significantly slower on my CPU.