| ▲ | Show HN: An API for word timestamps from speech, 75% within 20ms(quebec.yarn.so) | |||||||
| 68 points by jasperstory 3 days ago | 10 comments | ||||||||
Hey HN, we're Yarn (W24) and we make a video editing app for startups, and we built Quebec (https://quebec.yarn.so), an API that gives you accurate word timestamps from speech. We needed it because cutting a word even 50ms off from its true timing can ruin talking head footage (examples in the write up). Accurate word timestamps usually come from a forced aligner, which takes the speech audio and a transcript and works out exactly when each word starts and ends. We weren't satisfied with the accuracy of Montreal Forced Aligner (MFA) or new models like ElevenLabs Cue (https://elevenlabs.io/blog/cue-aligner). Giving LLMs access to a waveform/spectrogram didn't help much either. So we built our own. There are two particularly interesting parts: - We wrote our own decoder around MFA's acoustic model that only lets each word land within 200ms of the rough timings. That made search small enough to solve exactly, so we don't need to use MFA's beam search. - We added a small CNN, built on the first two layers of Whisper Tiny, that nudges each word boundary by up to 50ms. We couldn't find enough hand-labelled speech that allowed for commercial use, so we had Gemini label our own (17k word boundaries). It took a couple of hours and cost under $200. A net trained on Gemini's labels scored the same as one trained on human labels (73.9% vs. 74%). One way to measure accuracy is % of word boundaries within 20ms of hand labels. Here are the results on Buckeye (hand-labelled American speech that we didn't train on): - Quebec 75% - MFA 61% - ElevenLabs Cue 30% If you want to try it, it's free until tomorrow night. You can point your agent at our docs at https://quebec.yarn.so, and send audio on its own or with your own transcript. After that, it costs $0.32 per hour with no signup (minimum $10 top-up). We serve it at cost. Caveats: only tested in English and slower than ElevenLabs. Let us know if Quebec works well for you, and send us any samples that work badly (you can email support at yarn dot so). Also let us know if you've built or found a better aligner. In theory we would just switch to a better aligner, although we would probably just push a bit harder to make ours better. Thanks! Jasper | ||||||||
| ▲ | spudlyo 21 minutes ago | parent | next [-] | |||||||
How does this compare to using a model like facebook/wav2vec2-base960h for scoring at a 20ms per frame, and then something like torchaudio.functional.forced_align to map those frames onto the transcript words? I get pretty accurate results this way, along with some nudging heuristics based on the expected leading or trailing word sibilance. For multilingual work, facebook/mms-1b-all (trained on ~1100 languages) works extremely well, but is significantly slower on my CPU. | ||||||||
| ▲ | niwrad an hour ago | parent | prev | next [-] | |||||||
This is great - I run an AI Video startup. We've been testing different solutions around this recently. How's the performance across different languages? | ||||||||
| ▲ | totetsu 2 days ago | parent | prev | next [-] | |||||||
Neat. I was just looking for something like this for auto gating etc, for mixing podcast audio. Also phoneme level accuracy from something like this would be useful for realtime lipsync animation, and tools support pronunciation learning. | ||||||||
| ||||||||
| ▲ | HOWLD 4 hours ago | parent | prev | next [-] | |||||||
How does "free until tomorrow night" work? (or, on the page "free until Monday") - Is it just the $5 credit if signing up with a company email? | ||||||||
| ▲ | demibabs 4 hours ago | parent | prev | next [-] | |||||||
Cool! Does this work well with singing/music? If so this would be great for a project I’m doing. | ||||||||
| ▲ | ggm 4 hours ago | parent | prev | next [-] | |||||||
Um and err and so and but but but editing might be good with this. well gooder. It's hard when the umbut runs into the word. | ||||||||
| ▲ | londons_explore 2 days ago | parent | prev | next [-] | |||||||
The example use case really feels like making fake news and misleading edits easier... | ||||||||
| ||||||||
| ▲ | imguienjoyer30 2 days ago | parent | prev [-] | |||||||
whisper x | ||||||||