| ▲ | Our New Sam Audio Model Transforms Audio Editing(about.fb.com) |
| 128 points by ushakov 7 days ago | 47 comments |
| |
|
| ▲ | keepamovin 42 minutes ago | parent | next [-] |
| FB has been a pioneer in voice and audio, somehow. A couple of years ago FB-Research had a little repo on GitHub that was the best noise-removal / voice-isolation out there. I wanted to use it in Wisprnote and politely emailed the authors. Never heard back (that's okay), but I was so impressed with the perceptual quality and "wind removal" (so hard). |
|
| ▲ | yunwal 11 hours ago | parent | prev | next [-] |
| This is hilariously bad with music. Like I can type in the most basic thing like "string instruments" which should theoretically be super easy to isolate. You can generally one-shot this using spectral analysis libraries. And it just totally fails. |
| |
| ▲ | photon_garden 10 hours ago | parent | next [-] | | I had the same experience. It did okay at isolating vocals but everything else it failed or half-succeeded at. | | |
| ▲ | embedding-shape 2 hours ago | parent [-] | | Like most models released for publicity rather than usefulness, they'll do great at benchmarks and single specific use cases, but no one seem to be able to release actually generalized models today. |
| |
| ▲ | duped 10 hours ago | parent | prev [-] | | what in theory makes those "super easy" to isolate? Humans are terrible at this to begin with, it takes years to train one of them to do it mildly well. Computers are even worse - blind source separation and the cocktail party problem have been the white whale of audio DSP for decades (and only very recently did tools become passable). | | |
| ▲ | yunwal 10 hours ago | parent [-] | | The fact that you can do it with spectral analysis libraries, no LLM required. This is much easier than source separation. It would be different if I were asking to isolate a violin from a viola or another violin, you’d have to get much more specific about the timbre of each instrument and potentially understand what each instruments part was. But a vibration made from a string makes a very unique wave that is easy to pick out in a file. | | |
| ▲ | duped 10 hours ago | parent [-] | | Are you making this up? What spectral analysis libraries or tools? String instruments create similar harmonic series to horns, winds, and voice (because everything is a string in some dimension) and the major differences are in the spectral envelope, something that STFT tools are just ok at approximating because of the time/frequency tradeoff (aka: the uncertainty principle). This is a very hard problem "in theory" to me, and I'm just above casually versed in it. | | |
| ▲ | 613style 9 hours ago | parent | next [-] | | He's not making it up and there's no reason for that tone. Strings are more straightforward to isolate compared to vocals/horns/etc because they produce a near-perfect harmonic series in parallel lines in a spectrogram. The time/frequency tradeoff exists, but it's less of a problem for strings because of their slow attack. You can look up HPSS and python libraries like Essentia and Librosa. | | |
| ▲ | mrob 5 hours ago | parent | next [-] | | All wind instruments and all bowed string instruments produce a perfect harmonic series while emitting a steady tone. The most important difference between timbres of different instruments is in the attack, where inharmonic tones are also generated. Several old synths used this principle to greatly increase realism, by adding brief samples of attack transients to traditional subtractive synthesis, e.g.: https://en.wikipedia.org/wiki/Linear_arithmetic_synthesis | |
| ▲ | IndySun 8 hours ago | parent | prev [-] | | Hmmm... was 'tone' a pun? Why mention a strings 'slow attack' as less of a problem? No isolation software considers this an easy route. Vocals are more effectively isolated by virtue of the fact they are unique sounding. Strings (and other sounds) are the similar in some ways but far more generic. All software out there indicates this, including the examples mentioned. |
| |
| ▲ | jb1991 8 hours ago | parent | prev | next [-] | | If you look at the actual harmonics of a string and of horn, you will see how wrong you are. There is a reason why they sound different to the ear. It’s because of this that you can have a relatively inexpensive synthesizer (not sample or PCM based) that does a crude job of mimicking these different instruments by just changing the harmonics. | | |
| ▲ | mrob 5 hours ago | parent [-] | | There is one important difference between the harmonics of string and wind instruments: it's possible to build a wind instrument that suppresses (although not entirely eliminates) the even harmonics, e.g. a stopped organ pipe. If it sounds like a filtered square wave it's definitely a wind instrument. But if it sounds like a filtered sawtooth wave it could be either. |
| |
| ▲ | dleeftink 9 hours ago | parent | prev [-] | | I might misremember, but iZotope RX and Melodyne were pretty useful in this regard. |
|
|
|
|
|
| ▲ | ks2048 11 hours ago | parent | prev | next [-] |
| I recently discovered Audacity includes plug-ins for audio separation that work great (e.g. split into vocals track and instruments track). The model it uses also originated at Facebook (demucs). |
| |
| ▲ | embedding-shape 5 hours ago | parent | next [-] | | > for audio separation that work great What did you compare it to? Ableton recently launched a audio separation feature too, and probably the highest ROI on simple/useful/accurate so far I've tried, other solutions been lacking in one of the points before. | |
| ▲ | tantalor 11 hours ago | parent | prev | next [-] | | Is "demucs" a pun on demux (demultiplexer)? | | | |
| ▲ | TylerE 6 hours ago | parent | prev [-] | | Audacity is very very very far from state of the art in that respect. | | |
| ▲ | vhcr 6 hours ago | parent | next [-] | | This new SAM model actually competes against SOTA models. https://www.reddit.com/r/LocalLLaMA/comments/1pp9w31/ama_wit... | | |
| ▲ | embedding-shape 5 hours ago | parent [-] | | Their answer: > If you are interested in how well we do compared to demucs in particular, we can use the MUSDB18 dataset since that is the domain that demucs is trained to work well on. There our net win rate against demucs is ~17%, meaning we do perform better on the MUSDB18 test set. There are actually stronger competitors on both this domain and our "in-the-wild" instrument stem separation domain that we built for SAM Audio Bench, but we either match or beat all of the ones we tested (AudioShake, LalalAI, MoisesAI, etc.) So ~20% better than demucs, better than the ones they tested, but the acknowledge there are better models out there even today. So not sure "competes against SOTA models" is right, but "getting close to compete against SOTA models" might be more accurate. |
| |
| ▲ | wellthisisgreat 6 hours ago | parent | prev [-] | | What’s a good alternative ? | | |
| ▲ | embedding-shape 2 hours ago | parent | next [-] | | If you're already in the Ableton ecosystem, their newly released stem separation is actually very good, at least for the small amount of testing I've done so far. Much better than demucs, which shouldn't come as a surprise I suppose. | |
| ▲ | 5-0 5 hours ago | parent | prev [-] | | I suppose that depends on the use case. For mash-ups specifically, using yt-dlp to download music and split into stems with Demucs, using the UVR frontend, before importing into a DAW is effortless. The catch is that you can't expect to get OK-ish separation on anything other than vocals and "other", which really isn't a problem for mash-ups. https://github.com/Anjok07/ultimatevocalremovergui |
|
|
|
|
| ▲ | Oras an hour ago | parent | prev | next [-] |
| To try: https://aidemos.meta.com/segment-anything/editor/segment-aud... Github: https://github.com/facebookresearch/sam-audio I quite like adding effects such as making the isolated speech studio-quality or broadcast-ready. |
|
| ▲ | AkshatJ27 4 hours ago | parent | prev | next [-] |
| You can try it out in the playground: https://aidemos.meta.com/segment-anything/gallery/
There seem to be many more fun little demos by meta here like automatic video masking, making 3d models from 2d images, etc. |
|
| ▲ | samuell 4 hours ago | parent | prev | next [-] |
| I tried this to try to extract some speech from an audio track with heavy noise from wind (filmed out on a windy sea shore without mic windscreen), and the result unfortunately was less intelligible than the original. I got much better results, though still not perfect, with the voice isolator in ElevenLabs. |
|
| ▲ | yjftsjthsd-h 12 hours ago | parent | prev | next [-] |
| > Visual prompting: Click on the person or object in the video that’s making a sound to isolate their audio. How does that work? Correlating sound with movement? |
| |
| ▲ | janalsncm 10 hours ago | parent | next [-] | | If it’s anything like the original SAM, thousands of hours of annotator time. If I had to do it synthetically, take single subjects with a single sound and combine them together. Then train a model to separate them again. | |
| ▲ | yodon 12 hours ago | parent | prev [-] | | Think about it conceptually: Could you watch a music video and say "that's the snare drum, that's the lead singer, keyboard, bass, that's the truck that's making the engine noise, that's the crowd that's cheering, oh and that's a jackhammer in the background"? So can AI. Could you point out who is lead guitar and who is rhythm guitar? So can AI. | | |
| ▲ | scarecrowbob 10 hours ago | parent [-] | | I mean, sometimes I -mixing- a show and I couldn't tell you where a specific sound is coming from.... | | |
| ▲ | yodon 9 hours ago | parent [-] | | > sometimes I -mixing- a show and I couldn't tell you where a specific sound is coming from And in those situations it won't work. Is any of this really a surprise? |
|
|
|
|
| ▲ | throwaw12 6 hours ago | parent | prev | next [-] |
| This is super cool. Of course, it is possible to separate instrument sounds using specialized tools, but can't wait to see how people use this model for bunch of other use cases, where its not trivial to use those specialized tools: * remove background noise of tech products, but keep the nature * isolate the voice of a single person and feed into STT model to improve accuracy * isolating sound of events in games and many more |
|
| ▲ | teeray 11 hours ago | parent | prev | next [-] |
| I wonder if this would be nice for hearing aid users for reducing the background restaurant babble that overwhelms the people you want to hear. |
|
| ▲ | ajcp 12 hours ago | parent | prev | next [-] |
| Given TikToks insane creator adoption rate is Meta developing these models to build out a content creation platform to compete? |
| |
| ▲ | mgraczyk 11 hours ago | parent [-] | | I doubt it, although it's possible these models will be used for creator tools, I believe the main idea is to use them for data labeling. At the time the first SAM was created, Meta was already spending over 2B/year on human labelers. Surely that number is higher now and research like this can dramatically increase data labeling volume | | |
| ▲ | embedding-shape 2 hours ago | parent [-] | | > I doubt it, although it's possible these models will be used for creator tools, I believe the main idea is to use them for data labeling. How is creating 3D objects and characters (and something resembling bones/armature but isn't) supposed to help with data labeling? As synthetic data for training other models, maybe, but seems like this new release is aimed at improving their own tooling for content creators, hard to deny this considering their demos. For the original SAM releases, I agree, that was probably the purpose. But these new ones that generate stuff and do effects and what not, clearly go beyond that initial scope. |
|
|
|
| ▲ | ac2u 12 hours ago | parent | prev | next [-] |
| I wonder if the segmentation would work with a video of a ventriloquist and a dummy? |
|
| ▲ | 7734128 6 hours ago | parent | prev | next [-] |
| Finally a way to perhaps remove laugh tracks in the near future. |
| |
| ▲ | sefrost 6 hours ago | parent [-] | | There are examples on YouTube of laughter tracks being removed and there are lots of awkward pauses, so I think you'd need to edit the video to cut the pauses out entirely. - https://www.youtube.com/watch?v=23M3eKn1FN0 - https://www.youtube.com/watch?v=DgKgXehYnnw | | |
| ▲ | embedding-shape 5 hours ago | parent [-] | | Cutting the pauses will change the beats and rhythm of the scene, so you probably need to edit some of the voice lines and actual scenes too then. In the end, if you're not interested in the original performance and work, you might as well read the script instead and imagine it however you want, read it at the pace you want and so on. | | |
|
|
|
| ▲ | theflyestpilot an hour ago | parent | prev | next [-] |
| sample anything model? |
|
| ▲ | m3kw9 12 hours ago | parent | prev [-] |
| Can I create a continuous “who farted” detector? Would be great at parties |
| |
