What Makes Audio Stem Separation Technology Effective?
Audio stem separation explained: how masking and waveform models work, why training data and SDR matter, and how to test a separator on your own mix.
Audio stem separation is effective when a model has learned, from songs with known stems, how each instrument sounds alone, and can pull that part out of your mix with little bleed and few artifacts. The architecture, the training data, the quality of your file and testing on your own track decide how close it gets.
Below: how the main model families work, what training data and benchmark scores mean, what failure sounds like, and a checklist for your own song. Sources were read on 19 November 2026. For the tools themselves, see our comparison of audio editors with the best AI stem separation.
How audio stem separation works: masks, waveforms and hybrids
A separator estimates each part of a mix so the estimates add back up to it. Three model families do this.
Spectrogram masking. The model reads a spectrogram, a picture of energy over time and frequency, and predicts a mask: a value from 0 to 1 at every point, saying how much belongs to the target. In their 2017 U-Net paper, Andreas Jansson and colleagues at City, University of London and Spotify multiplied the mix's magnitude spectrogram by the mask and reused the mix's original phase to rebuild audio. It works, and it explains a classic artifact: where a vocal and a guitar share the same time-frequency points, the mask can only hand out fractions, and the reused phase still carries both.
Waveform models. These predict each stem's waveform directly. Demucs (Défossez and colleagues, Facebook AI Research, 2019) pairs a U-Net with a bidirectional LSTM. It averaged 6.3 dB SDR on MUSDB, 6.8 dB with 150 extra training songs, and listeners rated it more natural. The authors noted: "However, it suffers from some bleeding, especially between the vocals and other source."
Hybrids and band-split Transformers. Hybrid Demucs (2021) works in both domains and won Sony's Music Demixing Challenge 2021. HT Demucs (Rouard, Massa and Défossez, November 2022) adds a Transformer that attends within and across the two domains, reaching 9.20 dB SDR with 800 extra songs, though it did poorly when trained on MUSDB alone. Band-split models cut the spectrogram into frequency bands and model each band and the links between them, an idea introduced by Band-split RNN (Luo and Yu, 2022). BS-RoFormer (Lu and colleagues, September 2023) uses Transformers with rotary position embeddings: a smaller version scored 9.80 dB on MUSDB18-HQ without extra data, and the full system ranked first in the music track of the Sound Demixing Challenge 2023. Mel-RoFormer (October 2023) uses overlapping mel-scale bands and beat it on vocals, drums and other.

Why training data matters as much as the architecture
Separation models learn by example: they split a mix and get corrected against the true stems, which are scarce.
The research standard is MUSDB18: per its maintainers at SigSep, 150 full-length tracks, about 10 hours, with drums, bass, vocals and other stems, split into 100 for training and 50 for testing. Four details matter when you read scores:
- Four stems only. Guitar, piano and synths all fall into "other", so a MUSDB score says nothing about splitting keys from guitars.
- Bandwidth. The standard version stores stems as AAC at 256 kbps, limited to 16 kHz; MUSDB18-HQ offers WAV up to 22 kHz.
- Imperfect targets. The errata list songs where other instruments bleed into the vocal stem.
- Narrow taste. The Music Demixing Challenge 2021 organizers noted a lean toward Western pop and few mixing engineers.
So the best results add data: 500 extra songs for BS-RoFormer, 800 for HT Demucs, and about 20,000 professionally produced tracks paired with their instrumental versions for Jansson's U-Net. Volume alone doesn't win, though. In the Demucs repository results table, Spleeter, trained with 25,000 extra songs, scores 5.9 dB, while Band-split RNN reaches 8.2 dB with none. Cleanliness counts too: in the 2023 challenge, training one baseline on mislabeled stems cut average quality by 1.42 dB, and on stems with bleed by 0.83 dB.
How separation quality is measured: SDR, SI-SDR and listening tests
The standard score is signal-to-distortion ratio (SDR): ten times the log of the true stem's energy divided by the error's energy, averaged over vocals, bass, drums and other. Each 3 dB roughly halves the error energy.
SDR has critics. In 2018, Jonathan Le Roux and colleagues argued it had been misused, with papers "often arguing that differences on the order of 0.1 dB proved the effectiveness of a method over others." They proposed scale-invariant SDR (SI-SDR), which ignores an estimate's overall level.
The best public snapshot is the Sound Demixing Challenge 2023 summary (August 2023, revised April 2024). On the leaderboard open to any training data, team SAMI-ByteDance averaged 9.97 dB: 11.36 on vocals, 11.15 on bass, 10.27 on drums and 7.08 on other. The hidden test set held 30 songs produced by Sony Music Entertainment Japan, and the best system beat the 2021 winner by more than 1.6 dB.
Seven assessors, including award-winning singers, songwriters, producers and sound engineers, also rated stems and residuals (the mix minus the stem, which exposes what a model failed to remove). Producers leaned toward one finalist and musician-educators toward another, though the panel was too small to be sure. Scores and ears don't always agree.
What it sounds like when separation fails
- Bleed. Hi-hats ticking under the vocal, or a ghost of the singer in "other". Shared energy went to the wrong stem.
- Watery or phasey tone. A swirling, underwater sound on sustained vocals and pads, from masks carving holes in shared frequencies while the reused phase warbles.
- Smeared transients. A snare that loses its crack, or a pick attack split between guitar and drums. Attacks spread across many frequencies at once, and the model must choose where each slice goes.
- Reverb tails in the wrong place. A dry, gated-sounding vocal while its reverb lingers in "other", or tails that stop dead at phrase ends. Reverb spreads over time and is often shared, so it gets split inconsistently.
- Level jumps and clipping. The Demucs README warns that artifacts can push a stem past full scale; its tool rescales stems to prevent clipping, which can change their relative levels.
Our guide to techniques for editing audio stems covers cleaning these up with gates, EQ and spectral repair.
Input factors that help or hurt audio stem separation
- Dense mixes. Distorted guitars, a pad and a vocal stacked in the same midrange give the model the most overlap to untangle.
- Heavy reverb and delay. Long washes blur each source over time and blend tails from several parts.
- Mono sources. Panning is a strong clue; a guitar hard left is easier to pull from a centred vocal. In a mono recording, only pitch, timbre and timing remain.
- Lossy files. A low-bitrate MP3 has already discarded high-frequency detail and can add swirl and pre-echo that the separator mistakes for music. Use a WAV or FLAC master.
- Sample rate. MUSDB18 is encoded at 44.1 kHz, and Demucs writes 44.1 kHz stereo stems. A 22 kHz file has no top octave to separate; a 96 kHz file usually gets resampled and gains little.
- Unusual instruments. A harmonium, sitar or choir isn't one of the four benchmark stems, so it often lands in "other" or gets split across stems.
A checklist for judging a stem separator on your own track
Test on one of your own songs, or a mix you have the rights to. If you still have the multitracks, compare each stem with the original part.
- Pick the hardest 30 seconds. The densest chorus, with reverb, backing vocals and the full kit.
- Feed it your best file. A WAV or FLAC master at its original sample rate.
- Compare at matched loudness. Run the section through two or three separators or settings and level-match.
- Solo every stem on headphones. Listen for hi-hats in the vocal, watery sustains, lost s and t consonants, and a dull snare.
- Listen to the residual. Flip one stem's polarity against the mix; nothing of that part should remain.
- Check the sum. Add all the stems together and flip the result against the mix. The quieter the remainder, the less was lost or invented.
- Test in context. Put the a cappella over a new beat or mute a stem in the full mix. Hidden artifacts often surface in a new arrangement.
Where Soundverse fits
Soundverse is an AI music platform that takes you from idea to audience, built for independent artists who release music. Stem Separator splits a song into 2 stems (vocals and backing), 6 stems (vocals, drums, bass, guitar, piano and other), or a maximum split with every part on its own and a MIDI file for each. The 2- and 6-stem splits are on the Free plan; the maximum split needs Creator or above.
It takes one song up to 50 MB as MP3, WAV, FLAC, AAC, OGG or M4A, returns 2- and 6-stem splits as WAV, FLAC or MP3, and the maximum split as WAV plus MIDI. Run it from AI Tools, a song's menu in your Library, or a clip in AI Studio, where stems land on their own tracks. It can only split what's already in your mix, so give it your best copy and run the checklist above.
Licensing is set by plan: the Free plan is for personal use, and the Creator plan comes with a royalty-free license.
From 1:28, this walkthrough shows Stem Separator splitting a song, from input to finished stems.
FAQ
Which is better for audio stem separation, spectrogram or waveform models? Neither wins outright. Waveform models avoid reusing the mix's phase, spectrogram models resolve frequency detail well, and the top-ranked systems in the 2021 and 2023 challenges were a hybrid and a band-split spectrogram Transformer. Judge the stems, not the method.
Why does my separated vocal sound watery? Usually masking: the model removes energy it assigns to other parts, and what's left of shared frequencies warbles. Dense midrange, heavy reverb and a low-bitrate source make it worse, so try a lossless file or another model.
Do more stems mean better separation? No. Each extra stem is one more split to get wrong, and public benchmarks score only four. For the basics, read how AI stem separation works; for no-cost tools, see free alternatives for isolating vocals.
Sign up for Soundverse free and run the checklist on one of your own mixes. More production guides on the Soundverse blog.