Source Separation

Once mixed, there's no boundary left

In the pond, each species still swims separately. Sound is different. When several sounds share the same space, the air's vibrations simply add together into one wave. Inside a recorded file, the vocal and the instrumental aren't sitting side by side — only their sum survives.

In numbers, it's like being told the sum of several numbers and asked to guess the originals. There's no single correct answer. That makes source separation a job of guessing, not measuring.

Guessing is possible because each sound has a shape of its own. It's not random numbers added together — knowing that what was added looks like a voice, or looks like drums, opens room to pull them apart.

Splitting the shares on a sound picture

The actual work usually happens on a sound picture rather than cutting the waveform directly. Spread out a picture split by time and pitch, and for every cell, assign a ratio saying which part's share this sound is.

Apply this ratio map and one part survives while the rest is suppressed. Make one ratio map per part and each part's sound comes out separately. Adding all the parts back together is usually tuned to reconstruct the original sound.

It matters that no cell gets assigned to just one owner. It's common for two sounds to sit in the same cell together. Splitting it something like half and half keeps the separated sound from cutting in and out.

What gets used as a clue

The biggest clue is the pitch pattern. A voice draws regular stripes and drums draw a short, broad smear, so they look different in the picture. How long a sound sustains differs too — a string instrument rings on, a percussion hit vanishes in an instant.

How something moves over time is a clue too. A vocal rises and falls with the lyrics and breath, while an instrumental keeps the beat. This difference in flow becomes a hint for splitting apart stretches where they overlap.

If a recording is split left and right, direction is a clue as well — a sound louder on the left can be told apart from one centered in the middle. Even so, today's methods do a decent job splitting things apart even from a recording already mixed down to one channel.

It never comes apart cleanly

Listen to a separated vocal alone and a faint trace of the instrumental lingers behind it. The other side keeps a trace of the vocal's reverb. Sometimes the sound warbles as if underwater, or the edges rustle.

Some cases are especially hard: two very similar voices speaking at once, a recording made in a heavily reverberant room, or an instrument or sound the system never trained on. Sound that was already smeared together and stored that way in the original recording is hard to restore too — information that's already lost doesn't come back just because you split things apart.

Where it's used

The most familiar use is keeping just the instrumental of a song or pulling out just the vocal. It's also used in restoration work — splitting an old recording apart by instrument and remixing it — and in film, boosting dialogue while pulling background sound down.

On the speech side, it's used to keep just one person's voice in a meeting recording, or to split a voice from noise on a call made somewhere loud. Running transcription after separating noticeably improves accuracy, which is why it's often attached as a step right before it.