Spectrogram

The waveform alone is hard to read

Open a recording and the wavy shape you see plots how hard the air was shaking, moment by moment. Whether the sound is loud or soft, and when it starts and stops, shows up right there.

But the thing you actually want to know is hidden. Whether it's a low hum or a high screech, a human voice or a door closing, is buried in how tightly packed the wiggles are. Layer several sounds together and the wiggles merge into one line, making it even harder to tell apart.

That's why most work with sound doesn't look at the raw waveform at all — it converts to a picture split out by pitch first.

One card per short slice

Measure the pitch content of a whole sound at once and you only learn which notes were present across the entire piece — when they happened is gone. So it's measured in very short slices instead. One slice runs about a few dozen milliseconds — short enough that the sound barely changes within it.

Measuring the strength of every band from low to high within one slice gives a single vertical strip. Slide the slice a little to the side and repeat, and stitching the resulting strips together in time order completes the picture. The slices overlap as they slide, so the seams don't come out choppy.

The scale of the pitch bands is usually tuned to match human hearing. People notice small differences well at the low end and poorly at the high end, so the bands are split fine at the bottom and coarse at the top.

How to read the picture

Left to right is time, top to bottom is pitch, and darkness is loudness. A voice shows up in this picture as horizontal stripes running side by side. The lowest stripe is the base pitch the vocal cords vibrate at, and evenly spaced stripes stack up above it. Raise the note being sung, and this whole band of stripes shifts upward.

Consonant sounds like final stops or a hissing "s" show up not as stripes but as a blurred smear spread across the upper range. A clap or a door closing spreads short and wide like a vertical line. A quiet stretch is just blank space. Even the same word said by different people produces different stripe spacing and smear shapes, which is why this pattern acts like a voiceprint.

Picking a sharpness trades something away

Cut slices shorter and it becomes clearer exactly when something happened. But there's less wave packed into each slice, so the pitch bands blur together. Cut them longer and pitch splits out cleanly, but the moment a sound changed gets smeared over.

You can't make both sharp at the same time. You have to pick based on what you're trying to see. Transcribing speech uses short slices, since speech changes fast; studying musical chords uses somewhat longer ones.

This is where the work happens

Once sound is converted to a spectrogram, a sound problem becomes a picture problem. There's already a huge stock of methods for recognizing patterns in pictures, and those can be brought over as-is.

Transcription reads the pattern each sound makes in this picture and turns it into letters. Pitch detection means pinpointing one reference stripe among the bands. Source separation means splitting the overlapping patterns apart by instrument. Sound generation runs the other way — it draws this picture first, then converts it into a waveform.