Software that tells AI-generated music apart from music played by people, by listening to how the whole song is put together. I rebuilt it from scratch from two research papers.
What this is
Play someone ten seconds of AI-generated music and they probably won't spot it. The models got good. But play them the whole song and something starts to feel off, even if they can't say what.
Here's what it is. Real music comes back to itself. A chorus returns, a verse answers the one before it, and by the end the song has a shape you could draw. A lot of generated music just keeps going. It sounds fine bar to bar and never quite adds up to anything. Two researchers, Yumin Kim and Seonghyeon Go, noticed you could build a detector out of exactly that, and wrote it up in the two papers linked above.
Their trick is to stop asking what the music sounds like and start asking how it's put together. Cut the song into chunks, compare every chunk with every other chunk, and look at the pattern that falls out. Songs that mean something leave a pattern. Songs that wander don't.
I built the whole thing again from scratch, working only from the papers, because reading a method and actually making it run are very different things. This page is my honest account of how that went: what they found, what I built, what I measured, and where the line between their work and mine sits. I've kept their numbers and my numbers firmly apart, because muddling the two would be the easiest way to make this look better than it is.
How it works
It works in two passes, the same way the papers lay it out. The first one listens to small pieces of the song. The second one steps back and looks at how those pieces fit together. The second pass is where the interesting part happens, but it can't work without the first.
A short clip goes into a model that has already learned a lot about music, and comes out as a list of numbers summarising what it sounds like. A small classifier then guesses: human or machine? On its own this is the easy, fool-able version. It's the raw material for the second pass. Of the four such models I tried, the one called MERT summarised music best.
The song is cut into chunks on the beat, four bars each, and every chunk is summarised by Pass 1. Then each chunk is compared against every other chunk to build a grid of what-sounds-like-what. Real music lights that grid up in repeating patterns, because it genuinely returns to earlier ideas. The final model reads both the sound itself and that grid of repetition, and weighs how much to trust each one.
What the original authors found
Everything in this section is the original authors' published results, not mine. I'm showing them because they're what my rebuild is aiming at.
The short version: methods that pay attention to the shape of a whole song beat the ones that just stare at a picture of the sound. The older approaches in the table below treat audio like an image and look for visual artefacts. They do well. The structure-aware methods do better, and by a margin that's hard to dismiss.
How to read the numbers. All of them run from 0 to 1, and higher is better. Accuracy is simply how often the detector was right. F1 balances two ways of being wrong: missing a fake, and crying wolf on a real song. AUC asks whether the detector ranks fakes above real tracks; 0.5 is a coin flip and 1.0 is perfect. Scores this close to 1.0 mean these detectors were nearly always right on the data they were tested on.
| Method | Accuracy | F1 | AUC |
|---|---|---|---|
| Baselines · spectrogram / vision | |||
| ViT | not reported | 0.89 | not reported |
| EfficientViT | not reported | 0.95 | not reported |
| ConvNeXt | not reported | 0.96 | not reported |
| SpecTTTra-αSONICS | not reported | 0.97 | not reported |
| Structure-aware · MERT encoder | |||
| Segment Transformer | 0.9992 | 0.9992 | 0.9999 |
| Fusion Segment Transformer | 0.9999 | 0.9999 | 0.9999 |
“Not reported” means exactly that: the paper those baselines come from only published an F1 score, so there is no accuracy or AUC figure to quote. I've left the gaps rather than fill them with guesses.
| Method | FakeMusicCaps · F1 | SONICS · F1 | AUC |
|---|---|---|---|
| Baselines | |||
| MobileNet | 0.968 | not reported | not reported |
| ResNet18 | 0.924 | not reported | not reported |
| Segment encoders | |||
| Wav2Vec 2.0 | 0.988 | 0.995 | 0.944 |
| Music2vec | 0.952 | 0.997 | 0.937 |
| MERT | 0.996 | 0.997 | 0.998 |
| FXencoder | 0.913 | 0.966 | 0.924 |
| Method (MERT) | Accuracy | F1 | AUC |
|---|---|---|---|
| MERT standalone | 0.9708 | 0.9859 | 0.9939 |
| Segment Transformer | 0.9858 | 0.9859 | 0.9992 |
| Fusion Segment Transformer | 0.9867 | 0.9868 | 0.9995 |
What I measured myself
This is my own result, on my own smaller test. I trained the first pass on a free cloud GPU and then tested it on music it had never heard: 120 real recordings and 120 tracks made by an AI generator called MusicGen, split so nothing from training leaked into the test.
It got 0.875 right, and scored 0.972 on AUC. What that tells you is that the thing I built genuinely learns and genuinely separates real music from generated music. What it does not tell you is that I've matched the papers. Their numbers come from enormous datasets I'd need special access and a much longer GPU run to use. Mine is the smaller, honest version: proof the machine works, not proof it wins.
What I built
Every piece of the system described in the two papers, written from scratch and covered by 56 automated tests that check it does what it should:
What the detector actually sees
This is the heart of the idea, and you can see it with your own eyes. Each square below is one song, cut into chunks. Every chunk is compared with every other chunk, and the result is painted as a grid: dark where two chunks sound alike, pale where they don't. The line down the middle is just every chunk matching itself.
Look at the difference. Brahms is full of dark patches away from that centre line, because the piece keeps coming back to ideas it played earlier. The noise on the right has none of that: it's a scatter, because nothing in it ever returns. That is the whole signal the detector is built on. Both grids were computed by my code, on the real audio.
The same test, on every clip
Rather than show you two flattering examples, here is every clip I ran through it. The top row is music that genuinely holds together: real recordings, plus one deliberately repetitive synthetic loop. The bottom row is audio built to wander, ending with pure noise. The percentage under each is my detector's answer to one question: how much structure is in this?
Nothing here is hand-picked or hand-tuned, and the percentages are recomputed straight from the code by a script, so this page can't quietly disagree with what my software actually does.