How Does AI Detect and Preserve Audio Quality During Synthesis?

How AI keeps audio quality during synthesis: mel losses, GAN discriminators, codec bitrates, data filtering, FAD, ViSQOL, MUSHRA and quality gates.

AI preserves audio quality during synthesis mainly through how the model is built and trained: spectral and mel losses keep its output close to real recordings, adversarial discriminators learn to catch artifacts, and codec bitrate, sample rate and clean training data set the ceiling. Quality is then detected with objective metrics, listening tests and automatic checks on each output.

This is the engineering side: how models are built to keep quality and how they're checked. For what artifacts sound like in a mix and how to fix them, see our guide to the technical factors that affect AI track quality. Papers and standards cited were read on 19 November 2026.

Where AI detects audio quality: three checkpoints

  1. During training. Losses and discriminators score the model's output against real audio, and the model is updated to close the gap.
  2. During evaluation. Researchers compare versions with objective metrics and listening tests.
  3. After generation. Automatic checks can screen each output for silence, clipping, garbled words or low predicted quality.

Diagram of three checkpoints for AI audio quality: training losses and a discriminator, evaluation with metrics and listening tests, and an output gate that passes or regenerates each file

Reconstruction losses: multi-scale spectral and mel

Most music generators end in a decoder or vocoder that turns tokens, latents or spectrograms into a waveform; our tokenization explainer covers the tokens. The first defence is a reconstruction loss, a measure of how far the rebuilt audio is from the original.

Meta's EnCodec (2022) shows the standard recipe: an L1 loss on the waveform plus L1 and L2 losses on 64-band mel spectrograms at seven window sizes, from 32 to 2,048 samples. Short windows resolve transients and long ones resolve pitch and tone, so using several keeps the model from trading one for the other.

The mel loss carries a lot of weight. In the HiFi-GAN paper (Kakao Enterprise, NeurIPS 2020), it's weighted 45, against 2 for feature matching, and removing it dropped listeners' mean opinion score for the small model from 4.10 to 3.25 on a five-point scale. Real recordings scored 4.57. Vocoders are also the last stage of many singing synthesizers, as our explainer on how AI singing synthesis works describes.

Stereo needs its own term. DiffRhythm (2025) computes its multi-resolution STFT loss on the mid and side signals as well as left and right, with left and right weighted half as much, to keep the placement of sounds unambiguous.

Adversarial discriminators: the part of the model that detects artifacts

A spectral loss says which frequencies should be present, not whether the result sounds real. That is the discriminator's job: a second network trained to tell real audio from generated audio while the generator learns to fool it. During training, it is the AI that detects quality problems.

HiFi-GAN uses two kinds:

  • Multi-period discriminator. Five sub-discriminators each read every 2nd, 3rd, 5th, 7th or 11th sample, folding the waveform so periodic patterns line up. The periods are primes to avoid overlap.
  • Multi-scale discriminator. Three sub-discriminators judge the raw audio and versions smoothed by 2× and 4× average pooling, catching patterns at different resolutions.

The ablation shows how much the detector matters. Without the multi-period discriminator, the score fell from 4.10 to 2.28; without the multi-scale one, to 3.74.

EnCodec instead uses one multi-scale STFT discriminator: five networks reading spectrograms with windows from 2,048 down to 128 samples, with left and right processed separately. To stop any one term from dominating, its loss balancer sets each loss's weight as the fraction of the overall gradient it should represent. DiffRhythm quadrupled its discriminator's size to capture more high-frequency detail.

Codec bitrate and sample-rate choices

The codec or autoencoder sets a ceiling no generator can pass: sample rate fixes the highest possible frequency, and bitrate or frame rate how much detail survives.

  • One model, many bitrates. EnCodec varies how many residual codebooks it uses per batch, covering 1.5, 3, 6, 12 and 24 kbps at 24 kHz mono, and 3 to 24 kbps for 48 kHz stereo music. A dedicated discriminator for each bitrate improved quality.
  • Low rate, then upsample. The YuE team (2025) use a 16 kHz codec at 50 frames a second and keep 8 of its 12 codebook layers, since more brought no noticeable improvement, then a light vocoder upsamples to 44.1 kHz.
  • Named limits. The SongGen authors (2025) list their 16 kHz codec as a limitation and plan a renderer to upsample it.

Lower rates make long sequences affordable and leave less room for detail, so every system picks a point on that trade-off.

Training-data quality filtering

A model learns the noise, clipping and MP3 smear in its training data as faithfully as the music. Published pipelines filter hard:

  • SongGen drops clips with low vocal or accompaniment energy, clips whose vocals two Whisper models transcribe more than 20% differently, and clips whose captions score low on CLAP audio-text match. Its final fine-tune uses a stricter subset of 100,000 pairs.
  • DiffRhythm trained its decoder on 250,000 lossless files and fed it MP3-compressed input two-thirds of the time, so it learned to rebuild lossless-quality audio from lossy input.
  • Meta's Audiobox Aesthetics (2025) predicts four scores from 1 to 10: production quality, production complexity, content enjoyment and content usefulness. In its tests, deleting the lowest-scoring 25% or 50% of training data hurt alignment. Adding the predicted score to each training prompt, then asking for a high score at generation, beat filtering in listening tests and kept alignment.

Objective metrics: FAD, ViSQOL, CLAP, PESQ and STOI

Metrics split on one question: is there an original to compare against?

Full-reference metrics compare the output with the exact audio it should match, so they suit codecs and decoders. Google's ViSQOL, open-source as v3 since 2020, has speech and audio modes; EnCodec reports it alongside SI-SNR, and DiffRhythm scores its decoder with STOI, PESQ and mel cepstral distortion. PESQ and STOI were designed for speech, and singing differs in pitch, held vowels and vibrato, as our comparison of speech and singing synthesis explains.

Full-reference metrics break down for new music, which has no original. The ViSQOL authors say so directly: "These generative models typically do not lend themselves to being analyzed well by existing full reference speech quality metrics."

Reference-free metrics fill that gap:

  • FAD. Fréchet Audio Distance (Google, 2018) compares embedding statistics for a set of outputs with a large set of studio recordings. In its paper, FAD correlated with human ratings at 0.52, against 0.39 for signal-to-distortion ratio.
  • Per-song FAD. Microsoft researchers Azalea Gui and colleagues (2023) found FAD skewed by sample size, embedding choice and low-quality reference sets; many clips in the popular MusicCaps set are labelled low quality. Scoring each song separately looked useful for spotting outliers such as heavily distorted, noisy or non-musical samples.
  • CLAP score. It measures how well audio matches its text prompt, not sound quality, and the YuE team found it diverged from human judgments.

Loudness and peaks are plain measurement: ITU-R BS.1770-5 (November 2023) defines how to measure programme loudness and true-peak level.

Listening tests: MUSHRA and MOS

Metrics are proxies. The EnCodec authors put it plainly: "the evaluation of lossy audio codecs necessarily relies on human perception".

  • MUSHRA is standardised as ITU-R BS.1534-3 (October 2015). Listeners rate several versions of one excerpt side by side, including a hidden copy of the original and a deliberately degraded low anchor. EnCodec's raters scored five-second samples from 1 to 100, and anyone who often rated the hidden reference below 90 or the low anchor above 80 was removed.
  • MOS is simpler: each clip gets a score from 1 to 5. HiFi-GAN's numbers above are crowd-sourced MOS.

Automatic quality gates in a production pipeline

Papers describe the parts; companies rarely publish their exact gates. One built from published methods could run on every output:

  1. Signal checks. Flag silence, noise bursts, clipping, true-peak overs and loudness far from target. YuE's team tracked an "invalid probability", the chance that generation produces noise or silence segments, and found it varied with the codec.
  2. Predicted quality. Score the file with a no-reference predictor such as Audiobox Aesthetics, or compare it with a clean reference set the way per-song FAD does.
  3. Lyric check. Separate the vocal, run speech recognition and compare it with the lyrics. DiffRhythm scores phoneme errors rather than word errors, since a recogniser can hear the right sounds as different words, and YuE found Whisper unreliable on singing, so a failed check should mean review.
  4. Prompt match. Score text-audio match, knowing CLAP can disagree with listeners.
  5. Decide. Pass the file, regenerate it or flag it for a person, and recalibrate thresholds against listening tests.

Where Soundverse fits

Soundverse is an AI music platform that takes you from idea to audience, built for independent artists who release music. It publishes no sound-quality score for Soundverse v9, its newest song model, on Pro and Max. In October 2026 it posted a blind test on YouTube that pairs eight first-take Soundverse v9 songs with songs another AI music model made from the same prompts, loudness-matched and unedited. That is an informal listening test, not MUSHRA, so trust your own ears too.

For objective checks on your own files, Analysis One, free on every plan, shows gated LUFS, 4× oversampled true peak with an overs counter, a spectrogram and stereo correlation, so you can see how hot a generated song's peaks are before you master it.

The blind test plays eight Soundverse v9 songs against another model's songs made from the same prompts.

FAQ

What is a discriminator in AI audio? A second neural network trained to tell real recordings from generated ones. The generator improves by trying to fool it, so the discriminator works as a built-in artifact detector during training. In HiFi-GAN's tests, removing its multi-period discriminator cut listener scores from 4.10 to 2.28.

Is FAD a good measure of AI music quality? Useful but limited. FAD compares a whole set of outputs with a reference set, so a weak reference or small sample can skew it, and it says little about any single song. Per-song FAD and listening tests fill the gap.

Can a model restore low-quality input audio? Some decoders are trained to. DiffRhythm's learned to rebuild lossless-quality output from MP3-compressed input. That restoration is a prediction of the missing detail, so judge the result by ear.

Sign up for Soundverse free and check your next generation with your ears and a meter. More guides for producers on the Soundverse blog.

Setting up your studio…Warming up Agent One…Tuning the instruments…Getting your library ready…Almost there…