Can You Define the Role of Deep Learning in Realistic Audio Generation?
Deep learning audio generation explained: what makes audio sound real, WaveNet to HiFi-GAN, codecs, transformers, diffusion, MOS, FAD, limits and consent.
In deep learning audio generation, neural networks learn what real sound is like directly from recordings: the fine detail of a waveform, the timbre and dynamics of a voice or instrument, and the structure that holds a sentence or a song together. That is why generated speech, music and effects can now pass for recordings.
Below: what "realistic" means, the building blocks with dates, how realism is measured, where it breaks, and the consent questions it raises. Sources were read on 19 November 2026. This is published research, not a description of how any commercial tool, ours included, is built.
What makes generated audio sound realistic
Listeners judge three things:
- A clean signal. No clicks, metallic buzz, hiss or smeared consonants. One audible artifact breaks the illusion.
- Natural timbre and dynamics. The voice or instrument sounds like a real source in a real room: breath, attack, vibrato, and loudness that rises and falls with the phrase.
- Coherent structure. Words that follow each other sensibly, a melody that comes back, a groove that holds, a door creak that unfolds the way doors do.
The third is the hardest. Zalán Borsos and Neil Zeghidour of Google Research put the problem in one line in their AudioLM post (6 October 2022): "Generating realistic audio requires modeling information represented at different scales." Phonemes build words and sentences; notes build phrases and songs. A written sentence takes a few dozen characters; its waveform, the post notes, runs to hundreds of thousands of values.
Before neural networks, synthesis glued recorded fragments together or drove a hand-built signal model; parametric vs concatenative synthesis explains both.
The deep learning building blocks, and what each one fixed
Sample-by-sample waveform models: WaveNet (2016)
DeepMind's WaveNet (Aäron van den Oord and colleagues, September 2016) generated raw audio one sample at a time, predicting each from all the samples before it, on audio running at tens of thousands of samples per second. In text-to-speech tests, listeners rated it significantly more natural than the best parametric and concatenative systems, in both English and Mandarin. One model could voice many speakers by conditioning on speaker identity, and trained on music it made fragments the authors called often highly realistic.
What it fixed: fine detail. What it cost: speed, since every sample waits for the one before.
Neural vocoders trained against a critic: HiFi-GAN (2020)
Many speech systems predict a spectrogram, a picture of energy by frequency over time, and let a vocoder turn it into a waveform (how AI voice synthesis works covers the pipeline). Generative adversarial networks made vocoders fast: a generator makes audio while a discriminator learns to tell it from real recordings.
HiFi-GAN (Jungil Kong, Jaehyeon Kim and Jaekyoung Bae, October 2020) pinned down what earlier GAN vocoders missed: "As speech audio consists of sinusoidal signals with various periods, we demonstrate that modeling periodic patterns of an audio is crucial for enhancing sample quality." In its listening test, quality came close to human recordings, and a small version ran 13.4 times faster than real time on a CPU with quality comparable to an autoregressive model.
What it fixed: natural timbre at practical speed, and much of the buzz.
Neural audio codecs: SoundStream (2021) and EnCodec (2022)
Google's SoundStream (Neil Zeghidour and colleagues, July 2021) is a neural codec for speech, music and general audio: an encoder, a residual vector quantizer that turns audio into discrete codes, and a decoder, trained together with adversarial and reconstruction losses. In listening tests on 24 kHz audio, SoundStream at 3 kbps beat the Opus codec at 12 kbps. Meta's EnCodec (Alexandre Défossez and colleagues, October 2022) uses the same design, adds a spectrogram-based adversary to cut artifacts, and was evaluated on speech and music up to 48 kHz stereo.
What it fixed: long audio became a manageable sequence of codes that still decodes to high fidelity.
Transformers for long-range structure: AudioLM (2022)
Transformers, the network type behind text chatbots, predict the next token from a long context. Google's AudioLM (September 2022) used two kinds of token: heavily downsampled semantic tokens from w2v-BERT, a self-supervised audio model, for long-term structure, and SoundStream's acoustic tokens for detail such as the speaker's voice. Several Transformer models are chained, one per stage.
Trained on raw audio alone, with no transcripts, it continued speech prompts with plausible words in the same voice and prosody, and continued piano recordings coherently.
What it fixed: coherence over seconds without giving up fidelity.
Diffusion and latent diffusion: DiffWave (2020) and LDM (2021)
Diffusion models start from random noise and refine it over a fixed number of steps. DiffWave (Zhifeng Kong, Wei Ping and colleagues, September 2020) applied this to waveforms. It matched a strong WaveNet vocoder in listening tests (MOS 4.44 against 4.43) while synthesizing orders of magnitude faster, and on unconditional generation it beat autoregressive and GAN models on both audio quality and sample diversity.
Latent diffusion (Robin Rombach and colleagues, December 2021) moved the process into the compressed space of a pretrained autoencoder and added cross-attention layers so text could steer it. It was built for images, where pixel-space diffusion often consumed hundreds of GPU days. AudioLDM (Haohe Liu and colleagues, January 2023) carried the idea to text-to-audio and trained on a single GPU. Our guide to how neural networks generate music from text prompts shows these generators at work.
What it fixed: variety without losing fidelity, and text control at a manageable cost.

How the pieces combine for speech, music and sound effects
The pieces recombine by task. Speech systems often end in a neural vocoder; songs and vocals are codec tokens or latents generated by a transformer or diffusion model and conditioned on lyrics, style or melody; sound effects are general text-to-audio, AudioLDM's job, where texture and the order of events must ring true.
How is realism in AI audio measured?
Listening tests and MOS. The mean opinion score comes from telephony. ITU-T Recommendation P.800, approved in August 1996, sets out methods for subjective determination of transmission quality. Its listening-quality scale has five labels, Excellent (5), Good (4), Fair (3), Poor (2) and Bad (1), and the average of the ratings is the MOS. HiFi-GAN and DiffWave both report MOS. Compare MOS within one paper, not across papers, since listeners and instructions differ.
Real-or-generated tests. In the AudioLM post, raters deciding whether short clips were real speech or AudioLM continuations were right 51.2% of the time, not statistically different from guessing.
FAD. Fréchet Audio Distance (Kevin Kilgour and colleagues, Google, December 2018) adapts the Fréchet Inception Distance used to score generated images. It is reference-free: it compares embedding statistics for a set of generated clips with those of real recordings, and lower means closer. In its authors' tests it tracked perceived distortion better than signal-to-distortion ratio. It scores a set, not a song, and ignores whether lyrics make sense. MUSHRA, ViSQOL and FAD's weak spots are in how AI detects and preserves audio quality during synthesis.
This blind test pairs first-take Soundverse v9 songs with songs another AI music model made from the same prompts, loudness-matched and unedited, an informal version of the listening tests above.
Where deep learning audio generation still falls short
- Long-range structure. The AudioLM authors note that audio tokenizers trade reconstruction quality against long-term structure. Over minutes, songs can still drift or lose a chorus; the technical hurdles of long-form AI music explains why.
- Artifacts. The final decoder or vocoder leaves fingerprints: a metallic edge, smeared sibilants, a grainy top end. EnCodec trains with a dedicated spectrogram adversary to reduce them.
- Data. A model can only sound like what it has heard. AudioLM needed large corpora of raw audio, and its post notes that transcription-guided systems break down for untranscribed qualities such as a speaker's character. Expect the least realism where a style, language or instrument was rare in training.
- Compute. Latent diffusion was built to cut the cost of diffusion on raw pixels, and AudioLDM's authors name high computational cost as a problem of earlier text-to-audio systems.
Realism raises consent questions: voices and deepfakes
The progress that makes a vocal sound human also makes it easy to fake one. WaveNet already switched voices by conditioning on speaker identity, and AudioLM kept an unseen speaker's identity and prosody from a short prompt. When listeners could barely tell AudioLM's speech from real speech, Google kept it as research with no plans for a wider release, trained a classifier that detected its speech with 98.6% accuracy, and named audio watermarking as a possible next step.
For artists, the rules are short:
- Clone or reference only your own voice, or a voice whose owner has given written consent.
- Don't ask a model to imitate a named real singer.
- Keep the consent records, and disclose AI vocals wherever a platform asks you to.
Where Soundverse fits
Soundverse is an AI music platform that takes you from idea to audience, built for independent artists who release music, with tools for all three kinds of audio above. AI Speech Generator turns up to 3,000 characters of script per run into narration you can download as an MP3. AI Singing Generator sings your lyrics a cappella in a style such as pop, rap or choir, or in a voice like a reference of at least 30 seconds you have permission to use. AI Sound Effects Generator makes a clip from 0.5 to 30 seconds from a description, and Soundverse v9, on Pro and Max, turns your lyrics or a description into a full song with vocals.
Soundverse publishes no sound-quality score for Soundverse v9; its page says to listen and decide. Train Voice Clone builds a voice only once you confirm it's yours or the singer has consented.
FAQ
What did deep learning change in audio synthesis? It replaced hand-built rules and stitched recordings with models that learn waveform detail, timbre and timing from real recordings, which removed most of the joins and buzz that gave older synthesis away.
Is diffusion better than autoregressive generation for realistic audio? Neither wins everywhere. In DiffWave's tests, diffusion matched a WaveNet vocoder on speech quality and beat autoregressive and GAN models on unconditional generation, while token-by-token transformers such as AudioLM are strong at keeping structure over time. Many newer systems pair a codec or autoencoder with one of the two.
Can people tell AI-generated audio from real recordings? Often not by ear. In Google's AudioLM test, raters judged short speech clips correctly 51.2% of the time, close to chance, yet a simple classifier caught the generated ones with 98.6% accuracy. Detection tools and honest labels matter more than trained ears.
Sign up for Soundverse free and judge a generated vocal with your own ears. More explainers on the Soundverse blog.