How Does AI Voice Synthesis Actually Work?

How AI voice synthesis works, step by step: the text front end, acoustic models, neural vocoders, codec and diffusion models, singing, cloning and MOS.

AI voice synthesis works as a pipeline. A front end turns text or lyrics into phonemes and prosody, an acoustic model predicts how those sounds should look over time, usually as a mel-spectrogram or audio tokens, and a vocoder renders that as a waveform you can play. For singing, the notes supply each sound's pitch and length.

Text-to-speech, AI singers and voice cloning share that pipeline. What changes is the input, and how many stages a newer model folds into one. Here's each stage in plain English, with research details from the linked papers, checked on 19 November 2026.

Diagram of the AI voice synthesis pipeline: text or lyrics with notes go through a front end, an acoustic model that predicts a mel-spectrogram or audio tokens, and a vocoder that outputs a waveform

Step 1: The front end turns text into sounds and prosody

Before any audio exists, the front end has three jobs.

  1. Text normalisation. Anything that isn't a plain word gets expanded. "Dr." can be Doctor or Drive, "1/2" can be a half or a date, "2026" is read as a year and "£5" becomes "five pounds". Richard Sproat and Navdeep Jaitly tested neural networks on this job in a paper posted to arXiv in October 2016. Some scored very well on overall accuracy, but "the errors that are produced are problematic, since they would convey completely the wrong message if such a system were deployed in a speech application." Their fix was a simple finite-state filter on top, which reduced those errors.
  2. Grapheme to phoneme. Spelling becomes sounds, from a pronunciation dictionary plus a learned guess for words it doesn't know. Words spelled the same but said differently need context: "live" in "a live show" isn't the "live" in "where we live".
  3. Prosody. Which syllables take stress, where phrases break, and how pitch rises and falls across a sentence. In speech the model predicts this, and your main control is punctuation.

Step 2: The acoustic model, from stitched recordings to neural networks

The acoustic model decides how each sound should sound in this voice. It has been through three generations.

  • Concatenative. DeepMind's WaveNet announcement (8 September 2016) describes the approach that dominated until then: a very large database of short fragments recorded from one speaker, recombined into complete sentences. Changing the voice, emphasis or emotion meant recording a new database.
  • Statistical parametric. The same post explains the alternative: store the voice in a model's parameters and control it through the inputs. It was flexible but tended to sound less natural, partly because its output passed through signal-processing vocoders.
  • Neural. Tacotron 2, by Jonathan Shen and colleagues (December 2017), mapped characters to mel-spectrograms with a sequence-to-sequence network, then rendered them with a WaveNet vocoder. A mel-spectrogram is a picture of the energy in each frequency band over time, on a scale shaped like human hearing. Tacotron 2 scored a mean opinion score (MOS) of 4.53, against 4.58 for professionally recorded speech.

FastSpeech 2, by Yi Ren and colleagues (June 2020), generates in parallel instead of frame by frame. It learns duration, pitch and energy from real recordings in training and predicts them at synthesis time, so prosody becomes something a system can measure and adjust. Its authors call the core difficulty the one-to-many mapping problem, where many spoken versions fit the same text. Average them and you get a flat read.

VITS, by Jaehyeon Kim, Jungil Kong and Juhee Son (June 2021), goes from text to waveform in a single end-to-end model. Its stochastic duration predictor and uncertainty modelling let one sentence come out with different rhythms and pitches, and on the single-speaker LJ Speech dataset its MOS was comparable to the real recordings.

Step 3: The vocoder turns features into a waveform

A spectrogram still isn't sound, so the vocoder turns it into audio samples. WaveNet generated them one at a time, each conditioned on all the ones before. DeepMind's post notes that audio typically runs at 16,000 samples per second or more, which made this slow, but listeners rated it more natural than Google's best parametric and concatenative systems in US English and Mandarin.

HiFi-GAN, by Jungil Kong, Jaehyeon Kim and Jaekyoung Bae (October 2020), made neural vocoding fast. It uses a generative adversarial network, where one network learns to spot fake audio while the other learns to fool it. The authors report quality close to human recordings in listening tests, generating 22.05 kHz audio 167.9 times faster than real time on a single V100 GPU. A metallic buzz or a smeared "s" in an AI voice can come from this stage.

Codec language models and diffusion: the newer approaches

Newer research systems often skip the spectrogram and work on compressed audio. Two well-known examples:

  • Codec language models. A neural audio codec squeezes audio into a compact sequence of discrete codes, and a language model predicts those codes much as a chatbot predicts words. VALL-E, by Chengyi Wang and colleagues (January 2023), treats text-to-speech as language modelling on codec codes. Given a 3-second recording of an unseen speaker as a prompt, it speaks new text in that voice and keeps the prompt's emotion and acoustic environment.
  • Diffusion. NaturalSpeech 2, by Kai Shen and colleagues (April 2023), uses a codec too, but a diffusion model generates the codec's latent vectors from the text, refining them out of noise. Its authors argue that predicting tokens one by one causes unstable prosody, skipped or repeated words and poor voice quality. Trained on 44K hours of speech and singing, it could also sing zero-shot from only a speech prompt.

How singing changes the pipeline

Singing keeps the three stages and hands two of the model's guesses to the score. Where FastSpeech 2 predicts how long each sound lasts and where the pitch goes, a singing model takes both from the notes and adds detail around them. Each syllable has to fill its note at the song's tempo, vowels are held far longer than in speech, and the pitch curve adds scoops and vibrato around the written note.

For each stage in singing detail, see how AI voice synthesis generates singing vocals. For what separates the two jobs, read the primary differences between speech synthesis and singing synthesis.

Voice cloning and voice conversion, in one paragraph

Cloning conditions the acoustic model on a representation of one voice, either by fine-tuning on minutes of that voice's recordings or, zero-shot, by giving a short prompt at synthesis time, as VALL-E and NaturalSpeech 2 do. Conversion starts from a real performance instead of text: the model keeps the words, timing and pitch line of the recording and swaps in another voice's timbre, which is why converted vocals keep human phrasing. Both need the voice owner's consent. Models for both are compared in which machine learning models excel at cloning vocal timbres.

How AI voices are evaluated: MOS and its limits

The standard test is the mean opinion score. Listeners hear samples, rate each on a scale from 1 to 5, and the ratings are averaged. DeepMind's WaveNet post describes MOS as a standard measure for subjective sound quality, gathered in blind tests, over 500 ratings on 100 test sentences in its case.

MOS numbers don't travel well between papers. Ambika Kirkland and colleagues surveyed Interspeech and SSW papers from 2021 to 2022 for the 12th ISCA Speech Synthesis Workshop in 2023: "Although standards for collecting and reporting MOS exist, researchers seem to use the term inconsistently, and underreport the details of their testing methodologies." Across the surveyed papers, real speech averaged higher when listeners rated quality than when they rated naturalness, and in the authors' own tests, changing the scale increment or the instructions changed the scores for some systems.

Papers add speaker similarity, intelligibility and objective metrics. Our guide to how AI detects and preserves audio quality during synthesis covers those and MUSHRA tests. For your tracks, a blind A/B in the mix, on earbuds and a phone speaker, tells you more than any published score.

What each stage means for your tracks

  • A word said wrong: the front end. Respell it, and write numbers as words.
  • Flat delivery or odd emphasis: prosody. Punctuate for breath, shorten lines and, for singing, give the model a guide melody.
  • Metallic buzz or smeared sibilance: the vocoder or decoder. Generate again; EQ only softens it.
  • The voice drifts or sounds like someone else: the voice conditioning. Use a cleaner, drier reference.
  • Skipped or repeated words: a known weakness of token-by-token models. Generate again with shorter lines.

This walkthrough of Soundverse's Create tools shows AI Singing Generator from 0:31 and AI Speech Generator from 0:39.

Where Soundverse fits

Soundverse is an AI music platform that takes you from idea to audience, built for independent artists who release music, with tools for both speech and singing. AI Speech Generator turns up to 3,000 characters of text per run into narration in built-in voices; it doesn't clone voices. AI Singing Generator sings your lyrics as an a cappella vocal in a style such as pop, rap, rock or choir, or in a voice similar to a reference of at least 30 seconds, about the first 350 characters per take.

For your own voice, Train Voice Clone builds a reusable singing voice from one take with at least 15 seconds of singing, on the Creator plan and above, once you confirm the voice is yours or the singer has given explicit consent. Voice Swap is the conversion route: it re-sings your take in another voice and keeps your performance.

FAQ

Is AI voice synthesis the same as text-to-speech? Text-to-speech is its speech branch. AI voice synthesis also covers singing from lyrics and notes, voice cloning and voice conversion, all built on the same front end, acoustic model and vocoder ideas.

How much audio does an AI need to clone a voice? Research systems such as VALL-E imitate a voice from a 3-second prompt. Tools vary: Soundverse's Train Voice Clone needs one take with at least 15 seconds of singing, and Train Voice Model, built for Voice Swap, needs at least 10 minutes of clean, dry recordings. Use only your own voice or one you have consent to use.

Why do AI voices sometimes skip or repeat words? Models that generate audio tokens one by one can lose their place in the text, and the NaturalSpeech 2 authors list skipped and repeated words among the problems of that approach. Shorter lines and a fresh generation are the practical fix.

Sign up for Soundverse free and hear your lyrics spoken and sung. More vocal guides on the Soundverse blog.

Setting up your studio…Warming up Agent One…Tuning the instruments…Getting your library ready…Almost there…