How Voice Cloning Works in Modern Software, Explained
How voice cloning works in modern software: training audio, speaker embeddings, TTS vs voice conversion, fine-tuning, vocoders, singing and safeguards.
Here is how voice cloning works in modern software: the app cleans a recording of the target voice, turns it into a speaker representation, either a compact embedding or a fine-tuned model, and uses that to steer a text-to-speech or voice conversion model. A neural vocoder then renders the waveform, and consent checks gate the whole process.
Below, each stage in turn. For how synthetic voices are generated from text in general, read how AI voice synthesis works. Papers and vendor docs cited here were checked on 21 and 22 November 2026.

Step 1: Recording and cleaning the training audio
Every clone starts with recordings of one person. The software can't separate the voice from everything else in the file, so it learns the room, the hiss and the habits too. ElevenLabs' Professional Voice Cloning docs warn that long pauses and filler sounds like "uhm" will be mimicked.
Cleaning usually means:
- One speaker only: no other voices, music or crosstalk.
- No noise or reverb, ideally by recording dry.
- Long silences trimmed and levels matched across clips.
- For singing, the lead vocal isolated, without harmonies or doubles.
How much audio you need depends on the approach: seconds to minutes for instant clones, tens of minutes to hours for trained ones.
Step 2: Turning a voice into numbers with speaker embeddings
The core idea of modern cloning is the speaker embedding: a fixed-length list of numbers that describes how a voice sounds, independent of what it's saying. Clips from the same person land close together; different people land apart.
The idea comes from speaker verification, the technology that checks whether two recordings are the same person. In a 2018 paper, Jia et al. trained a speaker encoder on a verification task, using noisy speech from thousands of speakers with no transcripts, so it could produce an embedding from seconds of reference speech. A Tacotron 2-based synthesis network then generated a mel spectrogram from text, conditioned on that embedding, and a WaveNet-based vocoder turned it into audio. The authors found that training the encoder on a large and diverse set of speakers mattered for cloning voices it had never heard.
Step 3: Few-shot vs fine-tuned, two ways to learn a voice
There are two ways to get from a few recordings to a usable voice. A 2018 paper, "Neural Voice Cloning with a Few Samples", compared them directly:
- Speaker encoding: a separate network infers the embedding straight from the reference clips, and a multi-speaker model uses it as is. Nothing is retrained, so the clone is ready almost at once. Products call this instant or zero-shot cloning.
- Speaker adaptation: the multi-speaker model itself is fine-tuned on the target voice. ElevenLabs describes its professional tier this way, as training (fine-tuning) the model on large sets of one speaker's voice to create a custom model.
The abstract states the trade-off:
"While speaker adaptation can achieve better naturalness and similarity, the cloning time or required memory for the speaker encoding approach is significantly less, making it favorable for low-resource deployment." (Arik et al., 2018)
Text-to-speech vs voice conversion: two ways to use a clone
Text-to-speech cloning takes a script and produces new speech in the cloned voice. Timing, emphasis and pitch all come from the model. This suits narration, explainers and voiceovers.
Voice conversion, also called speech-to-speech, takes a real performance and changes who it sounds like. The Singing Voice Conversion Challenge 2023 describes how most such systems work: encoders extract a content feature carrying the words and phonetics, plus prosody features such as pitch (f0) and energy. The content encoder is trained to be speaker-independent, and a decoder pre-trained on many voices is then fine-tuned on the target or steered by a speaker embedding. Twelve of the challenge teams built their content features from self-supervised speech models such as HuBERT, WavLM and ContentVec.
Because timing and melody come from a human take, singers and performers lean on conversion.
The vocoder: turning features into sound
Most cloning systems don't output audio directly. The generator predicts an acoustic picture of the sound, usually a mel spectrogram, and a neural vocoder turns that into a waveform. The SVCC 2023 organisers found most teams still used this two-stage design, and every vocoder in the challenge was GAN-based.
Some familiar artifacts can come from this stage: a metallic buzz on sustained notes, smeared breaths, harsh sibilants.
Why zero-shot clones sound less like the speaker than trained ones
An embedding squeezes a voice into a short summary from a few seconds of audio. Everything the reference didn't contain, such as the speaker's top register, the way they laugh or how they shape particular vowels, the generator has to guess from voices it learned in training. A fine-tuned model has heard far more of the actual person, so it guesses less.
Three practical consequences follow:
- Voices unlike the training data, such as a rare accent or a distinctive rasp, tend to clone worse, because the embedding captures less of what makes them distinctive.
- Reference quality matters more for zero-shot. With seconds of audio, one noisy clip is a large share of what the model knows.
- Even the best systems don't fully match the person. In the SVCC 2023 listening tests, the top system reached human-level naturalness, but no team scored as high on similarity as the target speakers themselves.
Singing vs speech: what changes for voice cloning
The SVCC 2023 paper explains why singing is harder: compared with speech, it covers a wider range of pitch, energy, expression and style, and the output has to follow the song's notes while each singer's style varies. Converting singing with a model that only learned the target from speech was harder still, especially for similarity.
Pitch is the first thing singing software handles separately. Conversion tools extract the pitch contour from the guide vocal and pass it through, often with a semitone shift, so the melody survives the change of voice. Formants, the resonances that give a voice its size and vowel colour, are the second. Shift pitch without managing them and a deep voice pushed up an octave sounds small and cartoonish. Our explainer on how AI manages formant shifting covers the details.
Safeguards: consent verification, voice CAPTCHAs, watermarking and detection
Consent confirmation. Many tools ask you to confirm the voice is yours or that the speaker agreed, and won't start training until you do.
Voice CAPTCHAs. A stolen recording can't say a fresh phrase on demand, so some services ask the speaker to read one live. Speechify's voice cloning docs issue a single-use, short-lived challenge phrase and check that the person reading it matches the voice sample. ElevenLabs' API has a verification captcha endpoint for professional clones that takes an audio recording of the user. ElevenLabs' safety page puts it plainly:
"We also embed product features to deter bad or irresponsible actors, including blocking the cloning of celebrity and other high risk voices, and requiring technological verification for access to our Professional Voice Cloning tool."
Watermarking. A watermark is an inaudible signal written into generated audio so a detector can find it later. AudioSeal, presented at ICML 2024, was designed to detect AI-generated speech down to the level of individual samples, so it can flag which part of a clip was synthetic, and its detector is built to run fast enough for large-scale use. Detection. Classifiers estimate whether audio is synthetic. ElevenLabs' safety page describes an AI Speech Classifier that detects whether a clip was created with ElevenLabs. A classifier like that speaks for one vendor's output, so a clean result says little about other tools.
The legal frame for using someone's voice
In many countries a person's voice is protected through publicity, personality or data-protection rules, and those rules differ by jurisdiction. Our overview of voice cloning consent laws by country walks through the US, the EU and the UK.
The working rules: clone your own voice, or get written consent naming the uses, the term and how it can be withdrawn. Keep the consent record, label AI voices where platforms or laws require it, and never make a real person appear to say or sing something they didn't agree to.
Where Soundverse fits
Soundverse is an AI music platform that takes you from idea to audience, and its voice tools follow the two use modes above. Train Voice Clone takes one recording with at least 15 seconds of singing, and the AI Song Generator then sings brand-new songs in that voice. Train Voice Model takes at least 10 minutes of clean solo singing or speaking, trains for 10 to 350 epochs over hours, and builds a higher-fidelity model. That model works in Voice Swap, which re-sings an existing vocal in a new voice while keeping the original performance, with pitch shifts up to 24 semitones either way.
Both training tools keep Start Training disabled until you confirm the voice is yours or that the singer has given explicit consent.
This video trains a voice model from scratch on Soundverse and then uses it in Voice Swap to turn other vocals into that voice.
FAQ
How much audio does voice cloning need?
Instant, zero-shot clones work from seconds to a few minutes of clean audio; fine-tuned clones need tens of minutes to hours. In both cases, clean and consistent recordings matter more than raw length.
What is the difference between voice cloning and voice conversion?
Cloning captures a voice as an embedding or a fine-tuned model. Voice conversion is one way to use it, re-voicing a real performance while keeping its timing and melody; text-to-speech is the other, generating new speech from a script.
Can AI-cloned voices be detected?
Sometimes. A watermark written in at generation, as in the AudioSeal research system, can be found by a matching detector, but only if the generator added one. Classifiers can also flag synthetic audio, though a vendor's classifier is built around its own output.
Is it legal to clone someone's voice?
Cloning your own voice is the low-risk case. Cloning someone else's needs their explicit consent, and using a voice without permission can breach publicity, personality or data-protection rules in many places. Get consent in writing and check the rules where you and the speaker live.
Sign up for Soundverse free to train a voice clone of your own singing. More guides on the Soundverse blog.