Are There Specific Machine Learning Models That Excel at Cloning Vocal Timbres?
Cloning vocal timbres: what RVC, so-vits-svc, DDSP-SVC, Seed-VC, OpenVoice and ACE Studio each do well, how much data they need, and their limits.
Yes: for cloning vocal timbres in singing, the best-documented open model families are retrieval-based voice conversion (RVC), SoftVC VITS (so-vits-svc), diffusion-based SVC such as DDSP-SVC, and zero-shot Seed-VC. Speech models like OpenVoice and CosyVoice clone speaking voices, and ACE Studio's voice cloning turns a singer into a voice you write for in MIDI.
Which one excels depends on what you have: ten minutes of a singer's dry vocals or ten seconds, a sung guide or only notes and lyrics. Everything below comes from each model's paper or official repository, read on 19 November 2026. Most quality figures are the developers' own tests, so read them as directions rather than a ranking.
How models for cloning vocal timbres separate voice from performance
Singing voice conversion models share one idea. They split a vocal into what is sung (content), the melody (pitch, or F0) and who sings it (timbre), then rebuild the audio with a different timbre. RVC's README credits ContentVec, VITS and HiFi-GAN among its building blocks: a content encoder, a voice model and a vocoder.
The families differ in where the target timbre comes from (a model trained on one voice, or a short reference clip) and in how the audio is rendered. Singing adds a constraint speech doesn't have: the output must follow the guide's notes exactly, so singing models take the pitch track as an input. Our explainer on speech synthesis vs singing synthesis covers why.

Retrieval-based voice conversion (RVC)
Best at: a specific singer's timbre from a modest dataset, trained quickly on ordinary hardware.
The RVC WebUI recommends at least 10 minutes of low-noise audio for training and says it trains fast even on weak graphics cards, starting from a base model pretrained on nearly 50 hours of the open VCTK dataset. Its signature trick is retrieval: it stores features from your training set and, at conversion time, swaps in the closest match to reduce "tone leakage", where the guide singer's own timbre bleeds through. It uses the RMVPE pitch extractor, can merge two models to blend timbres, and ships a real-time GUI with 170 ms end-to-end latency, or 90 ms with ASIO audio devices, depending on drivers.
Limits: one model per voice, and the output can't beat the dataset. The so-vits-svc-fork README lists the original RVC repository as no longer maintained and points to Applio as an actively maintained RVC-family project.
SoftVC VITS (so-vits-svc)
Best at: singing voice conversion, which it was built for.
As its full name, SoftVC VITS Singing Voice Conversion, suggests, it pairs SoftVC content features with the VITS model. The popular so-vits-svc-fork added real-time conversion, more accurate pitch estimation with CREPE, a fix for how ContentVec was used, and roughly twice-as-fast training. It asks for training clips of about 10 seconds or less and at least 4 GB of GPU memory.
Limits: the fork is marked "No Longer Maintained", and its README is candid about why: "The voice changer boom that occurred in 2023 has come to an end, and many developers, not just those in this repository, have been not very active for a while."
Diffusion-based singing voice conversion: DiffSVC and DDSP-SVC
Best at: adding a diffusion stage for quality, from research systems to light home setups.
DiffSVC, a May 2021 paper, built singing voice conversion on a denoising diffusion model, using phonetic posteriorgrams for content plus F0 and loudness features. Its authors reported better naturalness and voice similarity than the state-of-the-art systems of the time.
DDSP-SVC aims to make free voice-changer software practical on personal computers. Its README says training and synthesis need far less hardware than so-vits-svc, training time drops by orders of magnitude to near RVC's, and real-time use costs less than so-vits-svc but probably slightly more than the latest RVC. Raw DDSP output isn't ideal, but with its shallow diffusion stage the README says quality can match so-vits-svc and RVC on some datasets. It recommends about 1,000 training clips, none shorter than 2 seconds, and says its default config suits a GTX 1660.
Limits: diffusion adds sampling steps. DDSP-SVC says you may hear a quality loss when you speed sampling past 20x, and its newest rectified-flow model is labelled experimental.
Zero-shot voice conversion: Seed-VC
Best at: cloning a timbre with no training at all, including singing.
Seed-VC, from a November 2024 paper, uses a diffusion transformer and perturbs the source timbre during training to curb leakage. It handles singing by adding F0 conditioning, and its README says it clones a voice from a 1 to 30 second reference. Fine-tuning is optional, from as little as one utterance per speaker.
Its evaluation page has the most useful comparisons, with the caveat that they are the developer's own. For speech, Seed-VC scored higher speaker similarity than CosyVoice and OpenVoice (0.868 against 0.844 and 0.755) with fewer word errors, though OpenVoice scored slightly higher on the audio quality measures. For singing, Seed-VC worked zero-shot from one random clip per voice against RVCv2 models trained on each voice. It scored slightly higher similarity (0.741 against 0.726) and fewer transcription errors on the lyrics, while RVC tracked pitch a little more closely.
Limits: real-time use carries about 300 ms of algorithm delay plus about 100 ms on the device, and the README strongly recommends a GPU. For singing it suggests 30 to 50 diffusion steps.
Speech cloning models: OpenVoice and CosyVoice
Best at: spoken parts: intros, skits, interludes, voice-overs.
These are text-to-speech systems. OpenVoice clones a speaker's tone colour from a short clip and generates speech in multiple languages, with control over emotion, accent, rhythm, pauses and intonation, and can clone into languages missing from its multi-speaker training set. Its README says both versions have been released under the MIT License since April 2024. CosyVoice, described in a July 2024 paper, is a multilingual zero-shot text-to-speech model that pairs a language model with flow matching.
Limits for singing: a speech model invents its own prosody from text, while a song needs exact notes and durations. Even Seed-VC, a conversion model, added F0 conditioning to handle singing. Use speech cloning for talking and a singing model for the vocal.
Singing synthesis voice models: Synthesizer V and ACE Studio
Best at: new melodies and lyrics in a voice, with no guide singer at all.
These sing what you write in a piano roll instead of converting a take. Reviewing Synthesizer V Studio 2 Pro for Sound On Sound in July 2025, John Walden wrote: "The engine utilises AI/machine learning to process detailed samples of the actual singer from which the selected voice library was built and to resynthesize those samples into the performance you have constructed." He adds that Dreamtonics explains how it collaborated with, and compensated, the singers behind each library.
ACE Studio goes a step further and clones your own voice into a voice you drive from MIDI and lyrics. It asks for the biggest dataset of any family here: 30 to 100 minutes of singing. Each sample file is treated as one language, so don't mix languages within a file, and ACE suggests a singer comfortable in two ranges, such as tenor and bass, train two voices.
Limits: the performance is only as expressive as your editing. Our guide to vocal expression control compares the controls.
Which machine learning model is best for cloning a vocal timbre?
- You have 10+ minutes of a singer's dry vocals and a sung guide: RVC, or an RVC-family app such as Applio.
- You have seconds of reference, or want to audition many voices: Seed-VC's singing model.
- You need live monitoring on modest hardware: RVC's real-time GUI, or DDSP-SVC.
- You have no guide singer, only MIDI and lyrics: a singing synth, or ACE Studio's voice cloning.
- The part is spoken: OpenVoice or CosyVoice.
- You'd rather not run models yourself: a hosted tool, compared in our guide to AI tools for cloning a specific vocal timbre.
Two rules hold for every family: no model rescues a noisy, narrow dataset, and every one assumes the singer's consent. DDSP-SVC's disclaimer tells you to train only on legally obtained, authorized data.
Where Soundverse fits
Soundverse is an AI music platform that takes you from idea to audience, and the same trade-off between a short reference and a long training run shows up in its two voice tools. Train Voice Clone needs one take with at least 15 seconds of singing, and the AI Song Generator then sings new songs in that voice; cloning with Soundverse v9 needs Pro. Train Voice Model learns from at least 10 minutes of clean, dry, solo recordings, trains for 10 to 350 epochs over hours, and puts your voice on existing vocals in Voice Swap, which keeps the original performance. Both need the Creator plan or above, and both stay locked until you confirm the voice is yours or the singer has given explicit consent. Licensing is set by plan: the Free plan is for personal use, and the Creator plan comes with a royalty-free license.
This video clones a singer's own voice, then makes full songs sung in it.
FAQ
Which machine learning model is best for cloning a singing voice?
For a voice you can record for 10 minutes or more, RVC remains the practical default: fast to train, retrieval to limit leakage, and real-time capable. With only a short clip, Seed-VC's zero-shot singing model is the best-documented option, and in its developer's tests it matched trained RVC models on similarity.
Can speech voice cloning models like OpenVoice sing?
They aren't built to. OpenVoice and CosyVoice generate speech from text and decide the rhythm and intonation themselves, while singing needs exact notes. Use a singing voice conversion model or a singing synth for the vocal.
Which voice cloning models run in real time?
RVC's GUI reports 170 ms end-to-end latency, or 90 ms with ASIO devices. so-vits-svc-fork, DDSP-SVC and Seed-VC also have real-time modes, and Seed-VC's adds up to about 400 ms of total delay. Seed-VC strongly recommends a GPU for live use, while so-vits-svc-fork says CPU inference can be fast enough.
Sign up for Soundverse free and hear your own voice on a song before you train anything locally.
More guides on vocals and AI singing on the Soundverse blog.