Foreword

When training your own AI voice, some common concerns are: Do I need a high-end graphics card? How stable is voice cloning? Are there many pitfalls with Chinese pronunciation? This post summarizes my own experience, documenting the entire process from hardware requirements and a comparison of mainstream open-source models, to a practical implementation of LoRA fine-tuning and how to handle Chinese pronunciation.

In the last year or two, LoRA fine-tuning has gradually matured, and with the emergence of open-source models like CosyVoice 3, the barrier to producing usable quality voices with consumer-grade hardware is now much lower than before.

I. Hardware and Training Thresholds

The phrase "it runs on 8GB of VRAM" is often misunderstood. We need to distinguish between two things: the requirements for inference and training are not the same.

• Inference: Simply using a pre-trained model to generate speech. 8GB of VRAM is generally sufficient for most TTS models (like Qwen3-TTS 1.7B, CosyVoice 0.5B).
• LoRA Fine-tuning: I personally use an RTX 5070 Ti (16GB) to run Qwen3-TTS LoRA and can train a voice of usable quality. A 16GB consumer-grade card is enough.
• Full-parameter Fine-tuning: This requires industrial-grade resources (the original CosyVoice pre-training used 64 V100-32GB cards), something individuals typically don't need to consider.

The reason 16GB is viable is that LoRA only updates a small number of low-rank parameters; the main memory overhead comes from the base model's weights and activations. The TTS model itself (1.5B–1.7B) isn't that large, and combined with a moderate batch size and sequence length, it fits within 16GB.

Below are my measured VRAM usage figures at various stages of running Qwen3-TTS LoRA on an RTX 5070 Ti (16GB):

As you can see, even during the most demanding LLM r64 training stage, the peak usage only reaches about 10.6GB, leaving about 5GB of headroom on a 16GB card. In other words, 16GB isn't a tight squeeze; it's quite comfortable, with still room to increase the batch size or rank.

Don't understand the terms in the table? This section is for you.

• r64 / r32 (rank): This is LoRA's "capacity knob." LoRA doesn't touch the original massive model; instead, it adds a small, trainable "plugin" of parameters, and rank determines how large this plugin is. The higher the number, the more voice details it can learn, but it also consumes more VRAM and training time, and is more prone to "memorizing" training samples (overfitting) when data is insufficient. r64 means rank=64, and r32 means rank=32. The number of plugin parameters for r32 is about half that of r64, saving more VRAM and training faster, while the difference in voice fidelity is usually minor. When the dataset is small (10-30 minutes), r32 is often sufficient and less prone to overfitting. Go for r64 if you want to squeeze out a bit more detail. My table shows results for r64 with a peak of just 10.6GB, so switching to r32 would be even easier.
• LLM Stage / Flow Stage: Qwen3-TTS is actually two models working in tandem. The LLM is responsible for "text -> what to say and the rhythm of phrasing." The Flow (flow matching acoustic decoder) is responsible for "turning that into an actual sound waveform" (timbre, quality). You need to fine-tune both with LoRA for the voice to sound like the target person.
• all-linear: This refers to applying LoRA to "all linear layers" of the model, rather than just a few attention layers. Broader coverage means more complete learning, with slightly more parameters, which is why the "Flow all-linear" row is listed separately.

As for QLoRA (Quantized LoRA), it can theoretically reduce VRAM requirements by another ~75%, but it's not yet mature in the TTS field. Unsloth officially advises against using 4-bit QLoRA for the Qwen3.5 series because quantization errors significantly impact speech quality. Since running native LoRA on 16GB is already feasible, most people don't actually need QLoRA.

In short, if you want to create a custom voice, a 16GB VRAM consumer-grade graphics card (like an RTX 5070 Ti, or an RTX 4060 Super 12GB) is enough to get started. If you want a larger batch size, a higher rank, or faster training, you can then upgrade to 24GB (RTX 3090 / 4090), but that's an option for "faster and better," not the entry-level requirement.

II. Comparison of Open-Source TTS Models: Qwen, CosyVoice

The choice of base model affects the accent, quality ceiling, and the difficulty of subsequent training. Here are a few I've tested:

Qwen3-TTS (Alibaba, released in January 2026)

It's quite feature-complete, supporting ten languages, with a speaker similarity of about 0.95 and a Chinese Character Error Rate (CER) of about 0.77%. However, in my experience, the generated voice still has a noticeable "mainland Chinese accent," and for Traditional Chinese, it has a tokenizer byte-fallback issue, leading to pronunciation errors for out-of-vocabulary (OOV) words.

BreezyVoice (MediaTek, based on CosyVoice 2)

Its Taiwanese Hokkien speech effect is excellent (MOS reaching 5.0) and supports Bopomofo annotation for disambiguation. A major limitation is that the official release only provides inference scripts; it doesn't support secondary training, so you can't use it to customize your own voice.

CosyVoice 3 (Alibaba, December 2025)

Currently the most balanced choice overall. It has 1.5B parameters and was trained on a million-hour-level dataset, with a Chinese CER of about 0.71%. Its voice is closer to a Taiwanese accent, and its emotional control is relatively natural. It also has a "pronunciation patching" feature, which allows you to directly override mispronounced words with Pinyin to correct them, without retraining.

If your goal is custom training in a Traditional Chinese context, CosyVoice 3 is the most practical starting point right now.

III. LoRA Fine-Tuning Implementation

Why Not Just Use Voice Cloning (ICL)?

Zero-shot voice cloning (In-Context Learning) is indeed convenient, requiring only a 3-10 second reference audio clip to work, and the results are good for short sentences. But as sentences get longer, the timbre tends to drift; this is a common problem across all current models. Extending the reference audio to about 20 seconds can improve stability, but for consistently stable quality, LoRA fine-tuning remains the more reliable approach.

Implementation Steps

Step 1: Prepare Training Data (The Most Time-Consuming Step)

The community commonly recommends 10-30 minutes of training data. You need to prepare clean, background-noise-free personal audio files and segment them into short 5-15 second clips.

In my experience, the training itself can be completed in about 30 minutes to an hour. The real time-sink is preparing the audio files: cleaning up noise, segmenting clips, and proofreading transcripts. This prep work accounts for over 70% of the entire process.

Step 2: Data Annotation

Map each audio file to its accurate transcript, usually in the format of a manifest file:

audio_001.wav|你好，歡迎來到我的技術部落格。
audio_002.wav|今天要分享的是關於語音合成的實戰經驗。

Step 3: Run LoRA Training

Taking CosyVoice 3 as an example, the basic training command is as follows:

python train_lora.py \
  --model_path pretrained/CosyVoice3-0.5B \
  --data_dir ./training_data \
  --output_dir ./output/my_voice \
  --lora_rank 64 \
  --epochs 50 \
  --batch_size 4 \
  --learning_rate 1e-4

Step 4: Validate Quality

After training is complete, you can use a TTS-ASR self-refinement loop for initial automated validation: first have the model generate speech, then use Whisper to transcribe it back to text, and compare the phoneme error rate. According to a 2025 study, this method can achieve a 55.88% relative error reduction in a mixed Chinese-English context.

However, the final quality must still be judged by the human ear. Automation can filter out obviously problematic samples, but determining if the intonation is natural and the emotion is appropriate is still something humans are better at.

IV. Handling Chinese Pronunciation

Two of the more troublesome problems with Chinese TTS are:

1. Polyphonic characters: The two "好" characters in "好好學習" (hǎo hǎo xué xí - study hard) have different pronunciations.
2. Specific vocabulary: "軟體" (ruǎntǐ) vs. "軟件" (ruǎnjiàn). The model's training data is primarily Simplified Chinese, so it hasn't seen many Traditional Chinese terms.

Traditional Approach: Three-Layer Preprocessing

輸入文字 → 字典覆寫（phrase override）
         → OpenCC 繁轉簡
         → g2pW 轉漢語拼音
         → 送入 TTS 模型

This process can solve most problems, but the maintenance cost is not low. Every time a new pronunciation error is encountered, the dictionary must be updated manually. Polyphonic characters are disambiguated using neural network models like Polyphone BERT or g2pW.

CosyVoice 3's Approach: Pronunciation Patching

CosyVoice 3's built-in pronunciation patching feature allows you to directly override the pronunciation of specific characters in the input text using Pinyin:

# 原始輸入（「軟體」的「體」可能被唸錯）
軟體工程師的日常

# 使用發音修補（拼音標註）
軟<phoneme ph="ti3">體</phoneme>工程師的日常

According to official tests, the correction rate of this method is close to 100%. It requires no code changes or retraining and can be handled directly at the input stage. This is quite practical for Traditional Chinese users.

Conclusion

To summarize, here are three key points for training a personal AI voice:

1. Hardware: A 16GB VRAM consumer-grade card (like an RTX 5070 Ti) is enough to get started; a 24GB card is not a must.
2. Model: Start with CosyVoice 3, which offers a good balance of sound quality, accent, and trainability.
3. Data: Spend time recording 10-30 minutes of high-quality training audio.

Although AI can already perform partial self-training and validation through TTS-ASR loops, the preparation of training data and the final auditory judgment of the results are still difficult to fully replace.

Also worth noting is that new technologies like LoRP-TTS (inference-time LoRA optimization) are making the process more automated. In the future, it might not be necessary to prepare training data in advance, as models could adapt to the target voice in real-time during inference. The overall barrier to entry should continue to decrease.

References

• Qwen3-TTS Technical Report
• CosyVoice 3 Paper
• CosyVoice 3 Tech Guide
• BreezyVoice GitHub
• LoRP-TTS: Inference-Time LoRA
• TTS-ASR Self-Refining Framework
• Qwen3-TTS LoRA Fine-Tuning Guide
• Unsloth TTS Fine-tuning Guide