---
title: "Training Your Own AI Voice: Hardware Requirements, Open-Source Model Comparison, and LoRA Fine-Tuning"
description: "My experience training AI voice: hardware differences for training vs. inference, comparing open-source TTS models like Qwen3-TTS and CosyVoice, LoRA fine-tuning implementation, and handling Chinese pronunciation."
canonical_url: "https://blog.markkulab.net/en/post/ai-voice-clone-lora-training-guide"
author: "Mark Ku"
author_url: "https://blog.markkulab.net/en/author/mark-ku"
site: "Mark Ku's Tech Notes"
date_published: "2026-06-07 15:41:44 +0800"
category: "AI"
language: "en"
license: "CC BY 4.0"
license_url: "https://creativecommons.org/licenses/by/4.0/"
attribution: "when reusing or quoting, credit the author and link back to the original"
---

# Training Your Own AI Voice: Hardware Requirements, Open-Source Model Comparison, and LoRA Fine-Tuning

## Foreword

When training your own AI voice, some common concerns are: Do I need a high-end graphics card? How stable is voice cloning? Are there many pitfalls with Chinese pronunciation? This post summarizes my own experience, documenting the entire process from hardware requirements and a comparison of mainstream open-source models, to a practical implementation of LoRA fine-tuning and how to handle Chinese pronunciation.

In the last year or two, LoRA fine-tuning has gradually matured, and with the emergence of open-source models like CosyVoice 3, the barrier to producing usable quality voices with consumer-grade hardware is now much lower than before.

## I. Hardware and Training Thresholds

The phrase "it runs on 8GB of VRAM" is often misunderstood. We need to distinguish between two things: the requirements for inference and training are not the same.

-   Inference: Simply using a pre-trained model to generate speech. 8GB of VRAM is generally sufficient for most TTS models (like Qwen3-TTS 1.7B, CosyVoice 0.5B).
-   LoRA Fine-tuning: I personally use an **RTX 5070 Ti (16GB)** to run Qwen3-TTS LoRA and can train a voice of usable quality. A 16GB consumer-grade card is enough.
-   Full-parameter Fine-tuning: This requires industrial-grade resources (the original CosyVoice pre-training used 64 V100-32GB cards), something individuals typically don't need to consider.

The reason 16GB is viable is that LoRA only updates a small number of low-rank parameters; the main memory overhead comes from the base model's weights and activations. The TTS model itself (1.5B–1.7B) isn't that large, and combined with a moderate batch size and sequence length, it fits within 16GB.

Below are my measured VRAM usage figures at various stages of running Qwen3-TTS LoRA on an RTX 5070 Ti (16GB):

| Stage | Measured VRAM Usage | 16GB Headroom |
|---|---|---|
| LLM r64 Training (batch 2) | ~10.6 GB | ~5GB remaining |
| Flow r64 all-linear Training | ~8 GB | ~8GB remaining |
| Audition Synthesis (loading full model in fp16) | ~5 GB | Plenty of room |

As you can see, even during the most demanding LLM r64 training stage, the peak usage only reaches about 10.6GB, leaving about 5GB of headroom on a 16GB card. In other words, 16GB isn't a tight squeeze; it's quite comfortable, with still room to increase the batch size or rank.

> **Don't understand the terms in the table? This section is for you.**
>
> -   **r64 / r32 (rank)**: This is LoRA's "capacity knob." LoRA doesn't touch the original massive model; instead, it adds a small, trainable "plugin" of parameters, and rank determines how large this plugin is. The higher the number, the more voice details it can learn, but it also consumes more VRAM and training time, and is more prone to "memorizing" training samples (overfitting) when data is insufficient. **r64 means rank=64, and r32 means rank=32. The number of plugin parameters for r32 is about half that of r64**, saving more VRAM and training faster, while the difference in voice fidelity is usually minor. When the dataset is small (10-30 minutes), r32 is often sufficient and less prone to overfitting. Go for r64 if you want to squeeze out a bit more detail. My table shows results for r64 with a peak of just 10.6GB, so switching to r32 would be even easier.
> -   **LLM Stage / Flow Stage**: Qwen3-TTS is actually two models working in tandem. The **LLM** is responsible for "text -> what to say and the rhythm of phrasing." The **Flow** (flow matching acoustic decoder) is responsible for "turning that into an actual sound waveform" (timbre, quality). You need to fine-tune both with LoRA for the voice to sound like the target person.
> -   **all-linear**: This refers to applying LoRA to "all linear layers" of the model, rather than just a few attention layers. Broader coverage means more complete learning, with slightly more parameters, which is why the "Flow all-linear" row is listed separately.

As for QLoRA (Quantized LoRA), it can theoretically reduce VRAM requirements by another ~75%, but it's not yet mature in the TTS field. Unsloth officially advises against using 4-bit QLoRA for the Qwen3.5 series because quantization errors significantly impact speech quality. Since running native LoRA on 16GB is already feasible, most people don't actually need QLoRA.

| Training Method | VRAM Requirement | Suitable For | Maturity |
|---|---|---|---|
| Inference | 8GB+ | Everyone | Stable |
| LoRA Fine-tuning | 16GB+ (tested) | Individuals / Small Teams | Stable |
| QLoRA | Theoretical 6-8GB | — | Stable, but quality is not as good |
| Full-parameter Fine-tuning | Hundreds of GB+ | Enterprises / Research Institutions | Stable but high cost |

In short, if you want to create a custom voice, a 16GB VRAM consumer-grade graphics card (like an RTX 5070 Ti, or an RTX 4060 Super 12GB) is enough to get started. If you want a larger batch size, a higher rank, or faster training, you can then upgrade to 24GB (RTX 3090 / 4090), but that's an option for "faster and better," not the entry-level requirement.

## II. Comparison of Open-Source TTS Models: Qwen, CosyVoice

The choice of base model affects the accent, quality ceiling, and the difficulty of subsequent training. Here are a few I've tested:

### Qwen3-TTS (Alibaba, released in January 2026)

It's quite feature-complete, supporting ten languages, with a speaker similarity of about 0.95 and a Chinese Character Error Rate (CER) of about 0.77%. However, in my experience, the generated voice still has a noticeable "mainland Chinese accent," and for Traditional Chinese, it has a tokenizer byte-fallback issue, leading to pronunciation errors for out-of-vocabulary (OOV) words.

### BreezyVoice (MediaTek, based on CosyVoice 2)

Its Taiwanese Hokkien speech effect is excellent (MOS reaching 5.0) and supports Bopomofo annotation for disambiguation. A major limitation is that the official release only provides inference scripts; it doesn't support secondary training, so you can't use it to customize your own voice.

### CosyVoice 3 (Alibaba, December 2025)

Currently the most balanced choice overall. It has 1.5B parameters and was trained on a million-hour-level dataset, with a Chinese CER of about 0.71%. Its voice is closer to a Taiwanese accent, and its emotional control is relatively natural. It also has a "pronunciation patching" feature, which allows you to directly override mispronounced words with Pinyin to correct them, without retraining.

| Comparison Item | Qwen3-TTS | CosyVoice 3 | BreezyVoice |
|---|---|---|---|
| Parameters | 1.7B | 1.5B | Based on CosyVoice 2 |
| Chinese CER | 0.77% | 0.71% | — |
| Accent Tendency | Leans toward mainland Chinese accent | Closer to Taiwanese accent | Best for Taiwanese Hokkien |
| Secondary Training | Supports LoRA | Supports LoRA | Inference only |
| Pronunciation Patching | Requires preprocessing | Built-in Pinyin override | Bopomofo annotation |
| Emotional Control | Average | More natural | — |

If your goal is custom training in a Traditional Chinese context, CosyVoice 3 is the most practical starting point right now.

## III. LoRA Fine-Tuning Implementation

### Why Not Just Use Voice Cloning (ICL)?

Zero-shot voice cloning (In-Context Learning) is indeed convenient, requiring only a 3-10 second reference audio clip to work, and the results are good for short sentences. But as sentences get longer, the timbre tends to drift; this is a common problem across all current models. Extending the reference audio to about 20 seconds can improve stability, but for consistently stable quality, LoRA fine-tuning remains the more reliable approach.

### Implementation Steps

**Step 1: Prepare Training Data (The Most Time-Consuming Step)**

The community commonly recommends 10-30 minutes of training data. You need to prepare clean, background-noise-free personal audio files and segment them into short 5-15 second clips.

> In my experience, the training itself can be completed in about 30 minutes to an hour. The real time-sink is preparing the audio files: cleaning up noise, segmenting clips, and proofreading transcripts. This prep work accounts for over 70% of the entire process.

**Step 2: Data Annotation**

Map each audio file to its accurate transcript, usually in the format of a manifest file:

```
audio_001.wav|你好，歡迎來到我的技術部落格。
audio_002.wav|今天要分享的是關於語音合成的實戰經驗。
```

**Step 3: Run LoRA Training**

Taking CosyVoice 3 as an example, the basic training command is as follows:

```bash
python train_lora.py \
  --model_path pretrained/CosyVoice3-0.5B \
  --data_dir ./training_data \
  --output_dir ./output/my_voice \
  --lora_rank 64 \
  --epochs 50 \
  --batch_size 4 \
  --learning_rate 1e-4
```

**Step 4: Validate Quality**

After training is complete, you can use a TTS-ASR self-refinement loop for initial automated validation: first have the model generate speech, then use Whisper to transcribe it back to text, and compare the phoneme error rate. According to a 2025 study, this method can achieve a 55.88% relative error reduction in a mixed Chinese-English context.

However, the final quality must still be judged by the human ear. Automation can filter out obviously problematic samples, but determining if the intonation is natural and the emotion is appropriate is still something humans are better at.

## IV. Handling Chinese Pronunciation

Two of the more troublesome problems with Chinese TTS are:

1.  Polyphonic characters: The two "好" characters in "好好學習" (hǎo hǎo xué xí - study hard) have different pronunciations.
2.  Specific vocabulary: "軟體" (ruǎntǐ) vs. "軟件" (ruǎnjiàn). The model's training data is primarily Simplified Chinese, so it hasn't seen many Traditional Chinese terms.

### Traditional Approach: Three-Layer Preprocessing

```
輸入文字 → 字典覆寫（phrase override）
         → OpenCC 繁轉簡
         → g2pW 轉漢語拼音
         → 送入 TTS 模型
```

This process can solve most problems, but the maintenance cost is not low. Every time a new pronunciation error is encountered, the dictionary must be updated manually. Polyphonic characters are disambiguated using neural network models like Polyphone BERT or g2pW.

### CosyVoice 3's Approach: Pronunciation Patching

CosyVoice 3's built-in pronunciation patching feature allows you to directly override the pronunciation of specific characters in the input text using Pinyin:

```
# 原始輸入（「軟體」的「體」可能被唸錯）
軟體工程師的日常

# 使用發音修補（拼音標註）
軟<phoneme ph="ti3">體</phoneme>工程師的日常
```

According to official tests, the correction rate of this method is close to 100%. It requires no code changes or retraining and can be handled directly at the input stage. This is quite practical for Traditional Chinese users.

## Conclusion

To summarize, here are three key points for training a personal AI voice:

1.  **Hardware**: A 16GB VRAM consumer-grade card (like an RTX 5070 Ti) is enough to get started; a 24GB card is not a must.
2.  **Model**: Start with CosyVoice 3, which offers a good balance of sound quality, accent, and trainability.
3.  **Data**: Spend time recording 10-30 minutes of high-quality training audio.

Although AI can already perform partial self-training and validation through TTS-ASR loops, the preparation of training data and the final auditory judgment of the results are still difficult to fully replace.

Also worth noting is that new technologies like LoRP-TTS (inference-time LoRA optimization) are making the process more automated. In the future, it might not be necessary to prepare training data in advance, as models could adapt to the target voice in real-time during inference. The overall barrier to entry should continue to decrease.

## References

-   [Qwen3-TTS Technical Report](https://arxiv.org/abs/2601.15621)
-   [CosyVoice 3 Paper](https://arxiv.org/abs/2505.17589)
-   [CosyVoice 3 Tech Guide](https://stable-learn.com/en/cosyvoice3-tech-guide/)
-   [BreezyVoice GitHub](https://github.com/mtkresearch/BreezyVoice)
-   [LoRP-TTS: Inference-Time LoRA](https://arxiv.org/abs/2502.07562)
-   [TTS-ASR Self-Refining Framework](https://arxiv.org/abs/2506.11130)
-   [Qwen3-TTS LoRA Fine-Tuning Guide](https://instavar.com/blog/ai-production-stack/LoRA_Finetuning_Qwen3_TTS_Custom_Voices)
-   [Unsloth TTS Fine-tuning Guide](https://unsloth.ai/docs/basics/text-to-speech-tts-fine-tuning)

---

## About this article and its author

Originally published on [Mark Ku's Tech Notes](https://blog.markkulab.net/en/post/ai-voice-clone-lora-training-guide)

License: [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) — when reusing or quoting, credit the author and link back to the original

### About the author

**[Mark Ku](https://blog.markkulab.net/en/author/mark-ku)** — Software Solution Provider

- 10+ years senior software engineer, now an AI Builder
- Focused on large-platform architecture — North-American e-commerce, AI SaaS subscription billing
- Combining AI Agents and automation to build evolvable product foundations

### Free tools built by the author

All of these are free to use:

- [Free PDF Sign Tool](https://blog.markkulab.net/en/tools/pdf-sign): Online PDF sign tool — draw, type, or upload a signature, then drag, resize, and download. Everything runs in your browser; nothing is uploaded.
- [VS Code Refactory](https://blog.markkulab.net/en/tools/refactory): Refactory is a VS Code refactoring extension: 34 actions plus a 37-rule code-smell inspection layer with a Code Health dashboard, across 18 languages, backed by 534 tests. It learns your repo's conventions: where interfaces live, where DI is registered, whether 'use client' belongs. It ranks files by git churn × complexity so you know what to fix first, and hands any smell to the Claude Code already on your machine. Free to use, and your source never leaves your computer.
- [DB-Kit Database Manager](https://blog.markkulab.net/en/tools/db-kit): DB-Kit is a lightweight, cross-platform database manager built with Tauri + Rust + React. Manage MySQL, MariaDB, PostgreSQL, SQL Server, Oracle, SQLite, MongoDB, Redis, Kafka, Elasticsearch and RabbitMQ from one consistent interface: passwords encrypted in the OS keychain, SSH tunnels, full CRUD, a visual query builder, stacked multi-statement result sets, cross-connection data transfer and compare/sync, Excel / CSV import & export, visualized execution plans, ER diagrams, scheduled backups, SQL stress testing with p50–p99 latency percentiles, a 15-rule SQL review engine, Kafka message browsing with monitoring & alerts, a bilingual UI (Traditional Chinese / English), a built-in AI assistant (natural-language SQL, AI review and tuning advice) and the dbk CLI. Free and open source (MIT), with installers for Windows, macOS and Linux.
- [VS Code Super Mermaid](https://blog.markkulab.net/en/tools/super-mermaid): Super Mermaid is a VS Code extension for beautiful Mermaid diagrams out of the box: auto-colored live preview, mouse pan & zoom, high-res PNG / SVG export, 21 templates and multiple themes. Free and open source (MIT).
- [React Super Mermaid](https://blog.markkulab.net/en/tools/react-super-mermaid): react-super-mermaid is an open-source React component library: render beautiful Mermaid diagrams with a single <MermaidViewer>, with built-in colorful / sketch themes, pan & zoom, in-diagram search, and high-res SVG / PNG export. Lightweight, SSR-safe, fully typed. Free and open source (MIT).
- [Jira / Confluence Super Mermaid](https://blog.markkulab.net/en/tools/jira-super-mermaid): An Atlassian Forge app: write Mermaid syntax directly inside a Jira issue or a Confluence page and get flowcharts, sequence diagrams, state machines and Gantt charts. 11 diagram types, SVG / PNG export, light and dark themes, full CJK support. Runs on Atlassian: your diagrams live in your own site and the app calls no third-party service. Free, coming soon to the Atlassian Marketplace.
- [Mermaid Live Preview](https://blog.markkulab.net/en/tools/mermaid-preview): Write Mermaid in your browser, see it render instantly, and share the whole diagram as a single link. No sign-up, nothing uploaded to a server, and mermaid.live share links work as-is.
- [React Intl Phone Number](https://blog.markkulab.net/en/tools/react-intl-phone-number): react-intl-phone-number is an open-source React component: framework-agnostic and antd-free, with E.164 in/out, a searchable flag / country-code dropdown, configurable validation levels (strict / mobile-strict / loose), themeable CSS, and i18n — phone logic powered by google-libphonenumber. Lightweight and fully typed. Free and open source (MIT).
- [Uptime Kuma Cluster](https://blog.markkulab.net/en/tools/uptime-kuma-cluster): Turn single-node Uptime Kuma into a highly available cluster: OpenResty + Lua smart load balancing, shared MariaDB state, health checks and automatic failover, plus cluster-management REST APIs. One Docker Compose command to start. Free and open source (MIT).
- [Special Education](https://blog.markkulab.net/en/education): Learning materials crafted for special education students

### Daily podcasts

- [Mark's Tech Insights — Daily AI News](https://blog.markkulab.net/en/category/tech-news): Daily curated AI and tech trends. Catch the latest developments via audio summaries — covering AI applications, software architecture, DevOps, and engineering practice. — RSS: https://blog.markkulab.net/feed.xml
- [AI股市蝦聊](https://blog.markkulab.net/en/category/ai-stock-chat): Every trading day, an AI-analyzed take on the Taiwan stock market, delivered as a two-host conversation covering the session and the next-day outlook. — RSS: https://blog.markkulab.net/ai-stock-chat/feed.xml
- [開源好物週報](https://blog.markkulab.net/en/category/open-source-weekly): A weekly two-host pick of free open-source tools surfaced from real Hacker News, GitHub, and Reddit buzz — what pain they solve and the fastest way to get started. — RSS: https://blog.markkulab.net/open-source-weekly/feed.xml

### Newsletter

[Subscribe to the newsletter](https://blog.markkulab.net/en/subscribe) — Be the first to know about new posts. No spam, unsubscribe anytime.