---
title: "Open-source AI Fine-tuning Tool - LLAMA Factory Environment Setup"
description: "A step-by-step guide on how to install Python, CUDA, PyTorch, and LLaMA Factory to build a complete development environment for fine-tuning AI models in a local NVIDIA GPU environment."
canonical_url: "https://blog.markkulab.net/en/post/set-up-llama-factory-enviorment"
author: "Mark Ku"
author_url: "https://blog.markkulab.net/en/author/mark-ku"
site: "Mark Ku's Tech Notes"
date_published: "2024-09-07 01:01:35 +0800"
category: "AI"
tags: ["llama factory", "ai", "fine-tune", "cuda", "pytorch", "gpu"]
language: "en"
license: "CC BY 4.0"
license_url: "https://creativecommons.org/licenses/by/4.0/"
attribution: "when reusing or quoting, credit the author and link back to the original"
---

# Open-source AI Fine-tuning Tool - LLAMA Factory Environment Setup

## Introduction
LLaMA Factory is a very popular framework for fine-tuning your own models. Because I want to build my own customer service chatbot, I looked up some resources and decided to try fine-tuning some open-source AI models.

## Potential AI Requirements
* The AI should be able to answer common customer service FAQs.
* The AI should be able to recommend pre-built computer packages.
* The AI should be able to configure a custom PC build based on a customer's budget.

## [Prerequisites](https://github.com/hiyouga/LLaMA-Factory)
### An Nvidia 4070 Ti Super graphics card
![Video card](https://blog.markkulab.net/content/markku/posts/set-up-llama-factory-enviorment/images/video-card.jpg)
### Hardware Requirements
| Method              | Bits | 7B   | 13B  | 30B  | 70B   | 110B  | 8x7B  | 8x22B  |
|---------------------|------|------|------|------|-------|-------|-------|--------|
| Full AMP            | 7    | 120GB| 240GB| 600GB| 1200GB| 2000GB| 900GB | 2400GB |
| Full                | 16   | 60GB | 120GB| 300GB| 600GB | 900GB | 400GB | 1200GB |
| Freeze              | 16   | 20GB | 40GB | 80GB | 200GB | 360GB | 160GB | 400GB  |
| LoRA/GaLore/BAdam   | 16   | 16GB | 32GB | 64GB | 160GB | 240GB | 120GB | 320GB  |
| QLoRA               | 8    | 10GB | 20GB | 40GB | 80GB  | 140GB | 60GB  | 160GB  |
| QLoRA               | 4    | 6GB  | 12GB | 24GB | 48GB  | 72GB  | 30GB  | 96GB   |
| QLoRA               | 2    | 4GB  | 8GB  | 16GB | 24GB  | 48GB  | 18GB  | 48GB   |

## Required Installations

### First, [download and install Python](https://www.python.org/downloads/). Here, we'll install a relatively stable version, 3.12.
### Next, [download and install CUDA](https://developer.nvidia.com/cuda-toolkit). This needs to match your graphics card version (it's best to install the driver at the same time).
![CUDA](https://blog.markkulab.net/content/markku/posts/set-up-llama-factory-enviorment/images/cuda-toolkit-download.png)

P.S. If the driver installation fails, try unchecking Nsight VSE and Visual Studio Integration. This can happen if you have multiple versions of Visual Studio installed, which may cause conflicts.
![install cuda](https://blog.markkulab.net/content/markku/posts/set-up-llama-factory-enviorment/images/install-cuda.png)

### Check Driver Version and Compatible CUDA Version

```
nvidia-smi 
```
![test cuda version](https://blog.markkulab.net/content/markku/posts/set-up-llama-factory-enviorment/images/test-cuda-version.png)

### Check Installed Driver and CUDA Version
```
nvcc -V
```

### Then, install [PyTorch](https://pytorch.org/)
```
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124
```
### Test if Python can communicate with the GPU (test-gpu-test.py)
```
import torch
print(torch.cuda.is_available())  # 是否可以用gpu False不能，True可以
print(torch.cuda.device_count())  # gpu數量， 0就是沒有，1就是檢測到了
```
![test gpu result](https://blog.markkulab.net/content/markku/posts/set-up-llama-factory-enviorment/images/test-gpu-result.png)

## Installing LLaMA Factory
### First, clone the repository
```
git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git
```
### Change into the project directory
```
cd LLaMA-Factory
```
### Install dependencies
```
pip install -e ".[torch,metrics]"
```
### Verify the installation
```
llamafactory-cli version
```
### Start the application
```
llamafactory-cli webui
```
### After selecting a dataset, you can start training the model
![llamafactory cli webui](https://blog.markkulab.net/content/markku/posts/set-up-llama-factory-enviorment/images/llamafactory-cli-webui.png)![llamafactory cli webui console](https://blog.markkulab.net/content/markku/posts/set-up-llama-factory-enviorment/images/llamafactory-cli-webui-console.png)

### P.S. Addendum - If you encounter the `CUDA environment was not detected.` error, you likely missed one of the previous installation steps.
### References
* [Reference 1](https://medium.com/@anannannan0102/pytorch-%E6%83%B3%E7%94%A8gpu%E8%B7%91ml%E7%92%B0%E5%A2%83%E5%8D%BB%E8%A3%9D%E4%B8%8D%E5%A5%BD-%E7%9C%8B%E5%AE%8C%E9%80%99%E7%AF%87%E5%B8%B6%E4%BD%A0%E9%81%BF%E9%96%8B%E5%90%84%E7%A8%AE%E9%9B%B7-3bf259fc7396)
* [Reference 2](https://blog.csdn.net/lx15983596831/article/details/140466261)

---

## About this article and its author

Originally published on [Mark Ku's Tech Notes](https://blog.markkulab.net/en/post/set-up-llama-factory-enviorment)

License: [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) — when reusing or quoting, credit the author and link back to the original

### About the author

**[Mark Ku](https://blog.markkulab.net/en/author/mark-ku)** — Software Solution Provider

- 10+ years senior software engineer, now an AI Builder
- Focused on large-platform architecture — North-American e-commerce, AI SaaS subscription billing
- Combining AI Agents and automation to build evolvable product foundations

### Free tools built by the author

All of these are free to use:

- [Free PDF Sign Tool](https://blog.markkulab.net/en/tools/pdf-sign): Online PDF sign tool — draw, type, or upload a signature, then drag, resize, and download. Everything runs in your browser; nothing is uploaded.
- [VS Code Refactory](https://blog.markkulab.net/en/tools/refactory): Refactory is a VS Code refactoring extension: 34 actions plus a 37-rule code-smell inspection layer with a Code Health dashboard, across 18 languages, backed by 534 tests. It learns your repo's conventions: where interfaces live, where DI is registered, whether 'use client' belongs. It ranks files by git churn × complexity so you know what to fix first, and hands any smell to the Claude Code already on your machine. Free to use, and your source never leaves your computer.
- [DB-Kit Database Manager](https://blog.markkulab.net/en/tools/db-kit): DB-Kit is a lightweight, cross-platform database manager built with Tauri + Rust + React. Manage MySQL, MariaDB, PostgreSQL, SQL Server, Oracle, SQLite, MongoDB, Redis, Kafka, Elasticsearch and RabbitMQ from one consistent interface: passwords encrypted in the OS keychain, SSH tunnels, full CRUD, a visual query builder, stacked multi-statement result sets, cross-connection data transfer and compare/sync, Excel / CSV import & export, visualized execution plans, ER diagrams, scheduled backups, SQL stress testing with p50–p99 latency percentiles, a 15-rule SQL review engine, Kafka message browsing with monitoring & alerts, a bilingual UI (Traditional Chinese / English), a built-in AI assistant (natural-language SQL, AI review and tuning advice) and the dbk CLI. Free and open source (MIT), with installers for Windows, macOS and Linux.
- [VS Code Super Mermaid](https://blog.markkulab.net/en/tools/super-mermaid): Super Mermaid is a VS Code extension for beautiful Mermaid diagrams out of the box: auto-colored live preview, mouse pan & zoom, high-res PNG / SVG export, 21 templates and multiple themes. Free and open source (MIT).
- [React Super Mermaid](https://blog.markkulab.net/en/tools/react-super-mermaid): react-super-mermaid is an open-source React component library: render beautiful Mermaid diagrams with a single <MermaidViewer>, with built-in colorful / sketch themes, pan & zoom, in-diagram search, and high-res SVG / PNG export. Lightweight, SSR-safe, fully typed. Free and open source (MIT).
- [Jira / Confluence Super Mermaid](https://blog.markkulab.net/en/tools/jira-super-mermaid): An Atlassian Forge app: write Mermaid syntax directly inside a Jira issue or a Confluence page and get flowcharts, sequence diagrams, state machines and Gantt charts. 11 diagram types, SVG / PNG export, light and dark themes, full CJK support. Runs on Atlassian: your diagrams live in your own site and the app calls no third-party service. Free, coming soon to the Atlassian Marketplace.
- [Mermaid Live Preview](https://blog.markkulab.net/en/tools/mermaid-preview): Write Mermaid in your browser, see it render instantly, and share the whole diagram as a single link. No sign-up, nothing uploaded to a server, and mermaid.live share links work as-is.
- [React Intl Phone Number](https://blog.markkulab.net/en/tools/react-intl-phone-number): react-intl-phone-number is an open-source React component: framework-agnostic and antd-free, with E.164 in/out, a searchable flag / country-code dropdown, configurable validation levels (strict / mobile-strict / loose), themeable CSS, and i18n — phone logic powered by google-libphonenumber. Lightweight and fully typed. Free and open source (MIT).
- [Uptime Kuma Cluster](https://blog.markkulab.net/en/tools/uptime-kuma-cluster): Turn single-node Uptime Kuma into a highly available cluster: OpenResty + Lua smart load balancing, shared MariaDB state, health checks and automatic failover, plus cluster-management REST APIs. One Docker Compose command to start. Free and open source (MIT).
- [Special Education](https://blog.markkulab.net/en/education): Learning materials crafted for special education students

### Daily podcasts

- [Mark's Tech Insights — Daily AI News](https://blog.markkulab.net/en/category/tech-news): Daily curated AI and tech trends. Catch the latest developments via audio summaries — covering AI applications, software architecture, DevOps, and engineering practice. — RSS: https://blog.markkulab.net/feed.xml
- [AI股市蝦聊](https://blog.markkulab.net/en/category/ai-stock-chat): Every trading day, an AI-analyzed take on the Taiwan stock market, delivered as a two-host conversation covering the session and the next-day outlook. — RSS: https://blog.markkulab.net/ai-stock-chat/feed.xml
- [開源好物週報](https://blog.markkulab.net/en/category/open-source-weekly): A weekly two-host pick of free open-source tools surfaced from real Hacker News, GitHub, and Reddit buzz — what pain they solve and the fastest way to get started. — RSS: https://blog.markkulab.net/open-source-weekly/feed.xml

### Newsletter

[Subscribe to the newsletter](https://blog.markkulab.net/en/subscribe) — Be the first to know about new posts. No spam, unsubscribe anytime.