Introduction

LLaMA Factory is a very popular framework for fine-tuning your own models. Because I want to build my own customer service chatbot, I looked up some resources and decided to try fine-tuning some open-source AI models.

Potential AI Requirements

• The AI should be able to answer common customer service FAQs.
• The AI should be able to recommend pre-built computer packages.
• The AI should be able to configure a custom PC build based on a customer's budget.

Prerequisites

An Nvidia 4070 Ti Super graphics card

Hardware Requirements

Required Installations

First, download and install Python. Here, we'll install a relatively stable version, 3.12.

Next, download and install CUDA. This needs to match your graphics card version (it's best to install the driver at the same time).

P.S. If the driver installation fails, try unchecking Nsight VSE and Visual Studio Integration. This can happen if you have multiple versions of Visual Studio installed, which may cause conflicts.

Check Driver Version and Compatible CUDA Version

nvidia-smi 

Check Installed Driver and CUDA Version

nvcc -V

Then, install PyTorch

pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124

Test if Python can communicate with the GPU (test-gpu-test.py)

import torch
print(torch.cuda.is_available())  # 是否可以用gpu False不能，True可以
print(torch.cuda.device_count())  # gpu數量， 0就是沒有，1就是檢測到了

Installing LLaMA Factory

First, clone the repository

git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git

Change into the project directory

cd LLaMA-Factory

Install dependencies

pip install -e ".[torch,metrics]"

Verify the installation

llamafactory-cli version

Start the application

llamafactory-cli webui

After selecting a dataset, you can start training the model

P.S. Addendum - If you encounter the CUDA environment was not detected. error, you likely missed one of the previous installation steps.

References

• Reference 1
• Reference 2