--- title: "Open-Source AI Fine-Tuning Tool - LLAMA Factory: Model Fine-Tuning" description: "This post documents the hands-on process of fine-tuning an open-source AI model with a local dataset using LLaMA Factory, and analyzes the limitations of consumer-grade GPUs on training precision." canonical_url: "https://blog.markkulab.net/en/post/refine-ai-by-llama" author: "Mark Ku" author_url: "https://blog.markkulab.net/en/author/mark-ku" site: "Mark Ku's Tech Notes" date_published: "2024-09-26 01:01:35 +0800" category: "AI" tags: ["llama factory", "fine-tune", "ai", "machine learning", "gpu", "huggingface"] language: "en" license: "CC BY 4.0" license_url: "https://creativecommons.org/licenses/by/4.0/" attribution: "when reusing or quoting, credit the author and link back to the original" --- # Open-Source AI Fine-Tuning Tool - LLAMA Factory: Model Fine-Tuning ## Picking Up from the Previous Llama Factory Fine-Tuning ## First, I plan to use a local dataset for training ### Define the training file acer_faqs.json in the local faqs folder ``` [ { "instruction": "Explain how much a good gaming PC costs.", "input": "How much is a good gaming PC?", "output": "There are many ways to define a good gaming PC, as different gamers have different needs. At Acer, you can purchase a solid, entry-level gaming PC for as little as $849. If you are looking for a high-performance PC that can handle the highest resolutions at high speeds, a good gaming PC would be around $4,000. On average, good gaming PCs range between $1,000 and $4,000." }, { "instruction": "List some good gaming PC options.", "input": "What are good gaming PCs?", "output": "Many manufacturers and builders offer good gaming PCs with quality components. Acer has a large inventory of great gaming PCs, including both Intel and AMD desktops. The latest 12th generation Intel desktops and Ryzen 9 desktops each provide some of the most seamless gaming experiences." }, { "instruction": "Define what a gaming PC is.", "input": "What is a gaming PC?", "output": "Gaming PCs are specially designed personal computers made for playing video games with high quality. They are more powerful and have more advanced components than typical computers to provide better graphics and working memory to handle the load of running games." }, { "instruction": "Where can someone buy a gaming PC?", "input": "Where to buy a gaming PC?", "output": "When considering buying a gaming PC, you have many options between in-store and online retailers. Acer offers great deals and discounts on gaming PCs that use top-tier components." } ] ``` ### Next, place the training data folder into LLaMA-Factory\data\ ![data set folder](https://blog.markkulab.net/content/markku/posts/refine-ai-by-llama/images/data-set-folder.png) ### Modify dataset_info.json to include the file we just defined. The acer_faqs dataset will then appear in the dataset dropdown menu. ``` { .... "acer_faqs": { "file_name": "acer_faqs.json" } ... } ``` ### Of course, you can also get open-source datasets to train your AI. [Huggingface's dataset](https://huggingface.co/datasets?sort=trending&search=taiwan) You can also use open-source datasets to train your AI model by modifying dataset_info.json. ## Starting the Training ![Starting the training with a TUF RTX 3080 10g](https://blog.markkulab.net/content/markku/posts/refine-ai-by-llama/images/start-training.png) ### Next, let's test the fine-tuned AI model ![image](https://blog.markkulab.net/content/markku/posts/refine-ai-by-llama/images/verify-fine-tune.png) I tried a few datasets, including the default wiki_faq dataset. It was pretty funny watching the AI I spent half a day training just spout nonsense. ## A quick Google search revealed that many people have run into this issue of nonsensical answers. I suspect that the VRAM on a typical PC graphics card is only sufficient for training these stripped-down versions of open-source models. To get more accurate results, you probably need to use the full model and train it on dedicated AI chips. ![hardware requirement](https://blog.markkulab.net/content/markku/posts/refine-ai-by-llama/images/hardware-requirement.png) ## Final Thoughts After watching the AI I spent half a day training do nothing but give nonsensical answers, it seems that if the goal is to build a functional AI bot, using pre-trained models and AI Agent technologies is a much more achievable path for now. To train an AI yourself, you need a massive amount of data. Fine-tuning large open-source models has steep requirements for graphics cards, and there are many complex parameters to learn. I don't think AI training technology is yet easy enough to operate or accessible enough for the average company to readily adopt. It makes you wonder just how much money and effort were poured into building the major AI models we see today. I'm a complete beginner when it comes to AI training, so if I've misunderstood anything, please feel free to discuss it in the comments or send me a private message. --- ## About this article and its author Originally published on [Mark Ku's Tech Notes](https://blog.markkulab.net/en/post/refine-ai-by-llama) License: [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) — when reusing or quoting, credit the author and link back to the original ### About the author **[Mark Ku](https://blog.markkulab.net/en/author/mark-ku)** — Software Solution Provider - 10+ years senior software engineer, now an AI Builder - Focused on large-platform architecture — North-American e-commerce, AI SaaS subscription billing - Combining AI Agents and automation to build evolvable product foundations ### Free tools built by the author All of these are free to use: - [Free PDF Sign Tool](https://blog.markkulab.net/en/tools/pdf-sign): Online PDF sign tool — draw, type, or upload a signature, then drag, resize, and download. Everything runs in your browser; nothing is uploaded. - [VS Code Refactory](https://blog.markkulab.net/en/tools/refactory): Refactory is a VS Code refactoring extension: 34 actions plus a 37-rule code-smell inspection layer with a Code Health dashboard, across 18 languages, backed by 534 tests. It learns your repo's conventions: where interfaces live, where DI is registered, whether 'use client' belongs. It ranks files by git churn × complexity so you know what to fix first, and hands any smell to the Claude Code already on your machine. Free to use, and your source never leaves your computer. - [DB-Kit Database Manager](https://blog.markkulab.net/en/tools/db-kit): DB-Kit is a lightweight, cross-platform database manager built with Tauri + Rust + React. Manage MySQL, MariaDB, PostgreSQL, SQL Server, Oracle, SQLite, MongoDB, Redis, Kafka, Elasticsearch and RabbitMQ from one consistent interface: passwords encrypted in the OS keychain, SSH tunnels, full CRUD, a visual query builder, stacked multi-statement result sets, cross-connection data transfer and compare/sync, Excel / CSV import & export, visualized execution plans, ER diagrams, scheduled backups, SQL stress testing with p50–p99 latency percentiles, a 15-rule SQL review engine, Kafka message browsing with monitoring & alerts, a bilingual UI (Traditional Chinese / English), a built-in AI assistant (natural-language SQL, AI review and tuning advice) and the dbk CLI. Free and open source (MIT), with installers for Windows, macOS and Linux. - [VS Code Super Mermaid](https://blog.markkulab.net/en/tools/super-mermaid): Super Mermaid is a VS Code extension for beautiful Mermaid diagrams out of the box: auto-colored live preview, mouse pan & zoom, high-res PNG / SVG export, 21 templates and multiple themes. Free and open source (MIT). - [React Super Mermaid](https://blog.markkulab.net/en/tools/react-super-mermaid): react-super-mermaid is an open-source React component library: render beautiful Mermaid diagrams with a single , with built-in colorful / sketch themes, pan & zoom, in-diagram search, and high-res SVG / PNG export. Lightweight, SSR-safe, fully typed. Free and open source (MIT). - [Jira / Confluence Super Mermaid](https://blog.markkulab.net/en/tools/jira-super-mermaid): An Atlassian Forge app: write Mermaid syntax directly inside a Jira issue or a Confluence page and get flowcharts, sequence diagrams, state machines and Gantt charts. 11 diagram types, SVG / PNG export, light and dark themes, full CJK support. Runs on Atlassian: your diagrams live in your own site and the app calls no third-party service. Free, coming soon to the Atlassian Marketplace. - [Mermaid Live Preview](https://blog.markkulab.net/en/tools/mermaid-preview): Write Mermaid in your browser, see it render instantly, and share the whole diagram as a single link. No sign-up, nothing uploaded to a server, and mermaid.live share links work as-is. - [React Intl Phone Number](https://blog.markkulab.net/en/tools/react-intl-phone-number): react-intl-phone-number is an open-source React component: framework-agnostic and antd-free, with E.164 in/out, a searchable flag / country-code dropdown, configurable validation levels (strict / mobile-strict / loose), themeable CSS, and i18n — phone logic powered by google-libphonenumber. Lightweight and fully typed. Free and open source (MIT). - [Uptime Kuma Cluster](https://blog.markkulab.net/en/tools/uptime-kuma-cluster): Turn single-node Uptime Kuma into a highly available cluster: OpenResty + Lua smart load balancing, shared MariaDB state, health checks and automatic failover, plus cluster-management REST APIs. One Docker Compose command to start. Free and open source (MIT). - [Special Education](https://blog.markkulab.net/en/education): Learning materials crafted for special education students ### Daily podcasts - [Mark's Tech Insights — Daily AI News](https://blog.markkulab.net/en/category/tech-news): Daily curated AI and tech trends. Catch the latest developments via audio summaries — covering AI applications, software architecture, DevOps, and engineering practice. — RSS: https://blog.markkulab.net/feed.xml - [AI股市蝦聊](https://blog.markkulab.net/en/category/ai-stock-chat): Every trading day, an AI-analyzed take on the Taiwan stock market, delivered as a two-host conversation covering the session and the next-day outlook. — RSS: https://blog.markkulab.net/ai-stock-chat/feed.xml - [開源好物週報](https://blog.markkulab.net/en/category/open-source-weekly): A weekly two-host pick of free open-source tools surfaced from real Hacker News, GitHub, and Reddit buzz — what pain they solve and the fastest way to get started. — RSS: https://blog.markkulab.net/open-source-weekly/feed.xml ### Newsletter [Subscribe to the newsletter](https://blog.markkulab.net/en/subscribe) — Be the first to know about new posts. No spam, unsubscribe anytime.