---
title: "🧪 6.1 M‑download 27B model: Unsloth‑edition Qwen 3.8 lets you run multimodal fine‑tuning on a 24 GB GPU | Local AI Lab"
description: "This issue's radar highlight is Unsloth's officially released Qwen3.8-27B-GGUF, with Hugging Face heat 200 and 6.1 million cumulative downloads, including mmproj multimodal projection and full LoRA training scripts, claimed to run Q4_K_M quantization on a 24GB GPU."
canonical_url: "https://blog.markkulab.net/en/local-ai-lab/local-ai-lab-2026-11-11"
author: "Mark Ku"
author_url: "https://blog.markkulab.net/en/author/mark-ku"
site: "Mark Ku's Tech Notes"
date_published: "2026-11-11T02:00:00.000Z"
category: "On-premises AI Lab"
tags: ["local-ai", "podcast", "地端模型", "lora", "開源模型", "Qwen3.8-27B", "Zonos3-mini", "Gemma-4-31B", "GGUF", "LoRA微調", "TTS"]
language: "en"
license: "CC BY 4.0"
license_url: "https://creativecommons.org/licenses/by/4.0/"
attribution: "when reusing or quoting, credit the author and link back to the original"
---

# 🧪 6.1 M‑download 27B model: Unsloth‑edition Qwen 3.8 lets you run multimodal fine‑tuning on a 24 GB GPU | Local AI Lab

## Opening
This week's radar highlight is Unsloth's official release of Qwen3.8-27B-GGUF, with 200 Hugging Face trending score and 6.1M cumulative downloads. It bundles mmproj multimodal projection and a complete LoRA training script, claiming Q4_K_M quantization runs on 24GB VRAM. Also featured: Zyphra's MoE-architecture lightweight TTS model Zonos3-mini, and community quantization expert mradermacher's Gemma 4 31B instruct package. Both are Apache-2.0 licensed with Traditional Chinese support. I'll tell you which suits offline customer-service voice and which handles government document extraction.

## This Week's Picks

### 1. unsloth/Qwen3.8-27B-GGUF: 27B Flagship with Built-in Multimodal & LoRA Scripts, Runs on 24GB VRAM
- **What is it**: Official Unsloth team release of Qwen3.8-27B quantized repo, 27B parameters, Apache-2.0 license. Provides 12 GGUF quantizations (Q2_K through Q8_0), plus mmproj multimodal projection files and one-click LoRA fine-tuning scripts. Deploys directly on llama.cpp, Ollama, LM Studio.
- **Use cases**:
  1. Indie devs with an RTX 3090 or 4090 (24GB VRAM) wanting an offline coding assistant and document Q&A system. Load Q4_K_M (~16.9 GB) and it runs; add mmproj and the model can read screenshots and charts for explanation.
  2. Engineers injecting proprietary domain knowledge can follow the official README example commands to complete Unsloth LoRA fine-tuning on a 16GB GPU with system RAM offloading, then quantize back to GGUF for deployment, full workflow without data leaving your machine.
  3. Mac users with 24GB+ unified memory can run `ollama run hf.co/unsloth/Qwen3.8-27B-GGUF:Q4_K_M` via Ollama for near-cloud Chinese and code generation quality (source: official README and community test reports, <https://blog.csdn.net/gitblog_00412/article/details/170436341>).
- **Does it run**: Official README lists 12 quantization sizes (Q2_K 11 GB ~ Q8_0 29.1 GB) and gives the rule of thumb "usable memory ≈ model size × 1.2~1.5": 12GB memory barely runs Q2_K, 16GB VRAM recommends Q3_K_L or IQ4_XS, 24GB VRAM sweet spot is Q4_K_S/M, 32GB+ handles Q6_K/Q8_0; Mac M-series 24GB+ unified memory runs Q4_K_M smoothly (source: official README with community references, <https://blog.csdn.net/gitblog_00412/article/details/170436341>).
- **This week's buzz**: HF trending 200, 6.1M cumulative downloads, 5.8k likes, 6 quantization/repackage variants in the same family this week.
- **Vs. peers**: Community quantizations of the same base model (e.g., mradermacher/ggml-org) only provide inference files. Unsloth version is maintained by the official team, exclusively includes complete LoRA training scripts and multimodal mmproj, ideal for engineers who want to "train then deploy" (source: model page description and Unsloth official tutorials).
- **How to get**: Official recommends downloading single .gguf from model page, launch with llama.cpp: `llama-server -m Qwen3.8-27B-Q4_K_M.gguf --mmproj Qwen3.8-27B.mmproj-Q8_0.gguf -c 8192 -ngl 999`; or directly via Ollama: `ollama run hf.co/unsloth/Qwen3.8-27B-GGUF:Q4_K_M` (source: model page README).
- **Link**: [https://huggingface.co/unsloth/Qwen3.8-27B-GGUF](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF)

### 2. Zyphra/ZONOS3-Mini: MoE-Architecture Lightweight Open-Source TTS, Runs Offline Chinese-English Speech Synthesis on 16GB VRAM
- **What is it**: Zyphra's third-gen lightweight text-to-speech model, using MoE sparse activation architecture. Parameter details unpublished but optimized for small-model high performance. Supports Chinese and English, formats include GGUF and safetensors, Apache-2.0 license for commercial use.
- **Use cases**:
  1. Backend engineers run Zonos3-mini GGUF quantized on 16GB VRAM to provide zero-latency, offline Chinese-English TTS for self-hosted customer service bots or voice notification systems.
  2. Content creators deploy locally on Mac (M-series 16GB+), pair with Gradio or WebUI to batch-convert articles into audiobooks, fully offline, no character limits, no API costs.
  3. Edge device developers (e.g., Jetson Orin, Raspberry Pi 5 + 16GB RAM) load GGUF via llama.cpp for fully offline voice assistant prototypes.
- **Does it run**: Official model card doesn't specify VRAM thresholds. Reference Zonos2 (same architecture) community tests: MoE sparse activation means 4-bit GGUF starts at ~2-3 GB VRAM, 8-bit at ~4-5 GB. Generation speed on consumer GPUs hits real-time factor <0.3. Above VRAM and speed numbers are from Zonos2 community tests; Zonos3-mini shares architecture for reference only, actual values may differ (source: Bilibili review video <https://www.bilibili.com/video/BV1Gypt64EnQ/>).
- **This week's buzz**: HF trending 196, 8.5k cumulative downloads, 224 likes.
- **Vs. peers**: Compared to Coqui XTTSv2, Bark, Fish Speech, and other open-source TTS, Zonos series leads with MoE sparse architecture, small parameters but fast synthesis and high naturalness. Zonos3-mini is Zyphra's local-deployment-optimized lightweight version, more permissive license (Apache-2.0 vs. Coqui's CPML), suitable for commercial self-hosting (source: Bilibili reviewer Hewkick's comprehensive Zonos2 evaluation).
- **How to get**: Model page provides GGUF files, launch with llama.cpp: `llama-server -m zonos3-mini-Q4_K_M.gguf -c 2048`; or reference Zonos2 official repo's `zonos serve` command to spin up OpenAI-compatible API (source: model page Files and https://github.com/Zyphra/Zonos2/ README).
- **Link**: [https://huggingface.co/Zyphra/ZONOS3-Mini](https://huggingface.co/Zyphra/ZONOS3-Mini)

### 3. mradermacher/gemma-4-31B-it-ARA-heretic-GGUF: Gemma 4 31B Multilingual Instruct, Top Pick for Traditional Chinese Document Processing
- **What is it**: Community quantization expert mradermacher's package of Google Gemma 4 31B instruct fine-tune (ARA-heretic version), 31B parameters, Apache-2.0 license, full GGUF quantization suite. Optimized for multilingual (including Traditional Chinese) instruction following and tool calling.
- **Use cases**:
  1. Engineers processing Traditional Chinese customer tickets and government documents load Q4_K_M (~18-19 GB) on 24GB VRAM for offline classification, summarization, and structured field extraction, zero data leakage risk.
  2. Multilingual product teams deploy on Mac Studio 64GB unified memory, can load higher-precision quantizations (e.g., Q6_K range ~20-24 GB, estimated from Q4_K_M 18.1 GiB and Q8_0 26.6 GiB), supporting Traditional Chinese, Japanese, Korean, and Southeast Asian languages in a single model service, simplifying deployment architecture.
  3. Developers comparing instruction-following quality can benchmark against Qwen3.8-27B, Llama-3.1-8B-Instruct on identical hardware with same benchmarks (e.g., IFEval, MT-Bench) to evaluate Gemma 4 on structured output and tool calling.
- **Does it run**: Official model card doesn't list hardware requirements. Reference same quantizer's notes on Qwen3.6-27B-GGUF: Q4_K_M ~18.1 GiB, Q8_0 ~26.6 GiB, BF16 ~50 GiB. Recommends 24GB VRAM for Q4_K_M, 32GB+ for Q8_0, Mac unified memory 32GB+ runs Q4_K_M smoothly (source: extrapolation from same author's quantization logic, <https://blog.csdn.net/gitblog_01081/article/details/170605107>).
- **This week's buzz**: HF trending 202, 96.3k cumulative downloads, 237 likes.
- **Vs. peers**: Eternal Digital's 2026/10 test of 16 open-source models: Gemma 4 31B scores 93-94% overall, perfect tool calling, Traditional Chinese expression outperforms most Chinese-origin models. vs. Qwen3.8-27B, Gemma 4 leads in instruction following, function calling, and multilingual consistency, but trails Qwen family slightly on code generation (source: <https://foreverwebs.com/blog/open-source-ai-model-comparison-taiwan-benchmark>).
- **How to get**: Model page offers GGUF split downloads, use with llama.cpp: `llama-server -m gemma-4-31b-it-ara-heretic-Q4_K_M.gguf -c 8192 -ngl 999`; or build Ollama Modelfile `FROM ./gemma-4-31b-it-ara-heretic-Q4_K_M.gguf` then `ollama create gemma4-31b-ara -f Modelfile` (source: model page Files and general llama.cpp/Ollama usage).
- **Link**: [https://huggingface.co/mradermacher/gemma-4-31B-it-ARA-heretic-GGUF](https://huggingface.co/mradermacher/gemma-4-31B-it-ARA-heretic-GGUF)

## Closing
All three models this week are Apache-2.0 licensed, support GGUF format, and run on consumer hardware, targeting "multimodal coding assistant," "offline speech synthesis," and "Traditional Chinese document multilingual processing" respectively. The one to try first is Unsloth's Qwen3.8-27B: it uniquely combines fine-tuning scripts with multimodal capability for maximum extensibility. Next issue we'll keep digging for open-source gems that fit in your own machine. See you then.

---

## About this article and its author

Originally published on [Mark Ku's Tech Notes](https://blog.markkulab.net/en/local-ai-lab/local-ai-lab-2026-11-11)

License: [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) — when reusing or quoting, credit the author and link back to the original

### About the author

**[Mark Ku](https://blog.markkulab.net/en/author/mark-ku)** — Software engineer

- 10+ years as a software engineer
- Built North-American e-commerce and AI SaaS subscription billing

### Free tools built by the author

All of these are free to use:

- [Free PDF Sign Tool](https://blog.markkulab.net/en/tools/pdf-sign): Online PDF sign tool — draw, type, or upload a signature, then drag, resize, and download. Everything runs in your browser; nothing is uploaded.
- [VS Code Refactory](https://blog.markkulab.net/en/tools/refactory): Refactory is a VS Code refactoring extension: 34 actions plus a 37-rule code-smell inspection layer with a Code Health dashboard, across 18 languages, backed by 534 tests. It learns your repo's conventions: where interfaces live, where DI is registered, whether 'use client' belongs. It ranks files by git churn × complexity so you know what to fix first, and hands any smell to the Claude Code already on your machine. Free to use, and your source never leaves your computer.
- [DB-Kit Database Manager](https://blog.markkulab.net/en/tools/db-kit): DB-Kit is a lightweight, cross-platform database manager built with Tauri + Rust + React. Manage MySQL, MariaDB, PostgreSQL, SQL Server, Oracle, SQLite, MongoDB, Redis, Kafka, Elasticsearch and RabbitMQ from one consistent interface: passwords encrypted in the OS keychain, SSH tunnels, full CRUD, a visual query builder, stacked multi-statement result sets, cross-connection data transfer, schema & data compare with sync SQL and schema snapshots, Excel / CSV import & export, visualized execution plans, ER diagrams, scheduled backups, SQL stress testing with p50–p99 latency percentiles, a 15-rule SQL review engine, Review & Run (AI review plus per-statement backups and an auto-generated rollback script), Kafka message browsing with monitoring & alerts, an SSH terminal with SFTP / FTP, Docker / Kubernetes, remote desktop (RDP / VNC / RustDesk), file / folder compare, stored procedure integration tests, six UI languages, a built-in AI assistant (a local CLI or any Anthropic / OpenAI-compatible API; natural-language SQL, AI review and tuning advice) and the dbk CLI. Free and open source (MIT), with installers for Windows, macOS and Linux.
- [VS Code Super Mermaid](https://blog.markkulab.net/en/tools/super-mermaid): Super Mermaid is a VS Code extension for beautiful Mermaid diagrams out of the box: auto-colored live preview, mouse pan & zoom, high-res PNG / SVG export, 21 templates and multiple themes. Free and open source (MIT).
- [React Super Mermaid](https://blog.markkulab.net/en/tools/react-super-mermaid): react-super-mermaid is an open-source React component library: render beautiful Mermaid diagrams with a single <MermaidViewer>, with built-in colorful / sketch themes, pan & zoom, in-diagram search, and high-res SVG / PNG export. Lightweight, SSR-safe, fully typed. Free and open source (MIT).
- [Jira / Confluence Super Mermaid](https://blog.markkulab.net/en/tools/jira-super-mermaid): An Atlassian Forge app: write Mermaid syntax directly inside a Jira issue or a Confluence page and get flowcharts, sequence diagrams, state machines and Gantt charts. 11 diagram types, SVG / PNG export, light and dark themes, full CJK support. Runs on Atlassian: your diagrams live in your own site and the app calls no third-party service. Free, coming soon to the Atlassian Marketplace.
- [Mermaid Live Preview](https://blog.markkulab.net/en/tools/mermaid-preview): Write Mermaid in your browser, see it render instantly, and share the whole diagram as a single link. No sign-up, nothing uploaded to a server, and mermaid.live share links work as-is.
- [React Intl Phone Number](https://blog.markkulab.net/en/tools/react-intl-phone-number): react-intl-phone-number is an open-source React component: framework-agnostic and antd-free, with E.164 in/out, a searchable flag / country-code dropdown, configurable validation levels (strict / mobile-strict / loose), themeable CSS, and i18n — phone logic powered by google-libphonenumber. Lightweight and fully typed. Free and open source (MIT).
- [Uptime Kuma Cluster](https://blog.markkulab.net/en/tools/uptime-kuma-cluster): Turn single-node Uptime Kuma into a highly available cluster: OpenResty + Lua smart load balancing, shared MariaDB state, health checks and automatic failover, plus cluster-management REST APIs. One Docker Compose command to start. Free and open source (MIT).
- [AI Podcast Cut](https://blog.markkulab.net/en/tools/ai-podcast-cut): Drop in a recording and it removes fillers and stutters, levels loudness segment by segment, and sends a second agent to review every cut. Cut points snap to word boundaries and zero crossings, every splice gets a fade, and sentence-end breaths are preserved. Desktop app for Windows, macOS and Linux. MIT licensed; the Windows installer bundles ffmpeg.
- [AI Video Cut](https://blog.markkulab.net/en/tools/ai-video-cut): Pick something in your video: type it (face, license plate, phone screen, logo), click points or drag a box, or let Claude Code / Codex pick it. AI Video Cut tracks it frame by frame, then mosaics or blurs it, recolors it, pins stickers and text to it, swaps a screen or poster for your own image or video, or removes it using background that other frames actually captured. Plus sequence editing, local captions and an AI assistant. Free, open-source (MIT) Windows desktop app (macOS / Linux experimental) that runs locally, nothing uploaded.
- [open-pos restaurant POS](https://blog.markkulab.net/en/tools/open-pos): One computer and one receipt printer is enough to open the shop. Your data lives on your own disk, no subscription, no lock-in, MIT licensed. Money is integer New Taiwan dollars with tax split by the statutory formula, so sales + tax always equals the total. Printing goes straight over ESC/POS on TCP 9100, with no vendor driver. Tauri + Rust + SQLite desktop app, v1.0 in development.
- [.NET e-commerce API template](https://blog.markkulab.net/en/tools/dotnet-ecommerce-api): An e-commerce backend API template built on .NET 10: Controller → Service → Repository layering, Autofac convention-based registration, SqlSugar for MySQL / MariaDB and SQL Server, Redis cache and message queue, Hangfire jobs, JWT auth, and an extensible payment provider module (ECPay, Newebpay). Apache 2.0.
- [Open Token Monitor team AI usage](https://blog.markkulab.net/en/tools/open-token-monitor): Collect token usage and equivalent cost from every employee's AI coding tools into a hub you host yourself: PostgreSQL, org roster, dashboards compared by company and department, and a reports API. Electron and Rust/Tauri clients upload every 30 minutes. MIT licensed, one Docker command.
- [ai-security-monitor defensive security](https://blog.markkulab.net/en/tools/ai-security-monitor): Your website is live. Who checks its certificates, security settings and known vulnerabilities? A self-hosted Security Hub brings website scans and host monitoring together, deduplicates findings and compares scans after fixes.

### Daily podcasts

- [Mark's Tech Insights — Daily AI News](https://blog.markkulab.net/en/category/tech-news): Daily curated AI and tech trends. Catch the latest developments via audio summaries — covering AI applications, software architecture, DevOps, and engineering practice. — RSS: https://blog.markkulab.net/feed.xml
- [AI股市蝦聊](https://blog.markkulab.net/en/category/ai-stock-chat): Every trading day, an AI-analyzed take on the Taiwan stock market, delivered as a two-host conversation covering the session and the next-day outlook. — RSS: https://blog.markkulab.net/ai-stock-chat/feed.xml
- [開源好物日報](https://blog.markkulab.net/en/category/open-source-weekly): A daily two-host pick of free open-source tools surfaced from real Hacker News, GitHub, and Reddit buzz — what pain they solve and the fastest way to get started. — RSS: https://blog.markkulab.net/open-source-weekly/feed.xml
- [AI 運動日報](https://blog.markkulab.net/en/category/sports-weekly): Two hosts talk NBA, MLB and world sport every day — scores, records and the stories behind them, from a Taiwanese fan perspective. — RSS: https://blog.markkulab.net/sports-weekly/feed.xml
- [AI 國際新聞快報](https://blog.markkulab.net/en/category/world-news): A daily two-host briefing that makes sense of the past 24 hours in world news — geopolitics, the global economy, conflict and security, disasters and climate — from a Taiwanese perspective, neutral and fully sourced. — RSS: https://blog.markkulab.net/world-news/feed.xml
- [地端 AI 實驗室](https://blog.markkulab.net/en/category/local-ai-lab): Every day, a two-host look at open-weight models and LoRAs you can actually run on your own machine: what they are for, whether your GPU can handle them, and how they compare — sourced from official model cards. — RSS: https://blog.markkulab.net/local-ai-lab/feed.xml
- [AI 行銷日報](https://blog.markkulab.net/en/category/ai-marketing): Every day, two hosts break down the latest in AI for marketing, advertising, SEO/GEO and content marketing: practical, never sponsored, every story sourced. — RSS: https://blog.markkulab.net/ai-marketing/feed.xml
- [AI 資安日報](https://blog.markkulab.net/en/category/ai-security): Every day, two hosts cover the latest where AI meets cybersecurity: LLM and AI agent vulnerabilities, prompt injection, model supply-chain risks and AI-powered attacks, from a defense-first angle with every story sourced. — RSS: https://blog.markkulab.net/ai-security/feed.xml

### Newsletter

[Subscribe to the newsletter](https://blog.markkulab.net/en/subscribe) — Be the first to know about new posts. No spam, unsubscribe anytime.