On-premises AI Lab

🧪 6.1 M‑download 27B model: Unsloth‑edition Qwen 3.8 lets you run multimodal fine‑tuning on a 24 GB GPU | Local AI Lab

Mark Ku

November 11, 2026•17 min read

Voice

Opening

This week's radar highlight is Unsloth's official release of Qwen3.8-27B-GGUF, with 200 Hugging Face trending score and 6.1M cumulative downloads. It bundles mmproj multimodal projection and a complete LoRA training script, claiming Q4_K_M quantization runs on 24GB VRAM. Also featured: Zyphra's MoE-architecture lightweight TTS model Zonos3-mini, and community quantization expert mradermacher's Gemma 4 31B instruct package. Both are Apache-2.0 licensed with Traditional Chinese support. I'll tell you which suits offline customer-service voice and which handles government document extraction.

This Week's Picks

1. unsloth/Qwen3.8-27B-GGUF: 27B Flagship with Built-in Multimodal & LoRA Scripts, Runs on 24GB VRAM

What is it: Official Unsloth team release of Qwen3.8-27B quantized repo, 27B parameters, Apache-2.0 license. Provides 12 GGUF quantizations (Q2_K through Q8_0), plus mmproj multimodal projection files and one-click LoRA fine-tuning scripts. Deploys directly on llama.cpp, Ollama, LM Studio.
Use cases:
1. Indie devs with an RTX 3090 or 4090 (24GB VRAM) wanting an offline coding assistant and document Q&A system. Load Q4_K_M (~16.9 GB) and it runs; add mmproj and the model can read screenshots and charts for explanation.
2. Engineers injecting proprietary domain knowledge can follow the official README example commands to complete Unsloth LoRA fine-tuning on a 16GB GPU with system RAM offloading, then quantize back to GGUF for deployment, full workflow without data leaving your machine.
3. Mac users with 24GB+ unified memory can run ollama run hf.co/unsloth/Qwen3.8-27B-GGUF:Q4_K_M via Ollama for near-cloud Chinese and code generation quality (source: official README and community test reports, https://blog.csdn.net/gitblog_00412/article/details/170436341).
Does it run: Official README lists 12 quantization sizes (Q2_K 11 GB _{Q8_0 29.1 GB) and gives the rule of thumb "usable memory ≈ model size × 1.2}1.5": 12GB memory barely runs Q2_K, 16GB VRAM recommends Q3_K_L or IQ4_XS, 24GB VRAM sweet spot is Q4_K_S/M, 32GB+ handles Q6_K/Q8_0; Mac M-series 24GB+ unified memory runs Q4_K_M smoothly (source: official README with community references, https://blog.csdn.net/gitblog_00412/article/details/170436341).
This week's buzz: HF trending 200, 6.1M cumulative downloads, 5.8k likes, 6 quantization/repackage variants in the same family this week.
Vs. peers: Community quantizations of the same base model (e.g., mradermacher/ggml-org) only provide inference files. Unsloth version is maintained by the official team, exclusively includes complete LoRA training scripts and multimodal mmproj, ideal for engineers who want to "train then deploy" (source: model page description and Unsloth official tutorials).
How to get: Official recommends downloading single .gguf from model page, launch with llama.cpp: llama-server -m Qwen3.8-27B-Q4_K_M.gguf --mmproj Qwen3.8-27B.mmproj-Q8_0.gguf -c 8192 -ngl 999; or directly via Ollama: ollama run hf.co/unsloth/Qwen3.8-27B-GGUF:Q4_K_M (source: model page README).
Link: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF

2. Zyphra/ZONOS3-Mini: MoE-Architecture Lightweight Open-Source TTS, Runs Offline Chinese-English Speech Synthesis on 16GB VRAM

What is it: Zyphra's third-gen lightweight text-to-speech model, using MoE sparse activation architecture. Parameter details unpublished but optimized for small-model high performance. Supports Chinese and English, formats include GGUF and safetensors, Apache-2.0 license for commercial use.
Use cases:
1. Backend engineers run Zonos3-mini GGUF quantized on 16GB VRAM to provide zero-latency, offline Chinese-English TTS for self-hosted customer service bots or voice notification systems.
2. Content creators deploy locally on Mac (M-series 16GB+), pair with Gradio or WebUI to batch-convert articles into audiobooks, fully offline, no character limits, no API costs.
3. Edge device developers (e.g., Jetson Orin, Raspberry Pi 5 + 16GB RAM) load GGUF via llama.cpp for fully offline voice assistant prototypes.
Does it run: Official model card doesn't specify VRAM thresholds. Reference Zonos2 (same architecture) community tests: MoE sparse activation means 4-bit GGUF starts at _{2-3 GB VRAM, 8-bit at}4-5 GB. Generation speed on consumer GPUs hits real-time factor <0.3. Above VRAM and speed numbers are from Zonos2 community tests; Zonos3-mini shares architecture for reference only, actual values may differ (source: Bilibili review video https://www.bilibili.com/video/BV1Gypt64EnQ/).
This week's buzz: HF trending 196, 8.5k cumulative downloads, 224 likes.
Vs. peers: Compared to Coqui XTTSv2, Bark, Fish Speech, and other open-source TTS, Zonos series leads with MoE sparse architecture, small parameters but fast synthesis and high naturalness. Zonos3-mini is Zyphra's local-deployment-optimized lightweight version, more permissive license (Apache-2.0 vs. Coqui's CPML), suitable for commercial self-hosting (source: Bilibili reviewer Hewkick's comprehensive Zonos2 evaluation).
How to get: Model page provides GGUF files, launch with llama.cpp: llama-server -m zonos3-mini-Q4_K_M.gguf -c 2048; or reference Zonos2 official repo's zonos serve command to spin up OpenAI-compatible API (source: model page Files and https://github.com/Zyphra/Zonos2/ README).
Link: https://huggingface.co/Zyphra/ZONOS3-Mini

3. mradermacher/gemma-4-31B-it-ARA-heretic-GGUF: Gemma 4 31B Multilingual Instruct, Top Pick for Traditional Chinese Document Processing

What is it: Community quantization expert mradermacher's package of Google Gemma 4 31B instruct fine-tune (ARA-heretic version), 31B parameters, Apache-2.0 license, full GGUF quantization suite. Optimized for multilingual (including Traditional Chinese) instruction following and tool calling.
Use cases:
1. Engineers processing Traditional Chinese customer tickets and government documents load Q4_K_M (~18-19 GB) on 24GB VRAM for offline classification, summarization, and structured field extraction, zero data leakage risk.
2. Multilingual product teams deploy on Mac Studio 64GB unified memory, can load higher-precision quantizations (e.g., Q6_K range ~20-24 GB, estimated from Q4_K_M 18.1 GiB and Q8_0 26.6 GiB), supporting Traditional Chinese, Japanese, Korean, and Southeast Asian languages in a single model service, simplifying deployment architecture.
3. Developers comparing instruction-following quality can benchmark against Qwen3.8-27B, Llama-3.1-8B-Instruct on identical hardware with same benchmarks (e.g., IFEval, MT-Bench) to evaluate Gemma 4 on structured output and tool calling.
Does it run: Official model card doesn't list hardware requirements. Reference same quantizer's notes on Qwen3.6-27B-GGUF: Q4_K_M _{18.1 GiB, Q8_0}26.6 GiB, BF16 ~50 GiB. Recommends 24GB VRAM for Q4_K_M, 32GB+ for Q8_0, Mac unified memory 32GB+ runs Q4_K_M smoothly (source: extrapolation from same author's quantization logic, https://blog.csdn.net/gitblog_01081/article/details/170605107).
This week's buzz: HF trending 202, 96.3k cumulative downloads, 237 likes.
Vs. peers: Eternal Digital's 2026/10 test of 16 open-source models: Gemma 4 31B scores 93-94% overall, perfect tool calling, Traditional Chinese expression outperforms most Chinese-origin models. vs. Qwen3.8-27B, Gemma 4 leads in instruction following, function calling, and multilingual consistency, but trails Qwen family slightly on code generation (source: https://foreverwebs.com/blog/open-source-ai-model-comparison-taiwan-benchmark).
How to get: Model page offers GGUF split downloads, use with llama.cpp: llama-server -m gemma-4-31b-it-ara-heretic-Q4_K_M.gguf -c 8192 -ngl 999; or build Ollama Modelfile FROM ./gemma-4-31b-it-ara-heretic-Q4_K_M.gguf then ollama create gemma4-31b-ara -f Modelfile (source: model page Files and general llama.cpp/Ollama usage).
Link: https://huggingface.co/mradermacher/gemma-4-31B-it-ARA-heretic-GGUF

Closing

All three models this week are Apache-2.0 licensed, support GGUF format, and run on consumer hardware, targeting "multimodal coding assistant," "offline speech synthesis," and "Traditional Chinese document multilingual processing" respectively. The one to try first is Unsloth's Qwen3.8-27B: it uniquely combines fine-tuning scripts with multimodal capability for maximum extensibility. Next issue we'll keep digging for open-source gems that fit in your own machine. See you then.