---
title: "🧪 Gemma 4 31B full precision on M4 Max, Qwen 27B GGUF million downloads | On-device AI Lab"
description: "Google's latest open Gemma 4 series sees the 31B Dense version hit 42 heat and 6 million downloads on Hugging Face; crucially, CloudInsight tests show M4 Max 128GB unified memory runs full precision, a single RTX 4090 needs just 17GB VRAM with Q4_K_M quantization, and a mid-range Android phone fits the 3.1GB E2B 4-bit."
canonical_url: "https://blog.markkulab.net/en/local-ai-lab/local-ai-lab-2026-11-21"
author: "Mark Ku"
author_url: "https://blog.markkulab.net/en/author/mark-ku"
site: "Mark Ku's Tech Notes"
date_published: "2026-11-21T02:00:00.000Z"
category: "On-premise AI Lab"
tags: ["local-ai", "podcast", "地端模型", "lora", "開源模型", "gemma-4-31B-it", "Huihui-Qwen3.8-27B-abliterated-GGUF", "Qwen3.8-27B-pi-adapted"]
language: "en"
license: "CC BY 4.0"
license_url: "https://creativecommons.org/licenses/by/4.0/"
attribution: "when reusing or quoting, credit the author and link back to the original"
---

# 🧪 Gemma 4 31B full precision on M4 Max, Qwen 27B GGUF million downloads | On-device AI Lab

## Opening
Google's newly released Gemma 4 series, the 31B Dense variant sits at 42 heat and 6 million downloads on Hugging Face, but the real story is CloudInsight's hands-on testing: full-precision on M4 Max 128GB unified memory, Q4_K_M quantization on a single RTX 4090 in just 17GB VRAM, and the E2B 4-bit fitting into 3.1GB on mid-range Android phones. Running hotter in community buzz is huihui-ai's Huihui-Qwen3.8-27B-abliterated-GGUF at 69 heat and 4.5 million downloads, Apache-2.0 licensed with refusal removed, GGUF format natively supporting Ollama and llama.cpp. There's also a vision adapter weighing only a few hundred MB that turns your Qwen 27B multimodal in seconds. Coming up: how to pick among these three, how to run them, and where the hardware bars sit.

## This Week's Picks

### 1. google/gemma-4-31B-it: Google's Latest 2026 Open Model, 31B Dense Full-Precision on Consumer Mac, 26B MoE Quantized on Single 4090
- **What it is**: Flagship of the Gemma 4 series, 31B parameter Dense architecture, plus E2B, E4B, 26B MoE and other sizes. Full series Apache-2.0 licensed, supports transformers, llama.cpp, Ollama, vLLM and other mainstream inference engines.
- **Where it fits**:
  - Solo developer with a MacBook Pro M4 Max (128GB unified memory) wants to run 31B full-precision offline as a local Copilot alternative, code gen, refactoring, unit tests at zero token cost, data never leaves the machine.
  - Edge device engineer deploying fully offline voice assistant, translation, summarization on mid-range Android (8GB RAM), medical or finance field demos where data can't upload, E2B 4-bit needs only ~3.1GB memory.
  - Small team with a single RTX 4090 (24GB) runs 26B MoE at Q4_K_M, 8K context for codebase-scale RAG Q&A, replacing paid APIs to cut long-term inference cost.
- **Can it run**: Per CloudInsight's tested hardware threshold table: E2B 4-bit ~3.1GB memory (mid-range Android works); E4B Q4 ~5GB (8GB RAM laptop, no dGPU needed); 26B MoE Q4_K_M ~17GB VRAM (practical floor 24GB card like RTX 4090, 16GB cards need Q4_0 at ~14.4GB but context space gets extremely tight); 31B Dense BF16 needs 62GB (H100 single card), M4 Max 128GB unified memory runs full-precision 31B. Official model card README examples use `device_map='auto'` and `torch_dtype='auto'` for loading.
- **This week's buzz**: HF heat 42, 6M downloads, 4.4k likes.
- **Versus peers**: CloudInsight notes Gemma 4 26B MoE at Q4_K_M loses ~3-5% quality, beating same-tier Qwen3 27B same-quantization's 5-8% loss; 31B Dense full-precision trails Llama-3.1-70B slightly on reasoning benchmarks but at less than half the parameters, deployment bar drops dramatically.
- **How to get**: Official Gemma 4 page links to Hugging Face model card, README example: `pip install -U transformers accelerate && python -c "from transformers import AutoModelForCausalLM, AutoTokenizer; m=AutoModelForCausalLM.from_pretrained('google/gemma-4-31B-it', device_map='auto', torch_dtype='auto'); t=AutoTokenizer.from_pretrained('google/gemma-4-31B-it')"`.
- **Link**: https://huggingface.co/google/gemma-4-31B-it

### 2. huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF: Qwen 27B with Refusal Removed, GGUF Quant Pack, Single 4090 Runs Q6_K Near FP16 Quality
- **What it is**: GGUF quantized version based on Qwen3.8-27B base, abliterated to remove refusal mechanism, Apache-2.0 licensed for worry-free commercial use, provides Q2_K through Q8_0 quantization files, natively supports llama.cpp, Ollama, LM Studio.
- **Where it fits**:
  - Backend engineer on RTX 3060 12GB laptop loads Q4_K_M via Ollama (~16GB file, ~19-21GB runtime VRAM), offline code review, SQL generation, API doc writing, proprietary code stays off cloud.
  - Data scientist on Mac Studio M2 Ultra (96GB unified memory) runs Q5_K_M at 32K context, processes hundred-page PDF financial reports for long-form summarization and key metric extraction, zero token cost per inference.
  - Individual dev on RTX 4090 24GB desktop uses Q6_K (~20.5GB file) for near-FP16 quality, core of local chatbot, integrate RAG and Function Calling to build private knowledge assistant.
- **Can it run**: Official model card doesn't list concrete hardware thresholds; per community GGUF quantization experience (CSDN hands-on "Qwen3.8-27B Local Deployment: Complete Guide from Quantization Tiers to Hardware Config"): Q2_K ~11GB/14-16GB VRAM (16GB runs but quality takes a hit), Q4_K_M ~16GB/19-21GB (24GB VRAM is the sweet spot), Q6_K ~20.5GB/24-27GB (recommend 24GB+), Q8_0 ~27GB/31GB+ (needs 32GB+ or dual-card).
- **This week's buzz**: HF heat 69, 4.5M downloads, 2.5k likes.
- **Versus peers**: Compared to same-base prism-ml/ternary-bonsai-2-27b-gguf (ternary quantization, 9x compression, ~6.1GB), huihui version keeps full weight precision, balances size and quality via standard GGUF quantization, fits engineers who don't want to wrestle experimental quantization schemes; versus official Qwen3-27B-Instruct (FP16 ~54GB), abliterated version drops refusal layer, direct output better suits code and technical doc generation scenarios.
- **How to get**: Model page offers multiple GGUF quantization downloads; Ollama example: `ollama run hf.co/huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF:Q4_K_M`; llama.cpp example: `./llama-cli -m Huihui-Qwen3.8-27B-abliterated-Q4_K_M.gguf -p "你的提示詞" -n 512`.
- **Link**: https://huggingface.co/huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF

### 3. harness-hub/Qwen3.8-27B-pi-adapted: Vision Adapter Weighing Only Hundreds of MB, Gives Your Qwen 27B Image-Text QA in Seconds
- **What it is**: Vision alignment LoRA/adapter for Qwen3.8-27B text base (task: image-text-to-text), weights only a few hundred MB safetensors, Apache-2.0 licensed, requires base model, library tagged peft.
- **Where it fits**:
  - MLOps engineer who already has Qwen3.8-27B base downloads adapter weights, loads via PEFT `PeftModel.from_pretrained`, runs image-text QA, screenshot-to-code, UI mockup-to-frontend-components on 24GB VRAM.
  - Frontend engineer runs base Qwen3.8-27B-Q4_K_M locally on Ollama, then loads this adapter via llama.cpp `--lora` params, "paste error screenshot → get fix suggestion" offline debugging workflow, no re-downloading full vision model.
  - Tech writer feeds architecture diagrams, ER diagrams to adapter-loaded model, auto-generates Mermaid diagrams and text descriptions, supports Chinese/English output, cuts manual doc time.
- **Can it run**: Official model card doesn't list hardware thresholds; adapter weights tiny (safetensors, ~hundreds of MB), actual bar depends on base model quantization: paired with Qwen3.8-27B Q4_K_M (~16GB file, 19-21GB runtime VRAM), 24GB VRAM cards (RTX 3090/4090) or M2/M3 Max 32GB+ unified memory suffice; if base uses Q6_K, recommend 24GB+ VRAM.
- **This week's buzz**: HF heat 613, 2.4k downloads, 849 likes; same family has 8 quantized/repacked variants.
- **Versus peers**: Versus full vision model google/gemma-4-31b-it (31B Dense, needs 62GB BF16 or 24GB Q4), Qwen2-VL-72B (72B, even higher bar), this adapter adds vision at <1% parameter count, deploy cost only base model's extra VRAM; similar LoRA exists lodestones/kromoss (for Moss Image diffusion model), but this targets Qwen3.8 text base vision alignment, different use case.
- **How to get**: Model card marks library: peft, load example: `from transformers import AutoModelForCausalLM; from peft import PeftModel; base=AutoModelForCausalLM.from_pretrained('Qwen/Qwen3.8-27B', device_map='auto', torch_dtype='auto'); model=PeftModel.from_pretrained(base, 'harness-hub/Qwen3.8-27B-pi-adapted')`; Ollama/llama.cpp need confirm vision adapter format support.
- **Link**: https://huggingface.co/harness-hub/Qwen3.8-27B-pi-adapted

## Closing
All three picks point to the same trend: the "last mile" of bringing large model capability to consumer hardware is being bridged by quantization formats, MoE architectures, and even tiny adapters. Gemma 4 gives you a full spectrum, full-precision on Mac, quantized on 4090, phone-ready too. Huihui Qwen 27B GGUF uses mature quantization pipelines to turn 27B into a drop-in local engine. Pi-adapted unlocks multimodal with <1% parameter overhead. If you already have Qwen 27B base, try the adapter first for best value; want zero-config, ready-to-run code generation workhorse, Huihui GGUF Q4_K_M is the community consensus sweet spot. Next issue we'll track Llama 4 and Nemotron quantization progress. See you then.

---

## About this article and its author

Originally published on [Mark Ku's Tech Notes](https://blog.markkulab.net/en/local-ai-lab/local-ai-lab-2026-11-21)

License: [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) — when reusing or quoting, credit the author and link back to the original

### About the author

**[Mark Ku](https://blog.markkulab.net/en/author/mark-ku)** — Software engineer

- 10+ years as a software engineer
- Built North-American e-commerce and AI SaaS subscription billing

### Free tools built by the author

All of these are free to use:

- [Free PDF Sign Tool](https://blog.markkulab.net/en/tools/pdf-sign): Online PDF sign tool — draw, type, or upload a signature, then drag, resize, and download. Everything runs in your browser; nothing is uploaded.
- [VS Code Refactory](https://blog.markkulab.net/en/tools/refactory): Refactory is a VS Code refactoring extension: 34 actions plus a 37-rule code-smell inspection layer with a Code Health dashboard, across 18 languages, backed by 534 tests. It learns your repo's conventions: where interfaces live, where DI is registered, whether 'use client' belongs. It ranks files by git churn × complexity so you know what to fix first, and hands any smell to the Claude Code already on your machine. Free to use, and your source never leaves your computer.
- [DB-Kit Database Manager](https://blog.markkulab.net/en/tools/db-kit): DB-Kit is a lightweight, cross-platform database manager built with Tauri + Rust + React. Manage MySQL, MariaDB, PostgreSQL, SQL Server, Oracle, SQLite, MongoDB, Redis, Kafka, Elasticsearch and RabbitMQ from one consistent interface: passwords encrypted in the OS keychain, SSH tunnels, full CRUD, a visual query builder, stacked multi-statement result sets, cross-connection data transfer, schema & data compare with sync SQL and schema snapshots, Excel / CSV import & export, visualized execution plans, ER diagrams, scheduled backups, SQL stress testing with p50–p99 latency percentiles, a 15-rule SQL review engine, Review & Run (AI review plus per-statement backups and an auto-generated rollback script), Kafka message browsing with monitoring & alerts, an SSH terminal with SFTP / FTP, Docker / Kubernetes, remote desktop (RDP / VNC / RustDesk), file / folder compare, stored procedure integration tests, six UI languages, a built-in AI assistant (a local CLI or any Anthropic / OpenAI-compatible API; natural-language SQL, AI review and tuning advice) and the dbk CLI. Free and open source (MIT), with installers for Windows, macOS and Linux.
- [VS Code Super Mermaid](https://blog.markkulab.net/en/tools/super-mermaid): Super Mermaid is a VS Code extension for beautiful Mermaid diagrams out of the box: auto-colored live preview, mouse pan & zoom, high-res PNG / SVG export, 21 templates and multiple themes. Free and open source (MIT).
- [React Super Mermaid](https://blog.markkulab.net/en/tools/react-super-mermaid): react-super-mermaid is an open-source React component library: render beautiful Mermaid diagrams with a single <MermaidViewer>, with built-in colorful / sketch themes, pan & zoom, in-diagram search, and high-res SVG / PNG export. Lightweight, SSR-safe, fully typed. Free and open source (MIT).
- [Jira / Confluence Super Mermaid](https://blog.markkulab.net/en/tools/jira-super-mermaid): An Atlassian Forge app: write Mermaid syntax directly inside a Jira issue or a Confluence page and get flowcharts, sequence diagrams, state machines and Gantt charts. 11 diagram types, SVG / PNG export, light and dark themes, full CJK support. Runs on Atlassian: your diagrams live in your own site and the app calls no third-party service. Free, coming soon to the Atlassian Marketplace.
- [Mermaid Live Preview](https://blog.markkulab.net/en/tools/mermaid-preview): Write Mermaid in your browser, see it render instantly, and share the whole diagram as a single link. No sign-up, nothing uploaded to a server, and mermaid.live share links work as-is.
- [React Intl Phone Number](https://blog.markkulab.net/en/tools/react-intl-phone-number): react-intl-phone-number is an open-source React component: framework-agnostic and antd-free, with E.164 in/out, a searchable flag / country-code dropdown, configurable validation levels (strict / mobile-strict / loose), themeable CSS, and i18n — phone logic powered by google-libphonenumber. Lightweight and fully typed. Free and open source (MIT).
- [Uptime Kuma Cluster](https://blog.markkulab.net/en/tools/uptime-kuma-cluster): Turn single-node Uptime Kuma into a highly available cluster: OpenResty + Lua smart load balancing, shared MariaDB state, health checks and automatic failover, plus cluster-management REST APIs. One Docker Compose command to start. Free and open source (MIT).
- [AI Podcast Cut](https://blog.markkulab.net/en/tools/ai-podcast-cut): Drop in a recording and it removes fillers and stutters, levels loudness segment by segment, and sends a second agent to review every cut. Cut points snap to word boundaries and zero crossings, every splice gets a fade, and sentence-end breaths are preserved. Desktop app for Windows, macOS and Linux. MIT licensed; the Windows installer bundles ffmpeg.
- [AI Video Cut](https://blog.markkulab.net/en/tools/ai-video-cut): Pick something in your video: type it (face, license plate, phone screen, logo), click points or drag a box, or let Claude Code / Codex pick it. AI Video Cut tracks it frame by frame, then mosaics or blurs it, recolors it, pins stickers and text to it, swaps a screen or poster for your own image or video, or removes it using background that other frames actually captured. Plus sequence editing, local captions and an AI assistant. Free, open-source (MIT) Windows desktop app (macOS / Linux experimental) that runs locally, nothing uploaded.
- [open-pos restaurant POS](https://blog.markkulab.net/en/tools/open-pos): One computer and one receipt printer is enough to open the shop. Your data lives on your own disk, no subscription, no lock-in, MIT licensed. Money is integer New Taiwan dollars with tax split by the statutory formula, so sales + tax always equals the total. Printing goes straight over ESC/POS on TCP 9100, with no vendor driver. Tauri + Rust + SQLite desktop app, v1.0 in development.
- [.NET e-commerce API template](https://blog.markkulab.net/en/tools/dotnet-ecommerce-api): An e-commerce backend API template built on .NET 10: Controller → Service → Repository layering, Autofac convention-based registration, SqlSugar for MySQL / MariaDB and SQL Server, Redis cache and message queue, Hangfire jobs, JWT auth, and an extensible payment provider module (ECPay, Newebpay). Apache 2.0.
- [Open Token Monitor team AI usage](https://blog.markkulab.net/en/tools/open-token-monitor): Collect token usage and equivalent cost from every employee's AI coding tools into a hub you host yourself: PostgreSQL, org roster, dashboards compared by company and department, and a reports API. Electron and Rust/Tauri clients upload every 30 minutes. MIT licensed, one Docker command.
- [ai-security-monitor defensive security](https://blog.markkulab.net/en/tools/ai-security-monitor): Your website is live. Who checks its certificates, security settings and known vulnerabilities? A self-hosted Security Hub brings website scans and host monitoring together, deduplicates findings and compares scans after fixes.

### Daily podcasts

- [Mark's Tech Insights — Daily AI News](https://blog.markkulab.net/en/category/tech-news): Daily curated AI and tech trends. Catch the latest developments via audio summaries — covering AI applications, software architecture, DevOps, and engineering practice. — RSS: https://blog.markkulab.net/feed.xml
- [AI股市蝦聊](https://blog.markkulab.net/en/category/ai-stock-chat): Every trading day, an AI-analyzed take on the Taiwan stock market, delivered as a two-host conversation covering the session and the next-day outlook. — RSS: https://blog.markkulab.net/ai-stock-chat/feed.xml
- [開源好物日報](https://blog.markkulab.net/en/category/open-source-weekly): A daily two-host pick of free open-source tools surfaced from real Hacker News, GitHub, and Reddit buzz — what pain they solve and the fastest way to get started. — RSS: https://blog.markkulab.net/open-source-weekly/feed.xml
- [AI 運動日報](https://blog.markkulab.net/en/category/sports-weekly): Two hosts talk NBA, MLB and world sport every day — scores, records and the stories behind them, from a Taiwanese fan perspective. — RSS: https://blog.markkulab.net/sports-weekly/feed.xml
- [AI 國際新聞快報](https://blog.markkulab.net/en/category/world-news): A daily two-host briefing that makes sense of the past 24 hours in world news — geopolitics, the global economy, conflict and security, disasters and climate — from a Taiwanese perspective, neutral and fully sourced. — RSS: https://blog.markkulab.net/world-news/feed.xml
- [地端 AI 實驗室](https://blog.markkulab.net/en/category/local-ai-lab): Every day, a two-host look at open-weight models and LoRAs you can actually run on your own machine: what they are for, whether your GPU can handle them, and how they compare — sourced from official model cards. — RSS: https://blog.markkulab.net/local-ai-lab/feed.xml
- [AI 行銷日報](https://blog.markkulab.net/en/category/ai-marketing): Every day, two hosts break down the latest in AI for marketing, advertising, SEO/GEO and content marketing: practical, never sponsored, every story sourced. — RSS: https://blog.markkulab.net/ai-marketing/feed.xml
- [AI 資安日報](https://blog.markkulab.net/en/category/ai-security): Every day, two hosts cover the latest where AI meets cybersecurity: LLM and AI agent vulnerabilities, prompt injection, model supply-chain risks and AI-powered attacks, from a defense-first angle with every story sourced. — RSS: https://blog.markkulab.net/ai-security/feed.xml

### Deals

- [eSIM 226](https://blog.markkulab.net/en/esim): Travel eSIM, 190+ destinations, online the moment you land，Promo code：TRAVEL30
- [PremLogin](https://blog.markkulab.net/en/premlogin): Subscription sharing for streaming and AI seats，Promo code：markku666

### Newsletter

[Subscribe to the newsletter](https://blog.markkulab.net/en/subscribe) — Be the first to know about new posts. No spam, unsubscribe anytime.
