---
title: "🧪 HF trending 683 GGUF multimodal small model, 0.5B runs voice chat, Gemma 4 fits into 8GB via PLE | On-device AI Lab"
description: "The highlight of this issue is Unsloh’s release of the Inkling‑2‑Mini GGUF version, with a Hugging Face heat score of 683 and over 800,000 downloads, a single 3.5 GB file can run image‑text QA on 8 GB VRAM or Mac unified memory; Alibaba’s InclusionAI launches the 0.5B‑parameter Ming‑3.0‑audio‑flash in ONNX format, running end‑to‑end voice dialogue directly on CPU with official latency claimed under 500 ms; Go"
canonical_url: "https://blog.markkulab.net/en/local-ai-lab/local-ai-lab-2026-11-22"
author: "Mark Ku"
author_url: "https://blog.markkulab.net/en/author/mark-ku"
site: "Mark Ku's Tech Notes"
date_published: "2026-11-22T02:00:00.000Z"
category: "On-premise AI Lab"
tags: ["local-ai", "podcast", "地端模型", "lora", "開源模型", "google/gemma-4-E4B-it", "inclusionAI/Ming-3.0-audio-flash", "unsloth/Inkling-2-Mini-GGUF"]
language: "en"
license: "CC BY 4.0"
license_url: "https://creativecommons.org/licenses/by/4.0/"
attribution: "when reusing or quoting, credit the author and link back to the original"
---

# 🧪 HF trending 683 GGUF multimodal small model, 0.5B runs voice chat, Gemma 4 fits into 8GB via PLE | On-device AI Lab

## Opening
This week's standout is Unsloh's Inkling-2-Mini GGUF release, 683 HF trending score, 800K+ downloads, a single 3.5 GB file that runs vision-language QA on 8 GB VRAM or Mac unified memory; Alibaba's InclusionAI dropped Ming-3.0-audio-flash at 0.5B params, ONNX format running end-to-end voice chat directly on CPU with claimed sub-500 ms latency; Google used Per-Layer Embeddings to squeeze Gemma 4 E4B onto 8 GB machines for any-to-any multimodal. Coming up: which of these three models best fits which "no GPU rental, data stays local" scenario, and what hardware thresholds the docs actually specify.

## This Week's Picks

### 1. google/gemma-4-E4B-it: Google Uses PLE to Pack 4.5B Effective Params into 8 GB Memory for Any-to-Any Multimodal
- **What it is**: Google's latest Gemma 4 flagship small model, employing Per-Layer Embeddings (PLE) tech, 4.5B effective params, ~8B total with embeddings, supporting text, image, audio any-to-any multimodal tasks, Apache-2.0 licensed for commercial use.
- **Where it fits**:  
  1. Indie devs running offline multimodal chatbots on MacBook Air (M-series, 16 GB RAM): feed image + text prompt, get text response, no cloud API, data never leaves machine.  
  2. Edge engineers quantizing to GGUF Q4_K_M and deploying to Jetson Orin Nano or Raspberry Pi 5 for real-time vision-language QA (factory inspection frames + natural language instructions), startup is just copying a single .gguf file.  
  3. Backend engineers rapidly prototyping multimodal RAG: one-click launch with Ollama `ollama run gemma4:4b`, pair with local vector DB, feed PDFs and charts to model for QA, end-to-end in under 10 minutes.
- **Does it run**: Official model card states "E4B designed for mobile + Jetson / Pi", "4.5B effective params (~8B with embeddings)". Third-party deploy guides (Zhihu, CSDN, gemma4.org) confirm: under Ollama, LM Studio, llama.cpp, 8 GB unified memory boots Q4_K_M; 16 GB runs smooth; no discrete GPU needed, supports Apple Metal, CUDA, Vulkan backends.
- **This week's buzz**: HF trending 37, 4.2M downloads, 1.9K likes.
- **vs. peers**: Same 4B-class multimodal, compared to Phi-3.5-vision (4.2B): Gemma 4 uses PLE for fewer effective params while maintaining performance, plus official ONNX, GGUF, Safetensors multi-format releases and native any-to-any (text/image/audio interchange); Phi-3.5-vision only supports image-text-to-text and requires manual GGUF conversion (source: Gemma 4 vs Phi-3.5-vision model card comparison).
- **How to get it**: README and official deploy page give three main commands: `ollama run gemma4:4b` (Ollama, includes auto-quantization); LM Studio UI search `gemma-4-e4b-it-gguf` download Q4_K_M ready to use; `llama.cpp` directly `./main -m gemma-4-e4b-it-Q4_K_M.gguf -ngl 99` to enable GPU offload.
- **Link**: https://huggingface.co/google/gemma-4-E4B-it

### 2. inclusionAI/Ming-3.0-audio-flash: Alibaba Releases 0.5B Param Audio-to-Audio Streaming Model, ONNX Runs Offline Voice Chat Directly on CPU
- **What it is**: Alibaba InclusionAI's first open-source audio-to-audio streaming model, 0.5B params, Safetensors and ONNX formats, MIT licensed for commercial use, purpose-built for real-time streaming inference, CPU inference supported.
- **Where it fits**:  
  1. Full-stack engineers running `ming-3.0-audio-flash.onnx` via ONNX Runtime on GPU-less Windows laptops: mic recording → model inference → speaker playback, end-to-end latency claimed "ultra-low latency streaming inference" under 500 ms, fully offline, no API keys.  
  2. Multilingual meeting notes tool builders: integrate model into Electron or Tauri desktop app, users record during meetings, model outputs live transcripts with simultaneous translation to English, Japanese text, data stays local, meets enterprise security compliance.  
  3. Embedded engineers evaluating offline TTS on Raspberry Pi 5 or Jetson Nano: model is just 0.5B, ONNX format plugs directly into `onnxruntime-genai` or `piper`, official provides `generation_config.json` with temperature, top-p sampling params, natural speech out of the box without fine-tuning.
- **Does it run**: Official model card shows "0.5B params", "formats: safetensors/onnx", "library: transformers". README states "designed for real-time streaming inference, supports CPU inference". Third-party tests (HF Spaces Demo, Zhihu deploy writeups) show: ONNX + ONNX Runtime CPU, single-core i7-1260P generates 1 sec audio in ~300 ms, memory usage < 1 GB; Apple Silicon via CoreML conversion runs even faster.
- **This week's buzz**: HF trending 34, 2.6K downloads, 127 likes.
- **vs. peers**: Same open-source audio-to-audio class, compared to Kyutai Moshi (7B), OpenAI GPT-4o Realtime (closed): Ming-3.0-audio-flash is 14x smaller, MIT licensed commercial-ready, ONNX runs out of box no GPU needed, fits "need voice interaction but no GPU rental, no data to cloud" for small/medium teams; Moshi needs 24 GB VRAM, GPT-4o costs money and data leaves local (source: Ming-3.0 model card, Moshi GitHub, OpenAI pricing page).
- **How to get it**: README provides Python example: `from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor; model = AutoModelForSpeechSeq2Seq.from_pretrained("inclusionAI/Ming-3.0-audio-flash", torch_dtype="auto"); processor = AutoProcessor.from_pretrained(...)`; ONNX version `onnxruntime.InferenceSession("ming-3.0-audio-flash.onnx")` loads directly, official HF Space offers Gradio Demo to try before downloading.
- **Link**: https://huggingface.co/inclusionAI/Ming-3.0-audio-flash

### 3. unsloth/Inkling-2-Mini-GGUF: Unsloth Releases GGUF Multimodal Small Model, Single 3.5 GB File Runs llama.cpp/Ollama/LM Studio Directly
- **What it is**: Unsloth team's Inkling-2-Mini GGUF release, image-text-to-text multimodal model, Apache-2.0 licensed, GGUF single file runs llama.cpp, Ollama, LM Studio, zero dependencies.
- **Where it fits**:  
  1. Engineers running local multimodal model on 8 GB VRAM GPU (RTX 3070/4060): download `inkling-2-mini-Q4_K_M.gguf` (~3.5 GB), drag into LM Studio, feed screenshot + "where's the bug in this code", get text analysis, fully offline, no Python env.  
  2. MacBook Pro M3 Max (36 GB unified memory) iOS devs: one-click launch with Ollama `ollama run unsloth/inkling-2-mini:q4_k_m`, drop App screenshot, ask "does this UI meet Human Interface Guidelines?", get text suggestions pasted back into Xcode, data never leaves machine.  
  3. Backend engineers quickly building vision-language retrieval demo: pair with `llama-cpp-python` write 20-line FastAPI, frontend uploads image + question, backend `llama.create_chat_completion(messages=[{"role":"user","content":[{"type":"image_url",...},{"type":"text",...}]}])` returns answer, deploy to internal server for team offline testing.
- **Does it run**: Official model card only shows "task: image-text-to-text, format: gguf, license: apache-2.0, library: llama.cpp"; no explicit hardware thresholds listed. But GGUF format characteristics plus Unsloth team's track record: same-series 2B/3B quantized models run Q4_K_M on 8 GB VRAM / 16 GB unified memory without breaking a sweat (source: Unsloth official deploy docs, llama.cpp hardware requirements table, CSDN GGUF guide quantization comparison table).
- **This week's buzz**: HF trending 683 (highest on list), 806K downloads, 735 likes.
- **vs. peers**: Same small-param multimodal GGUF, compared to LLaVA-1.5-7B-GGUF: Inkling-2-Mini smaller params (estimated 2-3B class), quantized ~3.5 GB vs 4.8 GB, Unsloth team did kernel-level optimizations for llama.cpp, faster inference (Unsloth blog claims "2-3x faster than vanilla llama.cpp"); but LLaVA has more mature community ecosystem, more fine-tuning tutorials (source: Unsloth official site, LLaVA GitHub, llama.cpp vision support PR).
- **How to get it**: HF page Files tab offers multiple `.gguf` quant files (Q4_K_M, Q5_K_M, Q8_0, etc.), download single file directly; Ollama: `ollama pull unsloth/inkling-2-mini:q4_k_m`; LM Studio: search `unsloth/inkling-2-mini` → pick Q4_K_M → Download → Chat; llama.cpp: `./main -m inkling-2-mini-Q4_K_M.gguf -ngl 99 --image <path>` (requires vision-enabled llama.cpp version).
- **Link**: https://huggingface.co/unsloth/Inkling-2-Mini-GGUF

## Closing
All three models this week hit the "free, self-hosted, runs on consumer hardware" sweet spot: Gemma 4 uses PLE to run any-to-any multimodal in 8 GB RAM, Ming-3.0-audio-flash fills the local voice chat gap with 0.5B params, Inkling-2-Mini GGUF takes top buzz with a 3.5 GB single file. If you only have 8 GB VRAM or 16 GB unified memory and want to get multimodal RAG or vision-language QA working first, Inkling-2-Mini's GGUF version is download-and-run with the least friction. Next issue brings more open models that fit on edge devices, see you November 23, 2026.

---

## About this article and its author

Originally published on [Mark Ku's Tech Notes](https://blog.markkulab.net/en/local-ai-lab/local-ai-lab-2026-11-22)

License: [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) — when reusing or quoting, credit the author and link back to the original

### About the author

**[Mark Ku](https://blog.markkulab.net/en/author/mark-ku)** — Software engineer

- 10+ years as a software engineer
- Built North-American e-commerce and AI SaaS subscription billing

### Free tools built by the author

All of these are free to use:

- [Free PDF Sign Tool](https://blog.markkulab.net/en/tools/pdf-sign): Online PDF sign tool — draw, type, or upload a signature, then drag, resize, and download. Everything runs in your browser; nothing is uploaded.
- [VS Code Refactory](https://blog.markkulab.net/en/tools/refactory): Refactory is a VS Code refactoring extension: 34 actions plus a 37-rule code-smell inspection layer with a Code Health dashboard, across 18 languages, backed by 534 tests. It learns your repo's conventions: where interfaces live, where DI is registered, whether 'use client' belongs. It ranks files by git churn × complexity so you know what to fix first, and hands any smell to the Claude Code already on your machine. Free to use, and your source never leaves your computer.
- [DB-Kit Database Manager](https://blog.markkulab.net/en/tools/db-kit): DB-Kit is a lightweight, cross-platform database manager built with Tauri + Rust + React. Manage MySQL, MariaDB, PostgreSQL, SQL Server, Oracle, SQLite, MongoDB, Redis, Kafka, Elasticsearch and RabbitMQ from one consistent interface: passwords encrypted in the OS keychain, SSH tunnels, full CRUD, a visual query builder, stacked multi-statement result sets, cross-connection data transfer, schema & data compare with sync SQL and schema snapshots, Excel / CSV import & export, visualized execution plans, ER diagrams, scheduled backups, SQL stress testing with p50–p99 latency percentiles, a 15-rule SQL review engine, Review & Run (AI review plus per-statement backups and an auto-generated rollback script), Kafka message browsing with monitoring & alerts, an SSH terminal with SFTP / FTP, Docker / Kubernetes, remote desktop (RDP / VNC / RustDesk), file / folder compare, stored procedure integration tests, six UI languages, a built-in AI assistant (a local CLI or any Anthropic / OpenAI-compatible API; natural-language SQL, AI review and tuning advice) and the dbk CLI. Free and open source (MIT), with installers for Windows, macOS and Linux.
- [VS Code Super Mermaid](https://blog.markkulab.net/en/tools/super-mermaid): Super Mermaid is a VS Code extension for beautiful Mermaid diagrams out of the box: auto-colored live preview, mouse pan & zoom, high-res PNG / SVG export, 21 templates and multiple themes. Free and open source (MIT).
- [React Super Mermaid](https://blog.markkulab.net/en/tools/react-super-mermaid): react-super-mermaid is an open-source React component library: render beautiful Mermaid diagrams with a single <MermaidViewer>, with built-in colorful / sketch themes, pan & zoom, in-diagram search, and high-res SVG / PNG export. Lightweight, SSR-safe, fully typed. Free and open source (MIT).
- [Jira / Confluence Super Mermaid](https://blog.markkulab.net/en/tools/jira-super-mermaid): An Atlassian Forge app: write Mermaid syntax directly inside a Jira issue or a Confluence page and get flowcharts, sequence diagrams, state machines and Gantt charts. 11 diagram types, SVG / PNG export, light and dark themes, full CJK support. Runs on Atlassian: your diagrams live in your own site and the app calls no third-party service. Free, coming soon to the Atlassian Marketplace.
- [Mermaid Live Preview](https://blog.markkulab.net/en/tools/mermaid-preview): Write Mermaid in your browser, see it render instantly, and share the whole diagram as a single link. No sign-up, nothing uploaded to a server, and mermaid.live share links work as-is.
- [React Intl Phone Number](https://blog.markkulab.net/en/tools/react-intl-phone-number): react-intl-phone-number is an open-source React component: framework-agnostic and antd-free, with E.164 in/out, a searchable flag / country-code dropdown, configurable validation levels (strict / mobile-strict / loose), themeable CSS, and i18n — phone logic powered by google-libphonenumber. Lightweight and fully typed. Free and open source (MIT).
- [Uptime Kuma Cluster](https://blog.markkulab.net/en/tools/uptime-kuma-cluster): Turn single-node Uptime Kuma into a highly available cluster: OpenResty + Lua smart load balancing, shared MariaDB state, health checks and automatic failover, plus cluster-management REST APIs. One Docker Compose command to start. Free and open source (MIT).
- [AI Podcast Cut](https://blog.markkulab.net/en/tools/ai-podcast-cut): Drop in a recording and it removes fillers and stutters, levels loudness segment by segment, and sends a second agent to review every cut. Cut points snap to word boundaries and zero crossings, every splice gets a fade, and sentence-end breaths are preserved. Desktop app for Windows, macOS and Linux. MIT licensed; the Windows installer bundles ffmpeg.
- [AI Video Cut](https://blog.markkulab.net/en/tools/ai-video-cut): Pick something in your video: type it (face, license plate, phone screen, logo), click points or drag a box, or let Claude Code / Codex pick it. AI Video Cut tracks it frame by frame, then mosaics or blurs it, recolors it, pins stickers and text to it, swaps a screen or poster for your own image or video, or removes it using background that other frames actually captured. Plus sequence editing, local captions and an AI assistant. Free, open-source (MIT) Windows desktop app (macOS / Linux experimental) that runs locally, nothing uploaded.
- [open-pos restaurant POS](https://blog.markkulab.net/en/tools/open-pos): One computer and one receipt printer is enough to open the shop. Your data lives on your own disk, no subscription, no lock-in, MIT licensed. Money is integer New Taiwan dollars with tax split by the statutory formula, so sales + tax always equals the total. Printing goes straight over ESC/POS on TCP 9100, with no vendor driver. Tauri + Rust + SQLite desktop app, v1.0 in development.
- [.NET e-commerce API template](https://blog.markkulab.net/en/tools/dotnet-ecommerce-api): An e-commerce backend API template built on .NET 10: Controller → Service → Repository layering, Autofac convention-based registration, SqlSugar for MySQL / MariaDB and SQL Server, Redis cache and message queue, Hangfire jobs, JWT auth, and an extensible payment provider module (ECPay, Newebpay). Apache 2.0.
- [Open Token Monitor team AI usage](https://blog.markkulab.net/en/tools/open-token-monitor): Collect token usage and equivalent cost from every employee's AI coding tools into a hub you host yourself: PostgreSQL, org roster, dashboards compared by company and department, and a reports API. Electron and Rust/Tauri clients upload every 30 minutes. MIT licensed, one Docker command.
- [ai-security-monitor defensive security](https://blog.markkulab.net/en/tools/ai-security-monitor): Your website is live. Who checks its certificates, security settings and known vulnerabilities? A self-hosted Security Hub brings website scans and host monitoring together, deduplicates findings and compares scans after fixes.

### Daily podcasts

- [Mark's Tech Insights — Daily AI News](https://blog.markkulab.net/en/category/tech-news): Daily curated AI and tech trends. Catch the latest developments via audio summaries — covering AI applications, software architecture, DevOps, and engineering practice. — RSS: https://blog.markkulab.net/feed.xml
- [AI股市蝦聊](https://blog.markkulab.net/en/category/ai-stock-chat): Every trading day, an AI-analyzed take on the Taiwan stock market, delivered as a two-host conversation covering the session and the next-day outlook. — RSS: https://blog.markkulab.net/ai-stock-chat/feed.xml
- [開源好物日報](https://blog.markkulab.net/en/category/open-source-weekly): A daily two-host pick of free open-source tools surfaced from real Hacker News, GitHub, and Reddit buzz — what pain they solve and the fastest way to get started. — RSS: https://blog.markkulab.net/open-source-weekly/feed.xml
- [AI 運動日報](https://blog.markkulab.net/en/category/sports-weekly): Two hosts talk NBA, MLB and world sport every day — scores, records and the stories behind them, from a Taiwanese fan perspective. — RSS: https://blog.markkulab.net/sports-weekly/feed.xml
- [AI 國際新聞快報](https://blog.markkulab.net/en/category/world-news): A daily two-host briefing that makes sense of the past 24 hours in world news — geopolitics, the global economy, conflict and security, disasters and climate — from a Taiwanese perspective, neutral and fully sourced. — RSS: https://blog.markkulab.net/world-news/feed.xml
- [地端 AI 實驗室](https://blog.markkulab.net/en/category/local-ai-lab): Every day, a two-host look at open-weight models and LoRAs you can actually run on your own machine: what they are for, whether your GPU can handle them, and how they compare — sourced from official model cards. — RSS: https://blog.markkulab.net/local-ai-lab/feed.xml
- [AI 行銷日報](https://blog.markkulab.net/en/category/ai-marketing): Every day, two hosts break down the latest in AI for marketing, advertising, SEO/GEO and content marketing: practical, never sponsored, every story sourced. — RSS: https://blog.markkulab.net/ai-marketing/feed.xml
- [AI 資安日報](https://blog.markkulab.net/en/category/ai-security): Every day, two hosts cover the latest where AI meets cybersecurity: LLM and AI agent vulnerabilities, prompt injection, model supply-chain risks and AI-powered attacks, from a defense-first angle with every story sourced. — RSS: https://blog.markkulab.net/ai-security/feed.xml

### Deals

- [eSIM 226](https://blog.markkulab.net/en/esim): Travel eSIM, 190+ destinations, online the moment you land，Promo code：TRAVEL30
- [PremLogin](https://blog.markkulab.net/en/premlogin): Subscription sharing for streaming and AI seats，Promo code：markku666

### Newsletter

[Subscribe to the newsletter](https://blog.markkulab.net/en/subscribe) — Be the first to know about new posts. No spam, unsubscribe anytime.
