---
title: "🧪 The 321B GLM-5.3-Flash won't fit on a single GPU, but this 9B model's popularity shot up to 113 | On-Premise AI Lab"
description: "GLM-5.3-Flash has 6.3M downloads on Hugging Face, 3.1k likes, MIT license, with the page listing 321B parameters, and the official model card even dedicates a section to \"Serve GLM-5.3-Flash Locally\" to teach you self-hosting, though a single consumer-grade GPU probably won't fit it."
canonical_url: "https://blog.markkulab.net/en/local-ai-lab/local-ai-lab-2026-11-05"
author: "Mark Ku"
author_url: "https://blog.markkulab.net/en/author/mark-ku"
site: "Mark Ku's Tech Notes"
date_published: "2026-11-05T02:00:00.000Z"
category: "On-Premises AI Lab"
tags: ["local-ai", "podcast", "地端模型", "lora", "開源模型", "GLM53Flash", "Qwopus38_27B_Coder_Flash_GGUF", "Qwen35_9B_Defiant_Fable", "HuggingFace", "GGUF", "llamacpp"]
language: "en"
license: "CC BY 4.0"
license_url: "https://creativecommons.org/licenses/by/4.0/"
attribution: "when reusing or quoting, credit the author and link back to the original"
---

# 🧪 The 321B GLM-5.3-Flash won't fit on a single GPU, but this 9B model's popularity shot up to 113 | On-Premise AI Lab

## Opening

GLM-5.3-Flash has 6.3M downloads on Hugging Face, 3.1k likes, MIT license, and the page lists 321B parameters. The official model card even dedicates a section to "Serve GLM-5.3-Flash Locally" to walk you through self-hosting, though a single consumer GPU almost certainly won't fit it. The other two in this episode are more down to earth: a 27B coding agent in GGUF, and a 9B writing model that the author clocked at 130 t/s and up on his own 5090 with LM Studio. As for why the "download champion" isn't the "HF trending champion," I'll get to that shortly.

## This Episode's Picks

### 1. zai-org/GLM-5.3-Flash: MIT-licensed flagship natively multimodal open weights, where the bottleneck is RAM, not your GPU

- **What it is**: Sourced from this show's "local model radar" segment, corresponding to the zai-org/GLM-5.3-Flash page on Hugging Face. The official model card states this is the first natively multimodal model in the GLM-5 series. The HF page lists 321B parameters with BF16 and F8_E4M3 tensor types, while the official Introduction describes 320B total parameters with 18B active. License is MIT, weights are public, no application required.

- **Where you'd use it**:
  - **Backend for a "nothing leaves the building" multimodal assistant on your corporate intranet**: the model card's image-text-to-text pipeline example literally passes in an image plus "describe this image." Picture a pile of internal system screenshots, report charts, and flow diagrams that you want to turn into a queryable knowledge portal, except compliance won't let you send anything to an external API. This model is positioned exactly for standing up your own endpoint.
  - **Evaluating whether to point your own agent at a self-hosted endpoint**: the official Introduction claims it approaches Claude Opus 4.8 on coding and agentic benchmarks, and the footnotes spell out the eval setup in considerable detail, for example DeepSWE using the mini-swe-agent harness at 400K context, and Terminal-Bench 2.1 using Claude Code 2.1.207. If you're building a coding or agent toolchain, those settings are worth comparing against your own harness conditions rather than just reading the scores.
  - **Server-side developers who need to control token cost and latency**: a Note in the model card explains you can use `reasoning_effort` to switch the thinking budget between low, high, and max, with max as the default, and recommends explicitly passing `clear_thinking=true` for production chat scenarios. In other words, the team left you a tunable "save tokens" switch, which anyone running a live service will appreciate.

- **Can you actually run it**: to be clear up front, the official model card lists no VRAM or RAM numbers at all. It only enumerates six deployment paths under "Serve GLM-5.3-Flash Locally": SGLang, vLLM, TokenSpeed, Transformers, KTransformers, and Unsloth, with KTransformers and Unsloth each having their own tutorial links. As for third-party numbers, DataCamp's KTransformers tutorial says the official KTransformers guide recommends reserving at least 350GB of system memory, with FP8 weights around 306 GiB. Unsloth, meanwhile, claims on an HF discussion that its dynamic quantization can run 3-bit in 128GB of memory and 1-bit in 100GB. Both of those are third-party claims, not figures stated on zai-org's official model card. What follows is this show's judgment, not a conclusion from the sources and not a measured result: since no official numbers exist, our rough tier estimate based on 321B parameters plus FP8/dynamic quantization is that this is a self-hosting project only viable on a large-memory workstation or Mac Studio class machine. If you have a single consumer GPU, skip it.

- **Buzz this episode**: HF trending score 62, 6.3M downloads, 3.1k likes. One clarification so this doesn't sound contradictory in a minute: the **highest download count** this episode belongs to GLM-5.3-Flash, but the **highest HF trending score** goes to the third pick, the 9B (score 113). Downloads and trending score are two different metrics, one measures cumulative volume and the other is the platform's composite heat score, so don't compare them directly.

- **How it compares**: against deepseek-ai/DeepSeek-V4.1-Flash-1017, which has comparable buzz this period, DeepSeek's official model card comparison table lists GLM-5.3 as a competitor and shows DeepSeek-V4.1-Flash-1017 scoring higher on Terminal-Bench 4.0, DeepSWE v1.1, and other columns. That's DeepSeek's framing. That said, the GLM-5.3-Flash page lists 321B parameters with 18B active, noticeably smaller than DeepSeek's model listed at 763B, which puts the bar meaningfully lower for anyone wanting to self-host.

- **How to get it**: the model card's vLLM section says to `pip install vllm` then run `vllm serve "zai-org/GLM-5.3-Flash"`; for SGLang it's `python3 -m sglang.launch_server --model-path "zai-org/GLM-5.3-Flash"`. Both bring up an OpenAI-compatible API. The page also has Docker and community quantization (Browse Quantizations) entry points.

- **Link**: https://huggingface.co/zai-org/GLM-5.3-Flash

---

### 2. Jackrong/Qwopus3.8-27B-Coder-Flash-GGUF: the 27B sweet spot for a local coding agent, and the author is refreshingly conservative about it

- **What it is**: also from this episode's "local model radar." The model card states this is a coding-oriented fine-tune of unsloth/Qwen3.8-27B (derived from Qwen/Qwen3.8-27B). The HF page lists 27B parameters, GGUF format, Apache-2.0 license, quantized files provided directly, no application required.

- **Where you'd use it**:
  - **Running a local coding agent on your own machine**: the README spells out the target workflow as Read → Reason → Call Tools → Edit → Test → Recover → Deliver, with training data including roughly 20K agent interaction and tool-call samples, mostly distilled from Qwen3.8-Flash-Max. In other words, it's aimed not at "finish this line for me" but at the full loop of picking a tool, reading the output, revising the plan, and shipping the result.
  - **Teams whose code can't leave the intranet**: the llama.cpp and Ollama commands on the HF page point straight at the Q4_K_M quant, so you can stand up an OpenAI-compatible endpoint internally and then wire in local terminal coding tools using the Pi, OpenClaw, and Hermes Agent config examples shown on the page. For projects where the client contract explicitly forbids putting code in the cloud, this route is very practical.
  - **People grinding algorithm problems and doing refactors**: the model card says that within roughly 200K code and complex-reasoning samples, the algorithm and code-generation portion (teacher models listed as Qwen3.8-Flash-Max, Qwen3.8-Max, and DeepSeek-V4-Flash) focused training on algorithm implementation, edge cases, refactoring, and debugging.

- **Can you actually run it**: no official VRAM or memory requirements are listed. The model card only has a "Checkpoint and runtime" section plus Transformers and vLLM inference examples, and the llama.cpp, Ollama, Docker Model Runner, and Lemonade commands on the page all default to pulling the Q4_K_M tag. What follows is this show's rough judgment, not a claim from the source and not a measured result: with no official numbers, our estimate based on 27B parameters plus Q4_K_M is that this is comfortable on "a GPU with a lot of VRAM, or a machine with generous unified memory." Before committing, actually load Q4_K_M once on your own hardware to confirm, and please don't buy a GPU based on this tiering alone.

- **Buzz this episode**: HF trending score 61, 23.8k downloads, 62 likes. Far smaller volume than the GLM model, but it's one of the few community fine-tunes this episode that actually has something to say for itself.

- **How it compares**: the author states in the model card that it's a coding fine-tune of unsloth/Qwen3.8-27B, differing from stock Qwen3.8-27B by the addition of distilled agent interaction and tool-call data. Credit where it's due: the card explicitly says this round of evaluation **did not measure** any "reduction in reasoning tokens," so the Flash in the name should not be read as a verified speedup. That caveat comes from the author himself.

- **How to get it**: the README and HF page give `llama serve -hf Jackrong/Qwopus3.8-27B-Coder-Flash-GGUF:Q4_K_M` to start a local OpenAI-compatible server, or `ollama run hf.co/Jackrong/Qwopus3.8-27B-Coder-Flash-GGUF:Q4_K_M`. Apache-2.0, no application required.

- **Link**: https://huggingface.co/Jackrong/Qwopus3.8-27B-Coder-Flash-GGUF

---

### 3. DavidAU/Qwen3.5-9B-The-Defiant-Fable-Uncensored-Heretic-NEO-IMATRIX-MAX-MTP-GGUF: the only one this episode you can genuinely run on a single gaming GPU, but it's fully uncensored

- **What it is**: also from this episode's "local model radar." The author describes it as a 9B model produced through multi-stage fine-tuning and merging, GGUF format, Apache-2.0, with the card listing 256k context and the README noting vision is enabled. Let's get this out of the way first: this is a **fully uncensored** model with no built-in content refusal boundaries. Output content and usage responsibility are entirely on you, so think hard before putting it anywhere near a corporate workflow.

- **Where you'd use it**:
  - **A fully offline writing and editing assistant when all you have is one gaming GPU**: 9B plus GGUF, and the author says it runs in all the standard AI apps. Combined with the card's 256k context, it suits the "I have a tens-of-thousands-of-words draft and want to hand the whole thing over for a pass" use case, all without leaving your machine.
  - **Local image Q&A**: the README says vision is enabled, but there's an easy trap here, you must **separately download an mmproj file** and put it in the same folder as the GGUF before the model can accept images. Grab only the main file and you'll think vision is broken.
  - **Local LLM enthusiasts who care about output speed and are willing to tune**: the author provides both regular and MTP (multi-token prediction) NEO IMATRIX quants, and recommends keeping temp below 1 and rep pen at 1 for the MTP version, falling back to the regular quant if token acceptance drops below 50%.
  - **Engineers who want a drop-in for an existing workflow**: the card lists recommended sampling settings, thinking mode at temp 1.0 / top_p 0.95 / top_k 20, temp 0.6 for precise coding tasks, instruct mode at temp 0.7 / top_p 0.80, and notes that reasoning defaults to xhigh and automatically drops to medium when you switch to instruct.

- **Can you actually run it**: no official minimum VRAM is listed. The hardware hints the author mentions are: all quants are NEO IMATRIX (he claims 2-4% better accuracy than ordinary GGUF), output tensors kept at 16-bit, MTP tensors set to Q8_0, and a recommendation to set context to at least 8k to 16k. The speed figures are **the author's own measurements on his own setup**: he writes that on a 5090 with Windows 11 and LM Studio, the regular Q4_K_S GGUF runs around 130 t/s, and the MTP version (2 tokens, 60% acceptance rate) can exceed 185 t/s, and he himself notes speeds vary by GPU, system, and app. Treat these as reference points for his environment, not as what your machine will do.

- **Buzz this episode**: HF trending score 113, 2.0M downloads, 1.4k likes. As mentioned earlier, this is the **highest HF trending score** of the three this episode, while the highest download count is still GLM-5.3-Flash's 6.3M. Two different metrics.

- **How it compares**: the author posts a self-run benchmark table on the card (arc/c, arc/e, boolq, hellaswag, openbookqa, piqa, winogrande), claiming this 9B beats base models like Qwen3.5-9B-Instruct and Qwen3.5-27B-Instruct on 7 of them, and in some cases approaches Qwen3.6-27B. These are entirely the author's own tests and claims, and he himself adds a caveat: beating a 27B on benchmarks doesn't mean it will beat a 27B on every task. That caveat is honestly more worth remembering than the scores.

- **How to get it**: the local commands on the HF page are `llama serve -hf DavidAU/Qwen3.5-9B-The-Defiant-Fable-Uncensored-Heretic-NEO-IMATRIX-MAX-MTP-GGUF:Q4_K_M`, or `ollama run hf.co/DavidAU/...:Q4_K_M`, and you can also just open it in LM Studio or Jan. For image input, remember to separately fetch the mmproj file and place it in the same folder, per the README.

- **Link**: https://huggingface.co/DavidAU/Qwen3.5-9B-The-Defiant-Fable-Uncensored-Heretic-NEO-IMATRIX-MAX-MTP-GGUF

---

## Closing

These three line up into a very clear ladder. GLM-5.3-Flash is the MIT-licensed, highest-download flagship, but with no official VRAM numbers and third parties citing 350GB-of-system-memory territory, it's a workstation project. Qwopus3.8-27B-Coder-Flash-GGUF is the mid-tier sweet spot for a local coding agent. DavidAU's 9B is the only one you can genuinely get hands-on with on a single gaming GPU. If you only want to try one tonight, the 9B is the fastest start, but remember two things: it's **fully uncensored**, so vet the output yourself, own the use case, and don't wire it straight into a public-facing service; and if you want vision, you must **separately download the mmproj file into the same folder**, otherwise images won't go through. One last note on timeliness: none of these three model cards list a release date, everything in this episode quotes the current contents of the HF pages and model cards, this show did not actually run any of these models, and pages can be updated at any time, so go by whatever you see on the page when you get there. See you next episode.

---

## About this article and its author

Originally published on [Mark Ku's Tech Notes](https://blog.markkulab.net/en/local-ai-lab/local-ai-lab-2026-11-05)

License: [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) — when reusing or quoting, credit the author and link back to the original

### About the author

**[Mark Ku](https://blog.markkulab.net/en/author/mark-ku)** — Software engineer

- 10+ years as a software engineer
- Built North-American e-commerce and AI SaaS subscription billing

### Free tools built by the author

All of these are free to use:

- [Free PDF Sign Tool](https://blog.markkulab.net/en/tools/pdf-sign): Online PDF sign tool — draw, type, or upload a signature, then drag, resize, and download. Everything runs in your browser; nothing is uploaded.
- [VS Code Refactory](https://blog.markkulab.net/en/tools/refactory): Refactory is a VS Code refactoring extension: 34 actions plus a 37-rule code-smell inspection layer with a Code Health dashboard, across 18 languages, backed by 534 tests. It learns your repo's conventions: where interfaces live, where DI is registered, whether 'use client' belongs. It ranks files by git churn × complexity so you know what to fix first, and hands any smell to the Claude Code already on your machine. Free to use, and your source never leaves your computer.
- [DB-Kit Database Manager](https://blog.markkulab.net/en/tools/db-kit): DB-Kit is a lightweight, cross-platform database manager built with Tauri + Rust + React. Manage MySQL, MariaDB, PostgreSQL, SQL Server, Oracle, SQLite, MongoDB, Redis, Kafka, Elasticsearch and RabbitMQ from one consistent interface: passwords encrypted in the OS keychain, SSH tunnels, full CRUD, a visual query builder, stacked multi-statement result sets, cross-connection data transfer, schema & data compare with sync SQL and schema snapshots, Excel / CSV import & export, visualized execution plans, ER diagrams, scheduled backups, SQL stress testing with p50–p99 latency percentiles, a 15-rule SQL review engine, Review & Run (AI review plus per-statement backups and an auto-generated rollback script), Kafka message browsing with monitoring & alerts, an SSH terminal with SFTP / FTP, Docker / Kubernetes, remote desktop (RDP / VNC / RustDesk), file / folder compare, stored procedure integration tests, six UI languages, a built-in AI assistant (a local CLI or any Anthropic / OpenAI-compatible API; natural-language SQL, AI review and tuning advice) and the dbk CLI. Free and open source (MIT), with installers for Windows, macOS and Linux.
- [VS Code Super Mermaid](https://blog.markkulab.net/en/tools/super-mermaid): Super Mermaid is a VS Code extension for beautiful Mermaid diagrams out of the box: auto-colored live preview, mouse pan & zoom, high-res PNG / SVG export, 21 templates and multiple themes. Free and open source (MIT).
- [React Super Mermaid](https://blog.markkulab.net/en/tools/react-super-mermaid): react-super-mermaid is an open-source React component library: render beautiful Mermaid diagrams with a single <MermaidViewer>, with built-in colorful / sketch themes, pan & zoom, in-diagram search, and high-res SVG / PNG export. Lightweight, SSR-safe, fully typed. Free and open source (MIT).
- [Jira / Confluence Super Mermaid](https://blog.markkulab.net/en/tools/jira-super-mermaid): An Atlassian Forge app: write Mermaid syntax directly inside a Jira issue or a Confluence page and get flowcharts, sequence diagrams, state machines and Gantt charts. 11 diagram types, SVG / PNG export, light and dark themes, full CJK support. Runs on Atlassian: your diagrams live in your own site and the app calls no third-party service. Free, coming soon to the Atlassian Marketplace.
- [Mermaid Live Preview](https://blog.markkulab.net/en/tools/mermaid-preview): Write Mermaid in your browser, see it render instantly, and share the whole diagram as a single link. No sign-up, nothing uploaded to a server, and mermaid.live share links work as-is.
- [React Intl Phone Number](https://blog.markkulab.net/en/tools/react-intl-phone-number): react-intl-phone-number is an open-source React component: framework-agnostic and antd-free, with E.164 in/out, a searchable flag / country-code dropdown, configurable validation levels (strict / mobile-strict / loose), themeable CSS, and i18n — phone logic powered by google-libphonenumber. Lightweight and fully typed. Free and open source (MIT).
- [Uptime Kuma Cluster](https://blog.markkulab.net/en/tools/uptime-kuma-cluster): Turn single-node Uptime Kuma into a highly available cluster: OpenResty + Lua smart load balancing, shared MariaDB state, health checks and automatic failover, plus cluster-management REST APIs. One Docker Compose command to start. Free and open source (MIT).
- [AI Podcast Cut](https://blog.markkulab.net/en/tools/ai-podcast-cut): Drop in a recording and it removes fillers and stutters, levels loudness segment by segment, and sends a second agent to review every cut. Cut points snap to word boundaries and zero crossings, every splice gets a fade, and sentence-end breaths are preserved. Desktop app for Windows, macOS and Linux. MIT licensed; the Windows installer bundles ffmpeg.
- [AI Video Cut](https://blog.markkulab.net/en/tools/ai-video-cut): Pick something in your video: type it (face, license plate, phone screen, logo), click points or drag a box, or let Claude Code / Codex pick it. AI Video Cut tracks it frame by frame, then mosaics or blurs it, recolors it, pins stickers and text to it, swaps a screen or poster for your own image or video, or removes it using background that other frames actually captured. Plus sequence editing, local captions and an AI assistant. Free, open-source (MIT) Windows desktop app (macOS / Linux experimental) that runs locally, nothing uploaded.
- [open-pos restaurant POS](https://blog.markkulab.net/en/tools/open-pos): One computer and one receipt printer is enough to open the shop. Your data lives on your own disk, no subscription, no lock-in, MIT licensed. Money is integer New Taiwan dollars with tax split by the statutory formula, so sales + tax always equals the total. Printing goes straight over ESC/POS on TCP 9100, with no vendor driver. Tauri + Rust + SQLite desktop app, v1.0 in development.
- [.NET e-commerce API template](https://blog.markkulab.net/en/tools/dotnet-ecommerce-api): An e-commerce backend API template built on .NET 10: Controller → Service → Repository layering, Autofac convention-based registration, SqlSugar for MySQL / MariaDB and SQL Server, Redis cache and message queue, Hangfire jobs, JWT auth, and an extensible payment provider module (ECPay, Newebpay). Apache 2.0.
- [Open Token Monitor team AI usage](https://blog.markkulab.net/en/tools/open-token-monitor): Collect token usage and equivalent cost from every employee's AI coding tools into a hub you host yourself: PostgreSQL, org roster, dashboards compared by company and department, and a reports API. Electron and Rust/Tauri clients upload every 30 minutes. MIT licensed, one Docker command.
- [ai-security-monitor defensive security](https://blog.markkulab.net/en/tools/ai-security-monitor): Your website is live. Who checks its certificates, security settings and known vulnerabilities? A self-hosted Security Hub brings website scans and host monitoring together, deduplicates findings and compares scans after fixes.

### Daily podcasts

- [Mark's Tech Insights — Daily AI News](https://blog.markkulab.net/en/category/tech-news): Daily curated AI and tech trends. Catch the latest developments via audio summaries — covering AI applications, software architecture, DevOps, and engineering practice. — RSS: https://blog.markkulab.net/feed.xml
- [AI股市蝦聊](https://blog.markkulab.net/en/category/ai-stock-chat): Every trading day, an AI-analyzed take on the Taiwan stock market, delivered as a two-host conversation covering the session and the next-day outlook. — RSS: https://blog.markkulab.net/ai-stock-chat/feed.xml
- [開源好物日報](https://blog.markkulab.net/en/category/open-source-weekly): A daily two-host pick of free open-source tools surfaced from real Hacker News, GitHub, and Reddit buzz — what pain they solve and the fastest way to get started. — RSS: https://blog.markkulab.net/open-source-weekly/feed.xml
- [AI 運動日報](https://blog.markkulab.net/en/category/sports-weekly): Two hosts talk NBA, MLB and world sport every day — scores, records and the stories behind them, from a Taiwanese fan perspective. — RSS: https://blog.markkulab.net/sports-weekly/feed.xml
- [AI 國際新聞快報](https://blog.markkulab.net/en/category/world-news): A daily two-host briefing that makes sense of the past 24 hours in world news — geopolitics, the global economy, conflict and security, disasters and climate — from a Taiwanese perspective, neutral and fully sourced. — RSS: https://blog.markkulab.net/world-news/feed.xml
- [地端 AI 實驗室](https://blog.markkulab.net/en/category/local-ai-lab): Every day, a two-host look at open-weight models and LoRAs you can actually run on your own machine: what they are for, whether your GPU can handle them, and how they compare — sourced from official model cards. — RSS: https://blog.markkulab.net/local-ai-lab/feed.xml
- [AI 行銷日報](https://blog.markkulab.net/en/category/ai-marketing): Every day, two hosts break down the latest in AI for marketing, advertising, SEO/GEO and content marketing: practical, never sponsored, every story sourced. — RSS: https://blog.markkulab.net/ai-marketing/feed.xml
- [AI 資安日報](https://blog.markkulab.net/en/category/ai-security): Every day, two hosts cover the latest where AI meets cybersecurity: LLM and AI agent vulnerabilities, prompt injection, model supply-chain risks and AI-powered attacks, from a defense-first angle with every story sourced. — RSS: https://blog.markkulab.net/ai-security/feed.xml

### Newsletter

[Subscribe to the newsletter](https://blog.markkulab.net/en/subscribe) — Be the first to know about new posts. No spam, unsubscribe anytime.