On-Premises AI Lab

🧪 The 321B GLM-5.3-Flash won't fit on a single GPU, but this 9B model's popularity shot up to 113 | On-Premise AI Lab

Mark Ku

November 5, 2026•27 min read

Opening

GLM-5.3-Flash has 6.3M downloads on Hugging Face, 3.1k likes, MIT license, and the page lists 321B parameters. The official model card even dedicates a section to "Serve GLM-5.3-Flash Locally" to walk you through self-hosting, though a single consumer GPU almost certainly won't fit it. The other two in this episode are more down to earth: a 27B coding agent in GGUF, and a 9B writing model that the author clocked at 130 t/s and up on his own 5090 with LM Studio. As for why the "download champion" isn't the "HF trending champion," I'll get to that shortly.

This Episode's Picks

1. zai-org/GLM-5.3-Flash: MIT-licensed flagship natively multimodal open weights, where the bottleneck is RAM, not your GPU

What it is: Sourced from this show's "local model radar" segment, corresponding to the zai-org/GLM-5.3-Flash page on Hugging Face. The official model card states this is the first natively multimodal model in the GLM-5 series. The HF page lists 321B parameters with BF16 and F8_E4M3 tensor types, while the official Introduction describes 320B total parameters with 18B active. License is MIT, weights are public, no application required.
Where you'd use it:
- Backend for a "nothing leaves the building" multimodal assistant on your corporate intranet: the model card's image-text-to-text pipeline example literally passes in an image plus "describe this image." Picture a pile of internal system screenshots, report charts, and flow diagrams that you want to turn into a queryable knowledge portal, except compliance won't let you send anything to an external API. This model is positioned exactly for standing up your own endpoint.
- Evaluating whether to point your own agent at a self-hosted endpoint: the official Introduction claims it approaches Claude Opus 4.8 on coding and agentic benchmarks, and the footnotes spell out the eval setup in considerable detail, for example DeepSWE using the mini-swe-agent harness at 400K context, and Terminal-Bench 2.1 using Claude Code 2.1.207. If you're building a coding or agent toolchain, those settings are worth comparing against your own harness conditions rather than just reading the scores.
- Server-side developers who need to control token cost and latency: a Note in the model card explains you can use reasoning_effort to switch the thinking budget between low, high, and max, with max as the default, and recommends explicitly passing clear_thinking=true for production chat scenarios. In other words, the team left you a tunable "save tokens" switch, which anyone running a live service will appreciate.
Can you actually run it: to be clear up front, the official model card lists no VRAM or RAM numbers at all. It only enumerates six deployment paths under "Serve GLM-5.3-Flash Locally": SGLang, vLLM, TokenSpeed, Transformers, KTransformers, and Unsloth, with KTransformers and Unsloth each having their own tutorial links. As for third-party numbers, DataCamp's KTransformers tutorial says the official KTransformers guide recommends reserving at least 350GB of system memory, with FP8 weights around 306 GiB. Unsloth, meanwhile, claims on an HF discussion that its dynamic quantization can run 3-bit in 128GB of memory and 1-bit in 100GB. Both of those are third-party claims, not figures stated on zai-org's official model card. What follows is this show's judgment, not a conclusion from the sources and not a measured result: since no official numbers exist, our rough tier estimate based on 321B parameters plus FP8/dynamic quantization is that this is a self-hosting project only viable on a large-memory workstation or Mac Studio class machine. If you have a single consumer GPU, skip it.
Buzz this episode: HF trending score 62, 6.3M downloads, 3.1k likes. One clarification so this doesn't sound contradictory in a minute: the highest download count this episode belongs to GLM-5.3-Flash, but the highest HF trending score goes to the third pick, the 9B (score 113). Downloads and trending score are two different metrics, one measures cumulative volume and the other is the platform's composite heat score, so don't compare them directly.
How it compares: against deepseek-ai/DeepSeek-V4.1-Flash-1017, which has comparable buzz this period, DeepSeek's official model card comparison table lists GLM-5.3 as a competitor and shows DeepSeek-V4.1-Flash-1017 scoring higher on Terminal-Bench 4.0, DeepSWE v1.1, and other columns. That's DeepSeek's framing. That said, the GLM-5.3-Flash page lists 321B parameters with 18B active, noticeably smaller than DeepSeek's model listed at 763B, which puts the bar meaningfully lower for anyone wanting to self-host.
How to get it: the model card's vLLM section says to pip install vllm then run vllm serve "zai-org/GLM-5.3-Flash"; for SGLang it's python3 -m sglang.launch_server --model-path "zai-org/GLM-5.3-Flash". Both bring up an OpenAI-compatible API. The page also has Docker and community quantization (Browse Quantizations) entry points.
Link: https://huggingface.co/zai-org/GLM-5.3-Flash

2. Jackrong/Qwopus3.8-27B-Coder-Flash-GGUF: the 27B sweet spot for a local coding agent, and the author is refreshingly conservative about it

What it is: also from this episode's "local model radar." The model card states this is a coding-oriented fine-tune of unsloth/Qwen3.8-27B (derived from Qwen/Qwen3.8-27B). The HF page lists 27B parameters, GGUF format, Apache-2.0 license, quantized files provided directly, no application required.
Where you'd use it:
- Running a local coding agent on your own machine: the README spells out the target workflow as Read → Reason → Call Tools → Edit → Test → Recover → Deliver, with training data including roughly 20K agent interaction and tool-call samples, mostly distilled from Qwen3.8-Flash-Max. In other words, it's aimed not at "finish this line for me" but at the full loop of picking a tool, reading the output, revising the plan, and shipping the result.
- Teams whose code can't leave the intranet: the llama.cpp and Ollama commands on the HF page point straight at the Q4_K_M quant, so you can stand up an OpenAI-compatible endpoint internally and then wire in local terminal coding tools using the Pi, OpenClaw, and Hermes Agent config examples shown on the page. For projects where the client contract explicitly forbids putting code in the cloud, this route is very practical.
- People grinding algorithm problems and doing refactors: the model card says that within roughly 200K code and complex-reasoning samples, the algorithm and code-generation portion (teacher models listed as Qwen3.8-Flash-Max, Qwen3.8-Max, and DeepSeek-V4-Flash) focused training on algorithm implementation, edge cases, refactoring, and debugging.
Can you actually run it: no official VRAM or memory requirements are listed. The model card only has a "Checkpoint and runtime" section plus Transformers and vLLM inference examples, and the llama.cpp, Ollama, Docker Model Runner, and Lemonade commands on the page all default to pulling the Q4_K_M tag. What follows is this show's rough judgment, not a claim from the source and not a measured result: with no official numbers, our estimate based on 27B parameters plus Q4_K_M is that this is comfortable on "a GPU with a lot of VRAM, or a machine with generous unified memory." Before committing, actually load Q4_K_M once on your own hardware to confirm, and please don't buy a GPU based on this tiering alone.
Buzz this episode: HF trending score 61, 23.8k downloads, 62 likes. Far smaller volume than the GLM model, but it's one of the few community fine-tunes this episode that actually has something to say for itself.
How it compares: the author states in the model card that it's a coding fine-tune of unsloth/Qwen3.8-27B, differing from stock Qwen3.8-27B by the addition of distilled agent interaction and tool-call data. Credit where it's due: the card explicitly says this round of evaluation did not measure any "reduction in reasoning tokens," so the Flash in the name should not be read as a verified speedup. That caveat comes from the author himself.
How to get it: the README and HF page give llama serve -hf Jackrong/Qwopus3.8-27B-Coder-Flash-GGUF:Q4_K_M to start a local OpenAI-compatible server, or ollama run hf.co/Jackrong/Qwopus3.8-27B-Coder-Flash-GGUF:Q4_K_M. Apache-2.0, no application required.
Link: https://huggingface.co/Jackrong/Qwopus3.8-27B-Coder-Flash-GGUF

3. DavidAU/Qwen3.5-9B-The-Defiant-Fable-Uncensored-Heretic-NEO-IMATRIX-MAX-MTP-GGUF: the only one this episode you can genuinely run on a single gaming GPU, but it's fully uncensored

What it is: also from this episode's "local model radar." The author describes it as a 9B model produced through multi-stage fine-tuning and merging, GGUF format, Apache-2.0, with the card listing 256k context and the README noting vision is enabled. Let's get this out of the way first: this is a fully uncensored model with no built-in content refusal boundaries. Output content and usage responsibility are entirely on you, so think hard before putting it anywhere near a corporate workflow.
Where you'd use it:
- A fully offline writing and editing assistant when all you have is one gaming GPU: 9B plus GGUF, and the author says it runs in all the standard AI apps. Combined with the card's 256k context, it suits the "I have a tens-of-thousands-of-words draft and want to hand the whole thing over for a pass" use case, all without leaving your machine.
- Local image Q&A: the README says vision is enabled, but there's an easy trap here, you must separately download an mmproj file and put it in the same folder as the GGUF before the model can accept images. Grab only the main file and you'll think vision is broken.
- Local LLM enthusiasts who care about output speed and are willing to tune: the author provides both regular and MTP (multi-token prediction) NEO IMATRIX quants, and recommends keeping temp below 1 and rep pen at 1 for the MTP version, falling back to the regular quant if token acceptance drops below 50%.
- Engineers who want a drop-in for an existing workflow: the card lists recommended sampling settings, thinking mode at temp 1.0 / top_p 0.95 / top_k 20, temp 0.6 for precise coding tasks, instruct mode at temp 0.7 / top_p 0.80, and notes that reasoning defaults to xhigh and automatically drops to medium when you switch to instruct.
Can you actually run it: no official minimum VRAM is listed. The hardware hints the author mentions are: all quants are NEO IMATRIX (he claims 2-4% better accuracy than ordinary GGUF), output tensors kept at 16-bit, MTP tensors set to Q8_0, and a recommendation to set context to at least 8k to 16k. The speed figures are the author's own measurements on his own setup: he writes that on a 5090 with Windows 11 and LM Studio, the regular Q4_K_S GGUF runs around 130 t/s, and the MTP version (2 tokens, 60% acceptance rate) can exceed 185 t/s, and he himself notes speeds vary by GPU, system, and app. Treat these as reference points for his environment, not as what your machine will do.
Buzz this episode: HF trending score 113, 2.0M downloads, 1.4k likes. As mentioned earlier, this is the highest HF trending score of the three this episode, while the highest download count is still GLM-5.3-Flash's 6.3M. Two different metrics.
How it compares: the author posts a self-run benchmark table on the card (arc/c, arc/e, boolq, hellaswag, openbookqa, piqa, winogrande), claiming this 9B beats base models like Qwen3.5-9B-Instruct and Qwen3.5-27B-Instruct on 7 of them, and in some cases approaches Qwen3.6-27B. These are entirely the author's own tests and claims, and he himself adds a caveat: beating a 27B on benchmarks doesn't mean it will beat a 27B on every task. That caveat is honestly more worth remembering than the scores.
How to get it: the local commands on the HF page are llama serve -hf DavidAU/Qwen3.5-9B-The-Defiant-Fable-Uncensored-Heretic-NEO-IMATRIX-MAX-MTP-GGUF:Q4_K_M, or ollama run hf.co/DavidAU/...:Q4_K_M, and you can also just open it in LM Studio or Jan. For image input, remember to separately fetch the mmproj file and place it in the same folder, per the README.
Link: https://huggingface.co/DavidAU/Qwen3.5-9B-The-Defiant-Fable-Uncensored-Heretic-NEO-IMATRIX-MAX-MTP-GGUF

Closing

These three line up into a very clear ladder. GLM-5.3-Flash is the MIT-licensed, highest-download flagship, but with no official VRAM numbers and third parties citing 350GB-of-system-memory territory, it's a workstation project. Qwopus3.8-27B-Coder-Flash-GGUF is the mid-tier sweet spot for a local coding agent. DavidAU's 9B is the only one you can genuinely get hands-on with on a single gaming GPU. If you only want to try one tonight, the 9B is the fastest start, but remember two things: it's fully uncensored, so vet the output yourself, own the use case, and don't wire it straight into a public-facing service; and if you want vision, you must separately download the mmproj file into the same folder, otherwise images won't go through. One last note on timeliness: none of these three model cards list a release date, everything in this episode quotes the current contents of the HF pages and model cards, this show did not actually run any of these models, and pages can be updated at any time, so go by whatever you see on the page when you get there. See you next episode.