---
title: "🛠️ Run a 125B Model on a 12GB Gaming Card? Strata's Buzz Surges to 5.9k Stars｜Open Source Weekly Picks"
description: "The craziest number this week: Strata runs a 125B-parameter model on a single 12GB gaming GPU, MIT-licensed and already past 5.9k stars in under a month. Two other local-first approaches surfaced the same week: Ledge.sh turns Markdown notes into a persistent terminal that can run commands, while Magnitude compiles kernels on the fly right on your machine, and I'll tell you the speedup it gives llama.cpp in a moment."
canonical_url: "https://blog.markkulab.net/en/open-source-weekly/open-source-weekly-2026-10-03"
author: "Mark Ku"
author_url: "https://blog.markkulab.net/en/author/mark-ku"
site: "Mark Ku's Tech Notes"
date_published: "2026-10-03T02:00:00.000Z"
category: "Open Source Weekly"
tags: ["open-source", "podcast", "開源工具", "開源", "Ledgesh", "Strata", "Magnitude", "本地大型語言模型", "Runbook", "CodingAgent"]
language: "en"
license: "CC BY 4.0"
license_url: "https://creativecommons.org/licenses/by/4.0/"
attribution: "when reusing or quoting, credit the author and link back to the original"
---

# 🛠️ Run a 125B Model on a 12GB Gaming Card? Strata's Buzz Surges to 5.9k Stars｜Open Source Weekly Picks

> **TL;DR** — Strata runs a 125B-parameter model on 12GB VRAM using dynamic weight distribution across VRAM, system RAM, and SSD, achieving 94 tokens/s output with Q2_0 quantization and 5.

## Opening

The most eye-popping number this week is Strata running a 125B-parameter model on a single 12GB gaming GPU, hitting 5.9k stars under an MIT license in under a month. Two other local-first approaches also surfaced the same week: Ledge.sh turns Markdown notes into a persistent terminal that can actually run commands, while Magnitude compiles kernels on the fly, right on your own machine. I'll tell you exactly how much of a speedup that gives over llama.cpp in a bit.

## This Week's Open Source Picks

### 1. Ledge.sh: Turning Markdown Notes into a Persistent, Runnable Terminal
- **What it is**: Ledge.sh lets you take the Markdown notes you already write and turn code blocks directly into one-click executable commands. The whole note shares a single persistent shell, so variables, `cd`, and virtual environments all persist across multiple runs, unlike Jupyter where each cell runs in isolation. It's a great fit for turning the "copy-paste into terminal" playbooks you already have into one-click, rerunnable runbooks.
- **Buzz this week**: It landed 202 points and 88 comments on Hacker News, with discussion centered on "this is exactly the runbook tool I wanted." That said, the GitHub repo currently sits at just 278 stars, so the buzz is mostly driven by this HN thread.
- **Standout features**:
  - Built-in support for running Shell, Python, Node, Ruby, PHP, and TypeScript directly; SQL and Redis blocks can connect to your existing databases, and adding other interpreters is just one line in the settings
  - Adding one line, `host:`, to the frontmatter routes the entire note's execution over SSH to that machine. Once connected to a Ledge Server, closing your laptop won't interrupt a running block, and the mobile app lets you check back on progress
  - Comes with an MCP server; running `claude mcp add ledge -- ledge mcp` lets Claude Code read, write, and search your notes, and even the prompt blocks themselves are executable
- **Quick start**: On macOS, download `.dmg`; on Windows, run `Ledge-Setup.exe` (requires WSL); on Linux, extract and run `./installer`. To keep it running persistently on a remote machine, run `curl -fsSL https://ledge.sh/server.sh | sh` on that machine. Once installed, open any note and hit execute on a code block, if the next block can still read a variable set by the previous one, that confirms the persistent shell is actually working. If you've connected a Ledge Server, the mobile app should show the block currently executing, confirming the server side is wired up too.
- **How it compares**: The biggest difference from Jupyter is that it's not a cell-based kernel but one persistent shell per note, and the file itself is just plain Markdown, no `.ipynb`-style diff hell. Compared to Runme (a similar tool that turns Markdown code blocks into executable runbooks, conceptually similar but without persistent SSH execution, a mobile client, or a built-in MCP server), Ledge adds remote persistence, mobile check-ins, and AI agent integration.
- **Who it's for**: Engineers sitting on a pile of "copy-paste into terminal" operational notes who want to turn their SOPs into one-click, rerunnable runbooks.
- **Link**: [ledge.sh](https://ledge.sh)

### 2. Strata: Running a 125B Model on a 12GB Gaming GPU
- **What it is**: Strata tackles the age-old problem of "I want to run a big model but my GPU isn't big enough." It dynamically distributes model weights across VRAM, system RAM, and SSD (dynamic expert routing), which is different from traditional layer-by-layer offloading, so even a 12GB VRAM card can fit a 125B-parameter model. It also exposes both OpenAI- and Anthropic-compatible endpoints, so your existing tooling can plug into a local model with barely any changes.
- **Buzz this week**: It shot up to 5.9k GitHub stars (5,931 exactly) within 30 days, under an MIT license, with C++ as the primary language.
- **Standout features**:
  - 94 tokens/s output, 2,650 tokens/s input, and a 70GB download: these three numbers all describe the same measured result, running the 125B-parameter Qwen3.8-Flash-Next on just 12GB VRAM with Q2_0 quantization. The 70GB figure is the size of the weights themselves, not something that needs to fully fit in the GPU; that's made possible by the VRAM/RAM/SSD distribution mentioned above. Switching to IQ3_S quantization drops throughput to 53 tokens/s, higher quantization levels consume more resources and cost you speed
  - Speculative decoding, "guess then verify," delivers a 1.6x to 1.8x speedup
  - The install script auto-detects your hardware, picks an appropriately sized model, downloads the weights, and spins up the service directly, complete with a browser UI for chatting, monitoring dashboards, and tuning settings. It supports NVIDIA RTX 20/30/40/50 series and AMD RX 7900 XT/XTX, 7800 XT, and 9070+ (12GB VRAM or more), with optional image input support
- **Quick start**: On Windows, just double-click `START-HERE.bat`; on Linux, run `./setup.sh`, and the installer handles the rest automatically. Make sure you have 32GB of RAM ready (64GB recommended) and roughly 80GB of free SSD space. Once installed, open `http://127.0.0.1:8080` in your browser, or just hit the OpenAI-compatible endpoint directly with `curl http://127.0.0.1:8080/v1/models`; seeing the model list respond confirms the service is actually up.
- **How it compares**: Ollama and LM Studio work by picking a model that fits within your VRAM; Strata flips that around, betting on "spread the whole machine's resources thin to force a big model to run," at the cost of a 70GB download and Q2-level quantization. The README is upfront that it's built on top of llama.cpp/ggml, and that AMD throughput runs roughly 35% to 65% lower than an equivalent NVIDIA setup.
- **Who it's for**: Anyone with just a single 12GB consumer GPU who wants to get hands-on with a 125B-class model locally, and is willing to trade some quality from Q2 quantization for the simple fact that it actually runs.
- **Link**: [niko1221/Strata](https://github.com/Niko1221/Strata)

### 3. Magnitude: Compiling Kernels On the Fly, One Click to Wire Up Your Coding Agent Locally
- **What it is**: Magnitude takes a different approach to local inference. Instead of shipping a pre-built kernel tuned for a broad category of hardware like llama.cpp does, it compiles and tunes on the spot, right on your own device, then lets you point coding agents like Claude Code, Codex, and Cline at your local model with a single click.
- **Buzz this week**: This was a Launch HN post, pulling in 194 points and 97 comments, with 6,263 GitHub stars accumulated under an Apache-2.0 license, primarily written in Rust.
- **Standout features**:
  - Official benchmarks against llama.cpp: 92% faster decoding on Metal, 19% faster on CUDA; prefill is 9% faster on Metal and 23% faster on CUDA. That said, the official numbers don't disclose the test hardware or the llama.cpp version used, so take them as a rough guide and benchmark on your own machine for the real answer
  - Memory management tuned specifically for agent workloads: each agent uses 27% less memory, memory is released once the agent stops, and a shared prefix cache across sessions prevents them from slowing each other down
  - The Connections page lets you one-click connect Pi, OpenCode, Hermes, Codex, Claude Code, and Cline, across Apple Silicon, NVIDIA, AMD, and CPU-only setups. Once the model is downloaded, the entire workflow requires no network connection
- **Quick start**: Install the desktop app (CLI included) from magnitude.dev/download, pick a model to download on the Discover page, then go to the Connections page and one-click wire up your coding agent. Once connected, go back to Claude Code or Cline and ask it something, if it responds normally even with your network unplugged, that confirms it's actually running on the local model instead of the cloud.
- **How it compares**: llama.cpp, Ollama, and LM Studio all ship pre-compiled kernels tuned for a wide range of hardware out of the box. Magnitude instead tunes per device on the spot, adding per-agent memory reclamation and a shared prefix cache across sessions. Compared to Strata from the same week: Strata is a Windows one-click installer that forces a single large model to run on limited hardware, while Magnitude leans toward Mac and multi-platform support, with a focus on directly plugging into coding agents. The README is also upfront that it's "hand-optimized for popular open-weight model families," not a universal solution for every model.
- **Who it's for**: Engineers already using coding agents like Claude Code and Cline who want to connect them to a local model, and who care about the performance of on-device tuning itself.
- **Link**: [magnitudedev/magnitude](https://github.com/magnitudedev/magnitude)

## Closing

These three tools are really three different answers to the same question: can local setups actually run something "genuinely big"? Ledge.sh tackles the daily grind of repetitive operations, while Strata and Magnitude represent two distinct approaches to local inference, one betting on distributing hardware resources, the other on on-the-fly compilation. If you can only try one first, I'd start with Strata's 12GB threshold, since that's a GPU most people already have sitting in their machine. I'll dig up more great finds next week, see you then.

---

## About this article and its author

Originally published on [Mark Ku's Tech Notes](https://blog.markkulab.net/en/open-source-weekly/open-source-weekly-2026-10-03)

License: [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) — when reusing or quoting, credit the author and link back to the original

### About the author

**[Mark Ku](https://blog.markkulab.net/en/author/mark-ku)** — Software engineer

- 10+ years as a software engineer
- Built North-American e-commerce and AI SaaS subscription billing

### Free tools built by the author

All of these are free to use:

- [Free PDF Sign Tool](https://blog.markkulab.net/en/tools/pdf-sign): Online PDF sign tool — draw, type, or upload a signature, then drag, resize, and download. Everything runs in your browser; nothing is uploaded.
- [VS Code Refactory](https://blog.markkulab.net/en/tools/refactory): Refactory is a VS Code refactoring extension: 34 actions plus a 37-rule code-smell inspection layer with a Code Health dashboard, across 18 languages, backed by 534 tests. It learns your repo's conventions: where interfaces live, where DI is registered, whether 'use client' belongs. It ranks files by git churn × complexity so you know what to fix first, and hands any smell to the Claude Code already on your machine. Free to use, and your source never leaves your computer.
- [DB-Kit Database Manager](https://blog.markkulab.net/en/tools/db-kit): DB-Kit is a lightweight, cross-platform database manager built with Tauri + Rust + React. Manage MySQL, MariaDB, PostgreSQL, SQL Server, Oracle, SQLite, MongoDB, Redis, Kafka, Elasticsearch and RabbitMQ from one consistent interface: passwords encrypted in the OS keychain, SSH tunnels, full CRUD, a visual query builder, stacked multi-statement result sets, cross-connection data transfer, schema & data compare with sync SQL and schema snapshots, Excel / CSV import & export, visualized execution plans, ER diagrams, scheduled backups, SQL stress testing with p50–p99 latency percentiles, a 15-rule SQL review engine, Review & Run (AI review plus per-statement backups and an auto-generated rollback script), Kafka message browsing with monitoring & alerts, an SSH terminal with SFTP / FTP, Docker / Kubernetes, remote desktop (RDP / VNC / RustDesk), file / folder compare, stored procedure integration tests, six UI languages, a built-in AI assistant (a local CLI or any Anthropic / OpenAI-compatible API; natural-language SQL, AI review and tuning advice) and the dbk CLI. Free and open source (MIT), with installers for Windows, macOS and Linux.
- [VS Code Super Mermaid](https://blog.markkulab.net/en/tools/super-mermaid): Super Mermaid is a VS Code extension for beautiful Mermaid diagrams out of the box: auto-colored live preview, mouse pan & zoom, high-res PNG / SVG export, 21 templates and multiple themes. Free and open source (MIT).
- [React Super Mermaid](https://blog.markkulab.net/en/tools/react-super-mermaid): react-super-mermaid is an open-source React component library: render beautiful Mermaid diagrams with a single <MermaidViewer>, with built-in colorful / sketch themes, pan & zoom, in-diagram search, and high-res SVG / PNG export. Lightweight, SSR-safe, fully typed. Free and open source (MIT).
- [Jira / Confluence Super Mermaid](https://blog.markkulab.net/en/tools/jira-super-mermaid): An Atlassian Forge app: write Mermaid syntax directly inside a Jira issue or a Confluence page and get flowcharts, sequence diagrams, state machines and Gantt charts. 11 diagram types, SVG / PNG export, light and dark themes, full CJK support. Runs on Atlassian: your diagrams live in your own site and the app calls no third-party service. Free, coming soon to the Atlassian Marketplace.
- [Mermaid Live Preview](https://blog.markkulab.net/en/tools/mermaid-preview): Write Mermaid in your browser, see it render instantly, and share the whole diagram as a single link. No sign-up, nothing uploaded to a server, and mermaid.live share links work as-is.
- [React Intl Phone Number](https://blog.markkulab.net/en/tools/react-intl-phone-number): react-intl-phone-number is an open-source React component: framework-agnostic and antd-free, with E.164 in/out, a searchable flag / country-code dropdown, configurable validation levels (strict / mobile-strict / loose), themeable CSS, and i18n — phone logic powered by google-libphonenumber. Lightweight and fully typed. Free and open source (MIT).
- [Uptime Kuma Cluster](https://blog.markkulab.net/en/tools/uptime-kuma-cluster): Turn single-node Uptime Kuma into a highly available cluster: OpenResty + Lua smart load balancing, shared MariaDB state, health checks and automatic failover, plus cluster-management REST APIs. One Docker Compose command to start. Free and open source (MIT).
- [AI Podcast Cut](https://blog.markkulab.net/en/tools/ai-podcast-cut): Drop in a recording and it removes fillers and stutters, levels loudness segment by segment, and sends a second agent to review every cut. Cut points snap to word boundaries and zero crossings, every splice gets a fade, and sentence-end breaths are preserved. Desktop app for Windows, macOS and Linux. MIT licensed; the Windows installer bundles ffmpeg.
- [AI Video Cut](https://blog.markkulab.net/en/tools/ai-video-cut): Pick something in your video: type it (face, license plate, phone screen, logo), click points or drag a box, or let Claude Code / Codex pick it. AI Video Cut tracks it frame by frame, then mosaics or blurs it, recolors it, pins stickers and text to it, swaps a screen or poster for your own image or video, or removes it using background that other frames actually captured. Plus sequence editing, local captions and an AI assistant. Free, open-source (MIT) Windows desktop app (macOS / Linux experimental) that runs locally, nothing uploaded.
- [open-pos restaurant POS](https://blog.markkulab.net/en/tools/open-pos): One computer and one receipt printer is enough to open the shop. Your data lives on your own disk, no subscription, no lock-in, MIT licensed. Money is integer New Taiwan dollars with tax split by the statutory formula, so sales + tax always equals the total. Printing goes straight over ESC/POS on TCP 9100, with no vendor driver. Tauri + Rust + SQLite desktop app, v1.0 in development.
- [.NET e-commerce API template](https://blog.markkulab.net/en/tools/dotnet-ecommerce-api): An e-commerce backend API template built on .NET 10: Controller → Service → Repository layering, Autofac convention-based registration, SqlSugar for MySQL / MariaDB and SQL Server, Redis cache and message queue, Hangfire jobs, JWT auth, and an extensible payment provider module (ECPay, Newebpay). Apache 2.0.
- [Open Token Monitor team AI usage](https://blog.markkulab.net/en/tools/open-token-monitor): Collect token usage and equivalent cost from every employee's AI coding tools into a hub you host yourself: PostgreSQL, org roster, dashboards compared by company and department, and a reports API. Electron and Rust/Tauri clients upload every 30 minutes. MIT licensed, one Docker command.
- [ai-security-monitor defensive security](https://blog.markkulab.net/en/tools/ai-security-monitor): ai-security-monitor is a single Rust binary for your computers and servers. It finds unusual connections, rates them safe / unknown / suspicious / dangerous against 26 public threat-intel sources, and checks firewalls, open ports, vulnerabilities and 60 hardening rules. Security Hub shows the whole fleet, and Claude Code / Codex explain it in plain words. MIT, free, self-hosted.

### Daily podcasts

- [Mark's Tech Insights — Daily AI News](https://blog.markkulab.net/en/category/tech-news): Daily curated AI and tech trends. Catch the latest developments via audio summaries — covering AI applications, software architecture, DevOps, and engineering practice. — RSS: https://blog.markkulab.net/feed.xml
- [AI股市蝦聊](https://blog.markkulab.net/en/category/ai-stock-chat): Every trading day, an AI-analyzed take on the Taiwan stock market, delivered as a two-host conversation covering the session and the next-day outlook. — RSS: https://blog.markkulab.net/ai-stock-chat/feed.xml
- [開源好物週報](https://blog.markkulab.net/en/category/open-source-weekly): A weekly two-host pick of free open-source tools surfaced from real Hacker News, GitHub, and Reddit buzz — what pain they solve and the fastest way to get started. — RSS: https://blog.markkulab.net/open-source-weekly/feed.xml
- [AI 運動週報](https://blog.markkulab.net/en/category/sports-weekly): Two hosts talk NBA, MLB and world sport three times a week — scores, records and the stories behind them, from a Taiwanese fan perspective. — RSS: https://blog.markkulab.net/sports-weekly/feed.xml
- [AI 國際新聞快報](https://blog.markkulab.net/en/category/world-news): A daily two-host briefing that makes sense of the past 24 hours in world news — geopolitics, the global economy, conflict and security, disasters and climate — from a Taiwanese perspective, neutral and fully sourced. — RSS: https://blog.markkulab.net/world-news/feed.xml
- [地端 AI 實驗室](https://blog.markkulab.net/en/category/local-ai-lab): Twice a week, a two-host look at open-weight models and LoRAs you can actually run on your own machine: what they are for, whether your GPU can handle them, and how they compare — sourced from official model cards. — RSS: https://blog.markkulab.net/local-ai-lab/feed.xml
- [AI 行銷週報](https://blog.markkulab.net/en/category/ai-marketing): Every Tuesday, two hosts break down the week in AI for marketing, advertising, SEO/GEO and content marketing: practical, never sponsored, every story sourced. — RSS: https://blog.markkulab.net/ai-marketing/feed.xml
- [AI 資安週報](https://blog.markkulab.net/en/category/ai-security): Every Thursday, two hosts cover the week where AI meets cybersecurity: LLM and AI agent vulnerabilities, prompt injection, model supply-chain risks and AI-powered attacks, from a defense-first angle with every story sourced. — RSS: https://blog.markkulab.net/ai-security/feed.xml

### Newsletter

[Subscribe to the newsletter](https://blog.markkulab.net/en/subscribe) — Be the first to know about new posts. No spam, unsubscribe anytime.