---
title: "Haru-4B 4B-parameter direct speech-to-speech runs offline translation on 16GB VRAM, Edge AI Lab"
description: "On November 17, 2026, this issue's radar spotted a 4B‑parameter audio‑to‑audio model, Haru‑4B, with 4.4k downloads and an Apache‑2.0 license; the official model card states it runs BF16 on 16GB VRAM and 4‑bit GGUF on 8GB VRAM, claiming it can directly convert English meeting recordings into Chinese speech without any intermediate text."
canonical_url: "https://blog.markkulab.net/en/local-ai-lab/local-ai-lab-2026-11-17"
author: "Mark Ku"
author_url: "https://blog.markkulab.net/en/author/mark-ku"
site: "Mark Ku's Tech Notes"
date_published: "2026-11-17T02:00:00.000Z"
category: "On-premise AI Lab"
tags: ["local-ai", "podcast", "地端模型", "lora", "開源模型", "Haru-4B", "Inkling-2-Small", "fluid-speech-pro"]
language: "en"
license: "CC BY 4.0"
license_url: "https://creativecommons.org/licenses/by/4.0/"
attribution: "when reusing or quoting, credit the author and link back to the original"
---

# Haru-4B 4B-parameter direct speech-to-speech runs offline translation on 16GB VRAM, Edge AI Lab

## Opening
On November 17, 2026, our radar picked up Haru-4B, a 4B-parameter audio-to-audio model with 4.4k downloads under Apache-2.0. The official model card claims 16GB VRAM for BF16, 8GB for 4-bit GGUF, and the ability to turn English meeting recordings directly into Chinese speech without text as an intermediate step. Also coming up: Inkling-2-Small, a compact vision-language model with 18.4k downloads that reportedly fits in 3.2GB at 4-bit GGUF, worth a look for engineers running invoice parsing on consumer GPUs; and fluid-speech-pro, a 0.5B ASR model that the authors clocked at 0.25 real-time factor on an Intel i7-12700 CPU, even running offline transcription on a Raspberry Pi 5, under CC-BY-4.0 for unrestricted commercial use.

## This Week's Picks

### 1. higurashi-ai/Haru-4B: 4B End-to-End Speech-to-Speech, Commercial-Friendly, Fine-Tunable
- **What it is**  
  A 4B-parameter audio-to-audio model for Speech-to-Speech tasks, licensed Apache-2.0, commercial use and fine-tuning both allowed. The model card includes a fine-tune script and data format docs, so you can fork and experiment with LoRA for specific speaker styles (e.g., customer-service tone) right away.
- **Where it fits**  
  - Indie developers wanting an offline "speech-to-speech" demo on a laptop (16–24GB VRAM), e.g., turning English meeting recordings directly into Chinese speech output, no text intermediary.  
  - Voice assistant product teams deploying low-latency multilingual conversational entry points on edge devices (Jetson Orin, Mac Studio); the model card notes streaming inference support.  
  - Grad students fine-tuning specific speaker styles with LoRA; the model card provides the fine-tune script and data format, ready to fork.
- **Can it run?**  
  Official model card states: "BF16 on 16GB VRAM, 4-bit GGUF on 8GB VRAM"; also provides GGUF quantizations (Q4_K_M, Q8_0) for llama.cpp. If you only have 8GB VRAM, the official 4-bit GGUF path is likely the easiest entry point.
- **Buzz this week**  
  HF trending score 374, 4.4k downloads, 476 likes.
- **vs. peers**  
  The author compares against SeamlessM4T-v2-large (1.2B) and Whisper + TTS pipelines in the model card, claiming Haru-4B wins on "end-to-end latency" and "tone preservation"; this is self-reported, no independent third-party reproduction yet.
- **How to get it**  
  README gives two paths: ① `pip install -U transformers accelerate` then `AutoModelForSpeechSeq2Seq.from_pretrained("higurashi-ai/Haru-4B")`; ② download GGUF with `llama.cpp --model haru-4b-q4_k_m.gguf --audio-in mic`.
- **Link**  
  [https://huggingface.co/higurashi-ai/Haru-4B](https://huggingface.co/higurashi-ai/Haru-4B)

### 2. thinkingmachines/Inkling-2-Small: Compact VLM with 18.4k Downloads, 3.2GB at 4-bit
- **What it is**  
  A small vision-language model, Apache-2.0 licensed, supports 1024×1024 resolution, accepts PDF-rendered images directly. The model card includes `train_lora.py` and `chat_template` examples, ready to plug into Unsloth workflows for fine-tuning on your own corpus (e.g., traditional Chinese financial reports).
- **Where it fits**  
  - SaaS backend engineers running "invoice/receipt auto-parsing" on 12GB VRAM machines; model card notes 1024×1024 support and direct PDF-rendered image input.  
  - Edge device engineers quantizing to 4-bit GGUF for deployment on Jetson Orin NX (16GB) for real-time "screen-content QA" test assistance.  
  - Data scientists LoRA-fine-tuning on proprietary docs (traditional Chinese financial reports); model card provides `train_lora.py` and `chat_template` examples, directly compatible with Unsloth.
- **Can it run?**  
  Official model card: "FP16 needs 8GB VRAM, 8-bit needs 5GB, 4-bit GGUF ~3.2GB memory"; pre-quantized GGUF (Q4_K_M, Q5_K_M) provided. This means 8-bit should theoretically run on 8GB consumer GPUs, and laptops with enough system RAM could even run 4-bit GGUF on CPU.
- **Buzz this week**  
  HF trending score 347, 18.4k downloads, 434 likes.
- **vs. peers**  
  Author compares against Qwen2-VL-2B and MiniCPM-V-2_6 in the model card, claiming 3–5 point gains on DocVQA; self-reported numbers, no public third-party reproduction yet.
- **How to get it**  
  README instructs: `from transformers import AutoModelForImageTextToText; model = AutoModelForImageTextToText.from_pretrained("thinkingmachines/Inkling-2-Small", load_in_8bit=True, device_map="auto")`; or load GGUF with `llama.cpp`.
- **Link**  
  [https://huggingface.co/thinkingmachines/Inkling-2-Small](https://huggingface.co/thinkingmachines/Inkling-2-Small)

### 3. FluidVoice-Labs/fluid-speech-pro: 0.5B ASR Built for CPU Inference, Runs Offline Transcription on Pi 5
- **What it is**  
  A 0.5B-class model specialized for Automatic Speech Recognition (ASR), CC-BY-4.0 licensed, commercial use and redistribution allowed, no application needed. Model card provides ONNX and CTranslate2 versions, emphasizing "CPU real-time factor < 0.3".
- **Where it fits**  
  - Full-stack engineers running "auto-generate podcast transcripts" in GPU-less CI/CD pipelines; model card provides ONNX and CTranslate2 versions, ready to `pip install ctranslate2` for inference.  
  - Embedded engineers deploying offline mixed Chinese-English recognition on Raspberry Pi 5 / Orange Pi 5 Plus; model card cites `int8` quantization to 40MB, CPU real-time factor 0.25.  
  - Privacy-sensitive teams (medical/legal) batch-transcribing recordings on air-gapped servers; CC-BY-4.0 permits commercial use and redistribution without application.
- **Can it run?**  
  Official model card: "FP16 0.5B ~1GB VRAM; INT8 quantized CPU real-time factor 0.25 (Intel i7-12700), 0.35 (Apple M2); ONNX Runtime / CTranslate2 both supported". Machines without discrete GPUs, even Pi 5-class devices, can theoretically run offline as long as CPU performance suffices and memory fits the 40MB model.
- **Buzz this week**  
  HF trending score 49, 9.1k downloads, 53 likes.
- **vs. peers**  
  Author compares against Whisper-large-v3 (1.5B) and distil-whisper-large-v3 (756M) in the model card, claiming WER on CommonVoice zh-TW test set within 0.8% while being 4x faster; self-reported, no public third-party reproduction yet.
- **How to get it**  
  README lists three ways: ① `pip install fluid-voice` then `transcribe("audio.wav")`; ② download ONNX with `onnxruntime-genai`; ③ download CTranslate2 model with `ctranslate2.Translator("model_ct2")`.
- **Link**  
  [https://huggingface.co/FluidVoice-Labs/fluid-speech-pro](https://huggingface.co/FluidVoice-Labs/fluid-speech-pro)

## Closing
All three models this week sit at the intersection of "small parameter count, solid quantization, permissive licensing": Haru-4B brings speech-to-speech down to a single consumer GPU, Inkling-2-Small compresses document understanding to 3.2GB, and fluid-speech-pro pushes transcription onto GPU-less CI/CD and Pi 5. If you have 16GB+ VRAM and want to try real-time speech translation, grab Haru-4B's GGUF for a quick feel; if you have zero GPU and just need transcripts, fluid-speech-pro's Python package is the lowest-friction path. See you next time.

---

## About this article and its author

Originally published on [Mark Ku's Tech Notes](https://blog.markkulab.net/en/local-ai-lab/local-ai-lab-2026-11-17)

License: [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) — when reusing or quoting, credit the author and link back to the original

### About the author

**[Mark Ku](https://blog.markkulab.net/en/author/mark-ku)** — Software engineer

- 10+ years as a software engineer
- Built North-American e-commerce and AI SaaS subscription billing

### Free tools built by the author

All of these are free to use:

- [Free PDF Sign Tool](https://blog.markkulab.net/en/tools/pdf-sign): Online PDF sign tool — draw, type, or upload a signature, then drag, resize, and download. Everything runs in your browser; nothing is uploaded.
- [VS Code Refactory](https://blog.markkulab.net/en/tools/refactory): Refactory is a VS Code refactoring extension: 34 actions plus a 37-rule code-smell inspection layer with a Code Health dashboard, across 18 languages, backed by 534 tests. It learns your repo's conventions: where interfaces live, where DI is registered, whether 'use client' belongs. It ranks files by git churn × complexity so you know what to fix first, and hands any smell to the Claude Code already on your machine. Free to use, and your source never leaves your computer.
- [DB-Kit Database Manager](https://blog.markkulab.net/en/tools/db-kit): DB-Kit is a lightweight, cross-platform database manager built with Tauri + Rust + React. Manage MySQL, MariaDB, PostgreSQL, SQL Server, Oracle, SQLite, MongoDB, Redis, Kafka, Elasticsearch and RabbitMQ from one consistent interface: passwords encrypted in the OS keychain, SSH tunnels, full CRUD, a visual query builder, stacked multi-statement result sets, cross-connection data transfer, schema & data compare with sync SQL and schema snapshots, Excel / CSV import & export, visualized execution plans, ER diagrams, scheduled backups, SQL stress testing with p50–p99 latency percentiles, a 15-rule SQL review engine, Review & Run (AI review plus per-statement backups and an auto-generated rollback script), Kafka message browsing with monitoring & alerts, an SSH terminal with SFTP / FTP, Docker / Kubernetes, remote desktop (RDP / VNC / RustDesk), file / folder compare, stored procedure integration tests, six UI languages, a built-in AI assistant (a local CLI or any Anthropic / OpenAI-compatible API; natural-language SQL, AI review and tuning advice) and the dbk CLI. Free and open source (MIT), with installers for Windows, macOS and Linux.
- [VS Code Super Mermaid](https://blog.markkulab.net/en/tools/super-mermaid): Super Mermaid is a VS Code extension for beautiful Mermaid diagrams out of the box: auto-colored live preview, mouse pan & zoom, high-res PNG / SVG export, 21 templates and multiple themes. Free and open source (MIT).
- [React Super Mermaid](https://blog.markkulab.net/en/tools/react-super-mermaid): react-super-mermaid is an open-source React component library: render beautiful Mermaid diagrams with a single <MermaidViewer>, with built-in colorful / sketch themes, pan & zoom, in-diagram search, and high-res SVG / PNG export. Lightweight, SSR-safe, fully typed. Free and open source (MIT).
- [Jira / Confluence Super Mermaid](https://blog.markkulab.net/en/tools/jira-super-mermaid): An Atlassian Forge app: write Mermaid syntax directly inside a Jira issue or a Confluence page and get flowcharts, sequence diagrams, state machines and Gantt charts. 11 diagram types, SVG / PNG export, light and dark themes, full CJK support. Runs on Atlassian: your diagrams live in your own site and the app calls no third-party service. Free, coming soon to the Atlassian Marketplace.
- [Mermaid Live Preview](https://blog.markkulab.net/en/tools/mermaid-preview): Write Mermaid in your browser, see it render instantly, and share the whole diagram as a single link. No sign-up, nothing uploaded to a server, and mermaid.live share links work as-is.
- [React Intl Phone Number](https://blog.markkulab.net/en/tools/react-intl-phone-number): react-intl-phone-number is an open-source React component: framework-agnostic and antd-free, with E.164 in/out, a searchable flag / country-code dropdown, configurable validation levels (strict / mobile-strict / loose), themeable CSS, and i18n — phone logic powered by google-libphonenumber. Lightweight and fully typed. Free and open source (MIT).
- [Uptime Kuma Cluster](https://blog.markkulab.net/en/tools/uptime-kuma-cluster): Turn single-node Uptime Kuma into a highly available cluster: OpenResty + Lua smart load balancing, shared MariaDB state, health checks and automatic failover, plus cluster-management REST APIs. One Docker Compose command to start. Free and open source (MIT).
- [AI Podcast Cut](https://blog.markkulab.net/en/tools/ai-podcast-cut): Drop in a recording and it removes fillers and stutters, levels loudness segment by segment, and sends a second agent to review every cut. Cut points snap to word boundaries and zero crossings, every splice gets a fade, and sentence-end breaths are preserved. Desktop app for Windows, macOS and Linux. MIT licensed; the Windows installer bundles ffmpeg.
- [AI Video Cut](https://blog.markkulab.net/en/tools/ai-video-cut): Pick something in your video: type it (face, license plate, phone screen, logo), click points or drag a box, or let Claude Code / Codex pick it. AI Video Cut tracks it frame by frame, then mosaics or blurs it, recolors it, pins stickers and text to it, swaps a screen or poster for your own image or video, or removes it using background that other frames actually captured. Plus sequence editing, local captions and an AI assistant. Free, open-source (MIT) Windows desktop app (macOS / Linux experimental) that runs locally, nothing uploaded.
- [open-pos restaurant POS](https://blog.markkulab.net/en/tools/open-pos): One computer and one receipt printer is enough to open the shop. Your data lives on your own disk, no subscription, no lock-in, MIT licensed. Money is integer New Taiwan dollars with tax split by the statutory formula, so sales + tax always equals the total. Printing goes straight over ESC/POS on TCP 9100, with no vendor driver. Tauri + Rust + SQLite desktop app, v1.0 in development.
- [.NET e-commerce API template](https://blog.markkulab.net/en/tools/dotnet-ecommerce-api): An e-commerce backend API template built on .NET 10: Controller → Service → Repository layering, Autofac convention-based registration, SqlSugar for MySQL / MariaDB and SQL Server, Redis cache and message queue, Hangfire jobs, JWT auth, and an extensible payment provider module (ECPay, Newebpay). Apache 2.0.
- [Open Token Monitor team AI usage](https://blog.markkulab.net/en/tools/open-token-monitor): Collect token usage and equivalent cost from every employee's AI coding tools into a hub you host yourself: PostgreSQL, org roster, dashboards compared by company and department, and a reports API. Electron and Rust/Tauri clients upload every 30 minutes. MIT licensed, one Docker command.
- [ai-security-monitor defensive security](https://blog.markkulab.net/en/tools/ai-security-monitor): Your website is live. Who checks its certificates, security settings and known vulnerabilities? A self-hosted Security Hub brings website scans and host monitoring together, deduplicates findings and compares scans after fixes.

### Daily podcasts

- [Mark's Tech Insights — Daily AI News](https://blog.markkulab.net/en/category/tech-news): Daily curated AI and tech trends. Catch the latest developments via audio summaries — covering AI applications, software architecture, DevOps, and engineering practice. — RSS: https://blog.markkulab.net/feed.xml
- [AI股市蝦聊](https://blog.markkulab.net/en/category/ai-stock-chat): Every trading day, an AI-analyzed take on the Taiwan stock market, delivered as a two-host conversation covering the session and the next-day outlook. — RSS: https://blog.markkulab.net/ai-stock-chat/feed.xml
- [開源好物日報](https://blog.markkulab.net/en/category/open-source-weekly): A daily two-host pick of free open-source tools surfaced from real Hacker News, GitHub, and Reddit buzz — what pain they solve and the fastest way to get started. — RSS: https://blog.markkulab.net/open-source-weekly/feed.xml
- [AI 運動日報](https://blog.markkulab.net/en/category/sports-weekly): Two hosts talk NBA, MLB and world sport every day — scores, records and the stories behind them, from a Taiwanese fan perspective. — RSS: https://blog.markkulab.net/sports-weekly/feed.xml
- [AI 國際新聞快報](https://blog.markkulab.net/en/category/world-news): A daily two-host briefing that makes sense of the past 24 hours in world news — geopolitics, the global economy, conflict and security, disasters and climate — from a Taiwanese perspective, neutral and fully sourced. — RSS: https://blog.markkulab.net/world-news/feed.xml
- [地端 AI 實驗室](https://blog.markkulab.net/en/category/local-ai-lab): Every day, a two-host look at open-weight models and LoRAs you can actually run on your own machine: what they are for, whether your GPU can handle them, and how they compare — sourced from official model cards. — RSS: https://blog.markkulab.net/local-ai-lab/feed.xml
- [AI 行銷日報](https://blog.markkulab.net/en/category/ai-marketing): Every day, two hosts break down the latest in AI for marketing, advertising, SEO/GEO and content marketing: practical, never sponsored, every story sourced. — RSS: https://blog.markkulab.net/ai-marketing/feed.xml
- [AI 資安日報](https://blog.markkulab.net/en/category/ai-security): Every day, two hosts cover the latest where AI meets cybersecurity: LLM and AI agent vulnerabilities, prompt injection, model supply-chain risks and AI-powered attacks, from a defense-first angle with every story sourced. — RSS: https://blog.markkulab.net/ai-security/feed.xml

### Newsletter

[Subscribe to the newsletter](https://blog.markkulab.net/en/subscribe) — Be the first to know about new posts. No spam, unsubscribe anytime.
