---
title: "🧪 322M beats 28B? convaiinnovations/laya claims to be 7.8x faster than the competition｜Local AI Lab"
description: "This week's hottest isn't a heavyweight model but a 322M decision classifier, convaiinnovations/laya. The official card pits it against 26B-28B class rivals, claiming higher accuracy and 7.8x faster speed, we'll tell you how to read this self-reported data in a moment."
canonical_url: "https://blog.markkulab.net/en/local-ai-lab/local-ai-lab-2026-10-16"
author: "Mark Ku"
author_url: "https://blog.markkulab.net/en/author/mark-ku"
site: "Mark Ku's Tech Notes"
date_published: "2026-10-16T02:00:00.000Z"
category: "Local AI Lab"
tags: ["local-ai", "podcast", "地端模型", "lora", "開源模型", "embeddinggemma2", "UnlimitedOCR", "laya", "RAG", "OCR", "LLM路由"]
language: "en"
license: "CC BY 4.0"
license_url: "https://creativecommons.org/licenses/by/4.0/"
attribution: "when reusing or quoting, credit the author and link back to the original"
---

# 🧪 322M beats 28B? convaiinnovations/laya claims to be 7.8x faster than the competition｜Local AI Lab

## Opening

This week's biggest buzz isn't about a giant model, it's a mere 322M decision classifier, convaiinnovations/laya. The official card itself compares it against 26B to 28B-class rivals, claiming higher accuracy and 7.8x faster speed, and we'll walk through how to read that self-reported data in a moment. This episode also features an OCR model, baidu/Unlimited-OCR, which has surged past 1.2 million downloads and ships with 37 quantized variants, plus google/embeddinggemma-2, a model that packs text, images, and audio into a single vector space.

## This Week's Picks

### 1. google/embeddinggemma-2: One Model, Three Modalities, a RAG Appetizer for Text, Images, and Audio

- **What it is**: google/embeddinggemma-2 is a multimodal embedding model from Google with 740M total parameters (270M text backbone + 170M vision encoder + 300M audio encoder), released under the Apache-2.0 license. The official model card states that text (including code), images, video, and audio are all mapped into the same 768-dimensional vector space, rather than being handled by several separately trained embedding models.
- **Where it fits**:
  - Backend engineers wanting to build an internal document search system can chunk technical docs and meeting notes exported from Confluence or Notion and feed them into pgvector or Qdrant. The official card lists support for 100+ languages and an 8192-token context, so mixed Chinese/English internal documents can share a single index without needing a separate embedding model for Chinese.
  - Small teams building a knowledge base who want to search presentation screenshots and meeting recordings with a single query: the card states that text, images, video, and audio all map into the same 768-dimensional vector space, eliminating the need to maintain three separate embedding services.
  - Teams facing tight vector database bills or hardware constraints: the card states it supports Matryoshka Representation Learning (MRL) truncation down to 512, 256, or 128 dimensions, with minimal quality loss at 256 dimensions, cutting vector storage costs by up to sixfold. It also allows loading only the modalities you need (text-only at 270M, or text + image at 440M), making it suitable for sidecar containers or laptops.
- **Can you actually run it?**: The official source doesn't give a specific VRAM figure, but the model card emphasizes it's "designed to run on consumer hardware such as mobile devices and laptops." Note that the official docs explicitly warn to use bfloat16 or float32 for inference, since float16 can produce NaNs or degrade embedding quality. The page currently lists no GGUF-style quantized versions.
- **This week's buzz**: HF trending score of 1005, with 144.3k downloads and 2.4k likes, the highest buzz of this episode. The same family also includes 2 quantized/repackaged variants.
- **How it stacks up**: To be clear upfront, the self-reported scores on the card, MTEB multilingual 61.36, MTEB code 78.68 (the official source claims roughly a 14% improvement on code tasks over the previous generation), plus the cross-modal scores MIEB 64.64 (image), MMEB 50.67 (video), and MSEB 69.54 (audio), are all compared only against "the previous generation EmbeddingGemma," not against third-party models like bge-m3 or multilingual-e5. Its real differentiator isn't so much the scores as the fact that one model handles four modalities and is small enough to fit on a laptop.
- **How to get it**: The README's shortest path is via sentence-transformers: `SentenceTransformer("google/embeddinggemma-2")` then `model.encode(sentences)`. You can also go through transformers with `AutoProcessor` plus `AutoModel.from_pretrained(..., device_map="auto")`.
- **Link**: [https://huggingface.co/google/embeddinggemma-2](https://huggingface.co/google/embeddinggemma-2)

### 2. baidu/Unlimited-OCR: Stop Uploading Invoices and Contracts to the Cloud, MIT-Licensed with 37 Quantized Variants Built In

- **What it is**: baidu/Unlimited-OCR is an OCR model from Baidu with 3B parameters in BF16, released under the MIT license. The README provides `model.infer()` for processing single images and `model.infer_multi()` for processing multi-page documents and PDFs.
- **Where it fits**:
  - Engineers doing freelance or SaaS backend work who need to process invoices, quotes, and multi-page PDF contracts, but can't upload them to the cloud due to privacy or confidentiality concerns, can keep the entire document on their own machine and parse it with `model.infer_multi()`.
  - Anyone digitizing paper materials into a personal knowledge base, scanning handouts, photographing whiteboards, or converting book pages into searchable text, can feed the results into Obsidian or Notion. The official page lists 37 quantized variants that work with Ollama, LM Studio, and Jan, so you don't have to deal with a CUDA setup yourself.
  - Teams currently paying per-page for a cloud OCR API and wanting to self-host instead: the model card provides a vLLM Docker image (available in CUDA 13.0 and 12.9 versions) and instructions for launching with SGLang, letting you spin up a persistent internal service to replace that external billed endpoint.
- **Can you actually run it?**: Officially it requires an NVIDIA GPU; the tested environments listed are `torch==2.10.0`, `torchvision==0.25.0`, and `transformers==4.57.1`, but the official source gives no specific VRAM threshold. The model itself is 3B in BF16; separately, the page lists 37 quantized variants compatible with llama.cpp, LM Studio, Jan, and Ollama, which lowers the hardware bar considerably further.
- **This week's buzz**: HF trending score of 320, with 1.2M downloads and 4.7k likes.
- **How it stacks up**: The card self-reports llamaindex/ParseBench results with a mean of 46.17 and Text Content at 86.81, but Text Formatting comes in at just 0.97, a figure the official source itself doesn't hide. This suggests plain text extraction and layout/format reconstruction are clearly two different things, so anyone needing structural layout preservation should verify it themselves. PaddleOCR and DeepSeek-OCR only appear in the card's acknowledgments section, with no direct comparison data against either provided officially. Also, this card provides no supported language list at all, so performance on Traditional Chinese and handwriting recognition can't be inferred from the official documentation.
- **How to get it**: The README's transformers path is `AutoModel.from_pretrained("baidu/Unlimited-OCR")`, which requires adding `trust_remote_code=True`, then calling either `model.infer()` or `model.infer_multi()`. For running it as a service, the official source also offers either a vLLM Docker image or an SGLang Python server.
- **Link**: [https://huggingface.co/baidu/Unlimited-OCR](https://huggingface.co/baidu/Unlimited-OCR)

### 3. convaiinnovations/laya: A Ticket Router That Runs Without a GPU, Claiming 7.8x Speed Over Rivals

- **What it is**: convaiinnovations/laya is a semantic classification/routing model that doesn't generate text, it only returns probabilities. It's released under the Apache-2.0 license and comes in two versions: an English version at 421M (ModernBERT-large backbone) and a multilingual version at 322M (mmBERT-base backbone).
- **Where it fits**:
  - Engineers running their own customer support or ticketing systems who need automated routing: the model card's examples include routing between billing and technical support, urgency scoring, and churn risk. You can put this model in front of your API to classify ticket category and urgency, without calling a large model for every single ticket.
  - Gatekeeping or routing in front of an LLM, deciding whether a request should go to a large model at all, and if so, which one: the card emphasizes that it returns "calibrated probabilities" and generates no text whatsoever, so there's no hallucination risk and no need to write defensive JSON parsing.
  - If all you have is a GPU-less VPS but you still want semantic classification capability, the official documentation mentions CPU fallback support and includes an `laya-serve` HTTP server plus LangChain integration, so you can spin up a classification endpoint right alongside your existing Python service.
- **Can you actually run it?**: No minimum hardware threshold is officially stated, but the model card's parameters and benchmark table are worth citing: the English version is 421M (about 808MB to download), the multilingual version is 322M (about 647MB). The benchmark table shows single-query latency of 32.8 to 39.5 milliseconds on a Tesla T4, and explicitly states CPU fallback is supported, with optional ONNX Runtime and TileLang GPU fast paths.
- **This week's buzz**: HF trending score of 189, with 48.9k downloads and 5.6k likes.
- **How it stacks up**: laya's own card directly names a rival, Jev, self-reporting a typed-decisions accuracy of 0.766 (fine-tuned version) versus Jev's 0.727, speed of 32.8ms versus 236 to 276ms (the official claim is 7.8x faster), and calibration ECE of 0.081 versus Jev's 0.246. All of these are laya's unilateral self-reported figures, not third-party benchmarks. The official page also points out that Cloudflare/clef's card states its test environment was a single H200, vllm-sr/d3's benchmark ran on an AMD Instinct MI325X, and autotrust/JEV-27B-VL explicitly states that full 256K context requires 80GB or more of VRAM. By comparison, laya needs only a few hundred MB and, per the official source, can fall back to CPU. The card also discloses its own limitations: although it claims support for 100+ languages, in actual testing only 45 of 51 languages tested met the usability bar of performing at least 3x better than random chance.
- **How to get it**: The README's shortest example uses `from laya import Router`, `router = Router()`, and `router.predict(state, questions)`. The official source also states you can use it directly via Hugging Face transformers, spin up an HTTP server with `laya-serve`, or integrate it through LangChain.
- **Link**: [https://huggingface.co/convaiinnovations/laya](https://huggingface.co/convaiinnovations/laya)

## Closing

This episode's three models happen to address three different flavors of self-hosting anxiety: embeddinggemma-2 solves the problem of whether you need to maintain several separate services for multimodal RAG, Unlimited-OCR solves the problem of documents that can't go to the cloud, and laya solves the problem of not wanting to burn large-model tokens on gatekeeping classification. If you can only pick one to try first, laya's small footprint and CPU fallback make it probably the lowest-barrier starting point, but remember all its comparison data is self-reported, so it's worth validating against your own real-world data before putting it into production. See you next episode.

---

## About this article and its author

Originally published on [Mark Ku's Tech Notes](https://blog.markkulab.net/en/local-ai-lab/local-ai-lab-2026-10-16)

License: [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) — when reusing or quoting, credit the author and link back to the original

### About the author

**[Mark Ku](https://blog.markkulab.net/en/author/mark-ku)** — Software engineer

- 10+ years as a software engineer
- Built North-American e-commerce and AI SaaS subscription billing

### Free tools built by the author

All of these are free to use:

- [Free PDF Sign Tool](https://blog.markkulab.net/en/tools/pdf-sign): Online PDF sign tool — draw, type, or upload a signature, then drag, resize, and download. Everything runs in your browser; nothing is uploaded.
- [VS Code Refactory](https://blog.markkulab.net/en/tools/refactory): Refactory is a VS Code refactoring extension: 34 actions plus a 37-rule code-smell inspection layer with a Code Health dashboard, across 18 languages, backed by 534 tests. It learns your repo's conventions: where interfaces live, where DI is registered, whether 'use client' belongs. It ranks files by git churn × complexity so you know what to fix first, and hands any smell to the Claude Code already on your machine. Free to use, and your source never leaves your computer.
- [DB-Kit Database Manager](https://blog.markkulab.net/en/tools/db-kit): DB-Kit is a lightweight, cross-platform database manager built with Tauri + Rust + React. Manage MySQL, MariaDB, PostgreSQL, SQL Server, Oracle, SQLite, MongoDB, Redis, Kafka, Elasticsearch and RabbitMQ from one consistent interface: passwords encrypted in the OS keychain, SSH tunnels, full CRUD, a visual query builder, stacked multi-statement result sets, cross-connection data transfer, schema & data compare with sync SQL and schema snapshots, Excel / CSV import & export, visualized execution plans, ER diagrams, scheduled backups, SQL stress testing with p50–p99 latency percentiles, a 15-rule SQL review engine, Review & Run (AI review plus per-statement backups and an auto-generated rollback script), Kafka message browsing with monitoring & alerts, an SSH terminal with SFTP / FTP, Docker / Kubernetes, remote desktop (RDP / VNC / RustDesk), file / folder compare, stored procedure integration tests, six UI languages, a built-in AI assistant (a local CLI or any Anthropic / OpenAI-compatible API; natural-language SQL, AI review and tuning advice) and the dbk CLI. Free and open source (MIT), with installers for Windows, macOS and Linux.
- [VS Code Super Mermaid](https://blog.markkulab.net/en/tools/super-mermaid): Super Mermaid is a VS Code extension for beautiful Mermaid diagrams out of the box: auto-colored live preview, mouse pan & zoom, high-res PNG / SVG export, 21 templates and multiple themes. Free and open source (MIT).
- [React Super Mermaid](https://blog.markkulab.net/en/tools/react-super-mermaid): react-super-mermaid is an open-source React component library: render beautiful Mermaid diagrams with a single <MermaidViewer>, with built-in colorful / sketch themes, pan & zoom, in-diagram search, and high-res SVG / PNG export. Lightweight, SSR-safe, fully typed. Free and open source (MIT).
- [Jira / Confluence Super Mermaid](https://blog.markkulab.net/en/tools/jira-super-mermaid): An Atlassian Forge app: write Mermaid syntax directly inside a Jira issue or a Confluence page and get flowcharts, sequence diagrams, state machines and Gantt charts. 11 diagram types, SVG / PNG export, light and dark themes, full CJK support. Runs on Atlassian: your diagrams live in your own site and the app calls no third-party service. Free, coming soon to the Atlassian Marketplace.
- [Mermaid Live Preview](https://blog.markkulab.net/en/tools/mermaid-preview): Write Mermaid in your browser, see it render instantly, and share the whole diagram as a single link. No sign-up, nothing uploaded to a server, and mermaid.live share links work as-is.
- [React Intl Phone Number](https://blog.markkulab.net/en/tools/react-intl-phone-number): react-intl-phone-number is an open-source React component: framework-agnostic and antd-free, with E.164 in/out, a searchable flag / country-code dropdown, configurable validation levels (strict / mobile-strict / loose), themeable CSS, and i18n — phone logic powered by google-libphonenumber. Lightweight and fully typed. Free and open source (MIT).
- [Uptime Kuma Cluster](https://blog.markkulab.net/en/tools/uptime-kuma-cluster): Turn single-node Uptime Kuma into a highly available cluster: OpenResty + Lua smart load balancing, shared MariaDB state, health checks and automatic failover, plus cluster-management REST APIs. One Docker Compose command to start. Free and open source (MIT).
- [AI Podcast Cut](https://blog.markkulab.net/en/tools/ai-podcast-cut): Drop in a recording and it removes fillers and stutters, levels loudness segment by segment, and sends a second agent to review every cut. Cut points snap to word boundaries and zero crossings, every splice gets a fade, and sentence-end breaths are preserved. Desktop app for Windows, macOS and Linux. MIT licensed; the Windows installer bundles ffmpeg.
- [AI Video Cut](https://blog.markkulab.net/en/tools/ai-video-cut): Pick something in your video: type it (face, license plate, phone screen, logo), click points or drag a box, or let Claude Code / Codex pick it. AI Video Cut tracks it frame by frame, then mosaics or blurs it, recolors it, pins stickers and text to it, swaps a screen or poster for your own image or video, or removes it using background that other frames actually captured. Plus sequence editing, local captions and an AI assistant. Free, open-source (MIT) Windows desktop app (macOS / Linux experimental) that runs locally, nothing uploaded.
- [open-pos restaurant POS](https://blog.markkulab.net/en/tools/open-pos): One computer and one receipt printer is enough to open the shop. Your data lives on your own disk, no subscription, no lock-in, MIT licensed. Money is integer New Taiwan dollars with tax split by the statutory formula, so sales + tax always equals the total. Printing goes straight over ESC/POS on TCP 9100, with no vendor driver. Tauri + Rust + SQLite desktop app, v1.0 in development.
- [.NET e-commerce API template](https://blog.markkulab.net/en/tools/dotnet-ecommerce-api): An e-commerce backend API template built on .NET 10: Controller → Service → Repository layering, Autofac convention-based registration, SqlSugar for MySQL / MariaDB and SQL Server, Redis cache and message queue, Hangfire jobs, JWT auth, and an extensible payment provider module (ECPay, Newebpay). Apache 2.0.
- [Open Token Monitor team AI usage](https://blog.markkulab.net/en/tools/open-token-monitor): Collect token usage and equivalent cost from every employee's AI coding tools into a hub you host yourself: PostgreSQL, org roster, dashboards compared by company and department, and a reports API. Electron and Rust/Tauri clients upload every 30 minutes. MIT licensed, one Docker command.

### Daily podcasts

- [Mark's Tech Insights — Daily AI News](https://blog.markkulab.net/en/category/tech-news): Daily curated AI and tech trends. Catch the latest developments via audio summaries — covering AI applications, software architecture, DevOps, and engineering practice. — RSS: https://blog.markkulab.net/feed.xml
- [AI股市蝦聊](https://blog.markkulab.net/en/category/ai-stock-chat): Every trading day, an AI-analyzed take on the Taiwan stock market, delivered as a two-host conversation covering the session and the next-day outlook. — RSS: https://blog.markkulab.net/ai-stock-chat/feed.xml
- [開源好物週報](https://blog.markkulab.net/en/category/open-source-weekly): A weekly two-host pick of free open-source tools surfaced from real Hacker News, GitHub, and Reddit buzz — what pain they solve and the fastest way to get started. — RSS: https://blog.markkulab.net/open-source-weekly/feed.xml
- [AI 運動週報](https://blog.markkulab.net/en/category/sports-weekly): Two hosts talk NBA, MLB and world sport three times a week — scores, records and the stories behind them, from a Taiwanese fan perspective. — RSS: https://blog.markkulab.net/sports-weekly/feed.xml
- [AI 國際新聞快報](https://blog.markkulab.net/en/category/world-news): A daily two-host briefing that makes sense of the past 24 hours in world news — geopolitics, the global economy, conflict and security, disasters and climate — from a Taiwanese perspective, neutral and fully sourced. — RSS: https://blog.markkulab.net/world-news/feed.xml
- [地端 AI 實驗室](https://blog.markkulab.net/en/category/local-ai-lab): Twice a week, a two-host look at open-weight models and LoRAs you can actually run on your own machine: what they are for, whether your GPU can handle them, and how they compare — sourced from official model cards. — RSS: https://blog.markkulab.net/local-ai-lab/feed.xml

### Newsletter

[Subscribe to the newsletter](https://blog.markkulab.net/en/subscribe) — Be the first to know about new posts. No spam, unsubscribe anytime.