--- title: "Self-Host a Local AI API Gateway with LiteLLM: One Entrance to Manage All Models, Handling Usage, Costs, Failover, and API Keys in One Go" description: "Self-host an on-prem AI API Gateway with LiteLLM Proxy: a single OpenAI-compatible endpoint integrating 29 cloud and local models, with use-case aliases, usage and cost dashboards, multi-tier Fallback chains, and virtual key management, plus a complete config.yaml example." canonical_url: "https://blog.markkulab.net/en/post/litellm-self-hosted-ai-api-gateway" author: "Mark Ku" author_url: "https://blog.markkulab.net/en/author/mark-ku" site: "Mark Ku's Tech Notes" date_published: "2026-11-07T09:15:05.000Z" category: "AI" tags: ["LiteLLM", "AI Gateway", "Ollama", "NVIDIA NIM", "Fallback", "LLMOps", "成本管理"] language: "en" license: "CC BY 4.0" license_url: "https://creativecommons.org/licenses/by/4.0/" attribution: "when reusing or quoting, credit the author and link back to the original" --- # Self-Host a Local AI API Gateway with LiteLLM: One Entrance to Manage All Models, Handling Usage, Costs, Failover, and API Keys in One Go ## Introduction I have several projects calling AI simultaneously: blog post and podcast generation, stock tools, eSIM plan comparison, plus the Agent I use for daily coding. Previously, each project integrated vendors independently: Claude for writing, NVIDIA free models for volume, OpenAI for images, local Ollama on the LAN for on-prem models. The result was three classic problems: - **Scattered keys**: Every project's `.env` held a copy of vendor keys; rotating one meant updating multiple places. - **Opaque costs**: You only found out which project burned the most money at month-end. - **Single points of failure**: NVIDIA quota exhausted, Claude returns 429, the whole pipeline stops, requiring manual code changes to switch models. So I self-hosted a **LiteLLM Proxy (v1.108)** as my personal on-prem AI API Gateway. Externally it presents `https://litellm.markkulab.net`. All applications hit this single OpenAI-compatible endpoint; the Gateway decides which model actually receives the request, falls back when quota runs out, and logs usage and spend. Upstream keys live only on the Gateway. Reasons for choosing LiteLLM were practical: 1. **OpenAI-compatible**: Applications only swap `base_url` and the key; code barely changes. 2. **Cloud and local treated equally**: Anthropic, NVIDIA, OpenAI, Ollama, self-hosted services are all just rows in `model_list`. 3. **Governance built-in**: Virtual Keys, budgets, rate limits, fallbacks, usage dashboard, all available out of the box, plus an Admin UI for web-based config changes. ## Overall Architecture ```mermaid flowchart LR subgraph Apps["應用端（各自一把 Virtual Key）"] A1["部落格產文
podcast"] A2["股票 / eSIM
小工具"] A3["Agent / IDE
（OpenAI 相容）"] end subgraph GW["LiteLLM Proxy：地端 AI API Gateway"] K["Virtual Key 驗證
預算・限速・可用模型"] R["Router
用途別名 text-default / image-default…"] F["Fallback 鏈
429・額度不足・逾時就往下掉"] L[("PostgreSQL
用量・花費・Logs")] C["LLM Credentials
上游金鑰集中保管"] K --> R --> F R -.記帳.-> L C -.注入.-> F end subgraph Cloud["雲端"] CL["Claude"] NV["NVIDIA 免費模型
mistral-large・glm-flash
step-5・nemotron・flux"] OA["OpenAI 出圖"] end subgraph Local["地端（區網 GPU）"] OL["Ollama
qwen3.8・ornith・qwen3.5-coder"] AG["ai-gen 出圖"] TT["TTS 服務"] end A1 --> K A2 --> K A3 --> K F --> CL F --> NV F --> OA F --> OL F --> AG F --> TT ``` A request passes through three layers in sequence: 1. **Virtual Key validation**: Which models this key can access, remaining budget, requests per minute. 2. **Router**: Maps the application-specified model name (usually a purpose-based alias) to an actual deployment. 3. **Fallback chain**: If the primary model fails, it tries the next in order; if all cloud options fail, it falls back to local. Every call's tokens, cost, latency, and success/failure are written to PostgreSQL; the Admin UI dashboard computes its metrics from this data. ## 1. Unifying Cloud and Local Models ![LiteLLM Models + Endpoints page: listing NVIDIA, ai-gen, OpenAI, and local Ollama models simultaneously](https://blog.markkulab.net/content/markku/posts/litellm-self-hosted-ai-api-gateway/images/litellm-models-endpoints.webp) Currently **29 models** are registered on the Gateway, broadly falling into four categories: | Source | Examples | Credentials | |--------|----------|-------------| | Cloud Claude | `cloud-claude`、`cloud-claude-sonnet`、`cloud-claude-haiku` | Anthropic | | NVIDIA free models | `cloud-nvidia-mistral-large`、`cloud-nvidia-glm-flash`、`cloud-nvidia-step-5`、`cloud-nvidia-nemotron-ultra`、`cloud-nvidia-nemotron-super-vl`、`cloud-nvidia-flux-klein` (image gen) | `nvidia` | | Cloud image gen | `cloud-openai-image`、`aigen-web-image-gpt-image-2.5-flare` | `openai-api` | | Local | Ollama's `local-ollama-qwen3.8-27b-text`、`local-ollama-ornith-1.5-35b-text`、`local-ollama-qwen3-coder-30b-text`、`local-ollama-qwen3.5-coder-35b-text`、`local-ollama-qwen3-vl-8b-text`; self-hosted image gen `image-default`、`aigen-image-native` | `aigen`／Manual | I deliberately prefixed names with `cloud-` and `local-` so logs and leaderboards instantly reveal whether a call was paid or local. The Credentials column in the screenshot is also telling: all NVIDIA models point to the same `nvidia` credential, ai-gen points to `aigen`, and local Ollama shows Manual since no key is needed. Upstream keys exist only on the Gateway; models merely "reference" them, key management expands on this below. ### config.yaml Example I usually add models via the Admin UI, but YAML is clearer for documentation. Below is a condensed example (keys all read from environment variables): ```yaml # 可重複使用的上游憑證：模型只寫名字，不重複貼金鑰 credential_list: - credential_name: nvidia credential_values: api_key: os.environ/NVIDIA_API_KEY api_base: https://integrate.api.nvidia.com/v1 model_list: # 雲端 Claude - model_name: cloud-claude litellm_params: model: anthropic/claude-sonnet-5 # 換成你要用的版本 api_key: os.environ/ANTHROPIC_API_KEY # NVIDIA 免費模型：走 OpenAI 相容格式，引用 nvidia 憑證 - model_name: cloud-nvidia-step-5 litellm_params: model: openai/stepfun-ai/step-5-preview litellm_credential_name: nvidia # 地端 Ollama：單價明確設 0，儀表板才不會把它算進花費 - model_name: local-ollama-qwen3.8-27b-text litellm_params: model: ollama_chat/qwen/qwen3.8-27b:latest api_base: http://:11434 input_cost_per_token: 0 output_cost_per_token: 0 ``` The `ollama_chat/` prefix routes to Ollama's `/api/chat`, which handles multi-turn chat and tool calling better than `ollama/`; recommended for local chat models. ## 2. Purpose Aliases: Applications Only Say "What I Want to Do" Applications don't hardcode model names; they call a handful of purpose aliases: | Purpose Alias | Purpose | |---------------|---------| | `text-default` | General text generation, summarization, translation | | `search-default` | Tasks requiring web search | | `vision-default` | Vision, OCR | | `image-default` | Text-to-image | | `tts-default` | Text-to-speech | On the blog side: when no model is specified via env var, it defaults to these aliases, and the Gateway decides the actual model and fallback order. Swapping models, reordering, adding a local host, only the Gateway changes; applications don't redeploy. ```python import os from openai import OpenAI client = OpenAI( base_url=os.environ["LITELLM_BASE_URL"], # 例如 https://litellm.markkulab.net/v1 api_key=os.environ["LITELLM_API_KEY"], # 這是 Virtual Key，不是供應商金鑰 ) resp = client.chat.completions.create( model="text-default", # 只說用途，不指定廠牌 messages=[{"role": "user", "content": "幫我把這段整理成三點摘要"}], ) print(resp.choices[0].message.content) ``` ## 3. Fault-Tolerant Failover: Auto-Degrade When Quota Exhausted ![LiteLLM Router Settings Fallbacks tab: each model's own fallback order](https://blog.markkulab.net/content/markku/posts/litellm-self-hosted-ai-api-gateway/images/litellm-router-fallbacks.webp) This is the highest-value part of the architecture. In Router Settings' Fallbacks tab, each model has its own fallback chain. Take `cloud-claude` for example: ```mermaid flowchart LR A["cloud-claude
付費・品質最好"] -->|429 / 額度不足 / 逾時| B["NVIDIA
mistral-large"] B --> C["NVIDIA
glm-flash"] C --> D["NVIDIA
step-5"] D --> E["NVIDIA
nemotron-ultra"] E --> F["NVIDIA
nemotron-super-vl"] F --> G["地端
ornith-1.5-35b"] G --> H["地端
qwen3.8-27b"] H --> I["地端
local-tts-text"] ``` The principle is **paid cloud → free cloud → local last resort**: - **Paid models first**, best quality. - **NVIDIA free models next**, they have rate limits and often return 429, but one fails, the next tries; overall availability stays high. - **Local at the end**, slower, but never fails due to quota exhaustion. Other chains follow the same logic: | Model | Fallback Order | |-------|----------------| | `text-default` | glm-flash → step-5 → nemotron-ultra → nemotron-super-vl → local ornith → local qwen3.8 → local-tts-text | | `search-default` | step-5 → mistral-large → glm-flash → nemotron-super-vl → local ornith → local qwen3.8 → local-tts-text | | `image-default` | NVIDIA flux-klein → cloud-openai-image | | `cloud-openai-image` | local ai-gen image gen | Image gen is two-stage: `image-default` failure falls back to NVIDIA's flux-klein, then OpenAI; while `cloud-openai-image` itself has local ai-gen as its final line of defense. In YAML it looks roughly like this: ```yaml router_settings: num_retries: 2 # 同一個模型先重試幾次 allowed_fails: 3 # 一分鐘內失敗幾次就先冷卻 cooldown_time: 60 # 冷卻秒數，期間直接跳過這個部署 max_fallbacks: 10 # 預設只會往下試 5 個，鏈長的話要調大 fallbacks: - cloud-claude: - cloud-nvidia-mistral-large - cloud-nvidia-glm-flash - cloud-nvidia-step-5 - cloud-nvidia-nemotron-ultra - cloud-nvidia-nemotron-super-vl - local-ollama-ornith-1.5-35b-text - local-ollama-qwen3.8-27b-text - local-tts-text ``` In the Admin UI Fallbacks tab, each row has a ▶ button to test the entire chain. Click it after reordering, far better than discovering a broken chain in production. ## 4. Usage Stats and Cost Calculation ![LiteLLM homepage dashboard: spend, request count, token usage, average cost and latency, plus model token leaderboard](https://blog.markkulab.net/content/markku/posts/litellm-self-hosted-ai-api-gateway/images/litellm-home-dashboard.webp) The LiteLLM homepage dashboard surfaces the most-watched numbers. Here are my cumulative figures: | Metric | Value | |--------|-------| | Total Spend | **$185.21** | | Total Requests | 2,135 (1,915 success, 220 failed) | | Total Tokens | 138.7M (1.8M cache reads) | | Avg Cost/Request | $0.0859 | | Avg Latency | 22.9 sec | | Success Rate | 89.7% | The Leaderboard below is more interesting, it shows each model's token share over the last 7 days: - **`mdq100/qwen3.5-coder:35b` (local Ollama) takes 77.7%**, 107.8M tokens, cost $0. - `ai-blog-grounded` 11.0%, `claude-api-text` 5.6%, `cloud-claude` 2.5%, `ai-blog-text` 1.7%. - `stock-text`, `esim-text` (application-named aliases) each under 1%; NVIDIA's models even less. In other words, **nearly 80% of tokens are consumed locally, contributing $0 to the $185.21**. Batch, long-context, code tasks go local; paid quota reserved for quality-critical work, this split is obvious at a glance on the dashboard. ### How Cost Is Calculated LiteLLM multiplies per-model unit price by input/output tokens on every call. Common cloud models have built-in pricing; missing prices must be supplied manually. The Models + Endpoints Costs column reveals three states: - **Local Ollama shows `IN $0.00 / OUT $0.00`**: I explicitly set it to 0, meaning "definitely free." - **`aigen-web-image-gpt-image-2.5-flare` shows `IN $5.00`**: Manually entered unit price, so cost accrues. - **NVIDIA models column is empty**: No unit price, so no spend generated. Fine for free models, but if you later switch to a paid tier, remember to add pricing or the dashboard will undercount. Recommend making "every model has an explicit unit price (even if 0)" a pre-deployment checklist item; only then is the Spend figure trustworthy. ## 5. Key Management: Applications Never Touch Vendor Keys Keys operate at two layers: | Layer | Location | Access | |-------|----------|--------| | Upstream vendor keys (Anthropic, NVIDIA, OpenAI...) | Gateway's LLM Credentials | Gateway only | | Virtual Keys | Each application's env vars | Each application owns its own | Each application gets its own Virtual Key, bound to allowed models, budget, and rate limits: ```bash curl -X POST "https://litellm.markkulab.net/key/generate" \ -H "Authorization: Bearer $LITELLM_MASTER_KEY" \ -H "Content-Type: application/json" \ -d '{ "key_alias": "ai-blog", "models": ["text-default", "search-default", "image-default", "tts-default"], "max_budget": 20, "budget_duration": "30d", "rpm_limit": 30, "tpm_limit": 200000, "metadata": { "project": "ai-blog" } }' ``` Benefits: - **Rotate upstream keys in one place**; applications notice nothing. - **If an application's key leaks, revoke that one key**; other applications unaffected, vendor keys never exposed. - **Budgets reset by `budget_duration` automatically**, e.g., `30d` resets every 30 days. Note the unit: `30m` is 30 minutes, not 30 months. - **`models` allowlist** restricts each application to its designated aliases; no one sneaks direct calls to the most expensive model. The `general_settings` master key and DB connection likewise come from env vars, never hardcoded in files: ```yaml general_settings: master_key: os.environ/LITELLM_MASTER_KEY database_url: os.environ/DATABASE_URL store_model_in_db: true # 讓 Admin UI 新增的模型、Fallback 存進資料庫 ``` ## 6. Wiring Up OpenCode in Practice Once the Gateway was up, the coding Agent joined the party. Using [OpenCode](https://blog.markkulab.net/post/opencode-cloud-free-models-ollama-mcp-skill-pitfall-guide) as an example, add LiteLLM as an OpenAI-compatible provider in its config: ![OpenCode model selector: LiteLLM (markkulab) listing both cloud Claude and local Qwen, Ornith models](https://blog.markkulab.net/content/markku/posts/litellm-self-hosted-ai-api-gateway/images/litellm-opencode-models.webp) ```json { "$schema": "https://opencode.ai/config.json", "provider": { "litellm": { "npm": "@ai-sdk/openai-compatible", "name": "LiteLLM (markkulab)", "options": { "baseURL": "https://litellm.markkulab.net/v1", "apiKey": "{env:LITELLM_API_KEY}" }, "models": { "cloud-claude-sonnet": { "name": "Claude Sonnet 5 (LiteLLM)" }, "cloud-claude-haiku": { "name": "Claude Haiku 4.5 (LiteLLM)" }, "local-ollama-ornith-1.5-35b-text": { "name": "Local Ornith 1.5 35B" }, "local-ollama-qwen3-coder-30b-text": { "name": "Local Qwen3 Coder 30B" }, "local-ollama-qwen3-vl-8b-text": { "name": "Local Qwen3-VL 8B", "limit": { "context": 32768, "output": 8192 } } } } } } ``` Fill `models`'s key with the `model_name` from LiteLLM; `name` is the display name in OpenCode's menu. Once configured, `/models` gains a whole "LiteLLM (markkulab)" group, exactly as in the screenshot: - **Cloud and local in one list**: Claude Opus 5, Sonnet 5, Haiku 4.5 sit alongside local Qwen3.5 Coder 35B, Ornith 1.5 35B, Qwen3-VL 8B; switching models requires zero provider config changes. - **Single Virtual Key**: The OpenCode machine sees no Anthropic or NVIDIA keys; `{env:LITELLM_API_KEY}` reads from env vars. - **Declare `limit.context` for local models**: Local context windows are often far smaller than cloud; telling OpenCode the limit lets it compress conversation history proactively instead of overflowing into errors. - **Fallback is transparent to the Agent**: Select Claude, hit 429, Gateway follows the fallback chain; OpenCode doesn't interrupt. Trade-off: later models may be weaker; if quality drops mid-task, check LiteLLM Logs to see which model actually responded. That same Virtual Key's usage appears on the dashboard too, Agent token consumption and cost tracked clearly alongside other apps. ## Minimal Runnable Deployment Example To try it yourself, LiteLLM + PostgreSQL is enough to run (example below; pin image versions for production): ```yaml # docker-compose.yml services: litellm: image: ghcr.io/berriai/litellm:main-stable command: ["--config", "/app/config.yaml", "--port", "4000"] ports: - "4000:4000" volumes: - ./config.yaml:/app/config.yaml:ro env_file: .env # LITELLM_MASTER_KEY、DATABASE_URL、各家 API key depends_on: - postgres restart: unless-stopped postgres: image: postgres:16 environment: POSTGRES_DB: litellm POSTGRES_PASSWORD: ${POSTGRES_PASSWORD} volumes: - pgdata:/var/lib/postgresql/data restart: unless-stopped volumes: pgdata: ``` Once up, open `http://localhost:4000/ui`, log in with the master key, and you're at the Admin UI from the screenshots. If you later run multiple LiteLLM instances, add Redis to share rate-limit counters and cooldown state. ## Pitfalls and Trade-offs ### 1. Fallback Chain Too Long, Tail Models Never Reached My `cloud-claude` chain has 8 models, but LiteLLM Router's default `max_fallbacks` is only 5. Without increasing it, the local last-resort models never get a turn, the config is effectively dead. Long-chain users: bump this value and test with the UI's ▶ button. ### 2. Local VRAM Is Shared Two 30B-class models launched simultaneously on the same GPU host via Ollama will fight for VRAM, causing errors that trigger the next fallback, cascading into chain failure. Mitigations: - On Ollama side, set `OLLAMA_MAX_LOADED_MODELS` and `OLLAMA_NUM_PARALLEL` to control concurrent loads and parallelism. - On LiteLLM side, set `rpm` or `max_parallel_requests` on local deployments so the Gateway doesn't flood them. ### 3. Average Latency of 22.9 Sec Needs Context This number is inflated by local models and long-form generation. Real-time interactions (chat, completion) should put fast cloud models at the front of the chain; batch, long-context, code refactoring can prefer local. One alias doesn't have to serve all scenarios; create a dedicated real-time alias if needed. ### 4. Having Fallback Doesn't Mean Zero Failures The dashboard still shows 220 failures. Some errors survive model-switching: context window exceeded, content blocked by safety policies. LiteLLM offers `context_window_fallbacks` and `content_policy_fallbacks` to handle these cases; the rest require manual Log inspection. ### 5. Pick One Source of Truth: UI or YAML Enabling `store_model_in_db` makes UI changes persist to the database, while `config.yaml` also holds a copy. Editing both invites confusion over which is live. My practice: daily tweaks in UI; YAML holds only boot-time essentials. ## Conclusion | Aspect | Before | After | |--------|--------|-------| | Model Management | Each project wired vendors independently | Single entry, 29 models, a few purpose aliases | | Keys | Vendor keys scattered across projects | Upstream keys only on Gateway; apps hold Virtual Keys | | Cost | Month-end surprise | Real-time Spend dashboard, per-model token share | | Resilience | Cloud 429 = full stop | Auto-fallback; cloud down → local catches | | Token Economics | All paid | Local consumes ~77.7% of tokens at $0 | Next steps: 1. **Set budgets on every Virtual Key**: Dashboard still says "No budget set", top priority. 2. **Add Guardrails for input filtering**: Block PII and sensitive content before hitting upstream. 3. **Integrate OpenTelemetry**: Currently only aggregate numbers; want visibility into queue vs. inference vs. fallback latency. If you share the "multi-project, multi-model, runaway cost, single-point-of-failure" pain, start with one `config.yaml` plus PostgreSQL, get the `text-default` fallback chain working, experience "cloud fails, local catches," then gradually migrate keys, budgets, and observability. ## References - [LiteLLM Official Docs: Proxy Server](https://docs.litellm.ai/docs/proxy) - [LiteLLM Official Docs: Fallbacks, Retries, and Cooldowns](https://docs.litellm.ai/docs/proxy/reliability) - [LiteLLM Official Docs: Virtual Keys](https://docs.litellm.ai/docs/proxy/virtual_keys) - [LiteLLM Official Docs: Cost Tracking](https://docs.litellm.ai/docs/proxy/cost_tracking) - [Ollama API Docs](https://github.com/ollama/ollama/blob/main/docs/api.md) - [Docker Install Ollama LLM with Hardware Acceleration Benchmark](https://blog.markkulab.net/post/build-your-ollama-ai-with-hardware-acceleration) - [NVIDIA Free AI API Hands-on and Terms](https://blog.markkulab.net/post/nvidia-free-llm-api-limits-and-terms) - [Token Monitor Hands-on and Team Edition Revamp](https://blog.markkulab.net/post/token-monitor-ai-token-dashboard) - [More Flexible Open-Source API Gateway APISIX](https://blog.markkulab.net/post/apisix-openresty-gateway) - [Tools First, Then MCP, Then Agent: Building a Fully Local AI Assistant with Hermes Agent](https://blog.markkulab.net/post/hermes-agent-local-mcp-lora) --- ## About this article and its author Originally published on [Mark Ku's Tech Notes](https://blog.markkulab.net/en/post/litellm-self-hosted-ai-api-gateway) License: [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) — when reusing or quoting, credit the author and link back to the original ### About the author **[Mark Ku](https://blog.markkulab.net/en/author/mark-ku)** — Software engineer - 10+ years as a software engineer - Built North-American e-commerce and AI SaaS subscription billing ### Free tools built by the author All of these are free to use: - [Free PDF Sign Tool](https://blog.markkulab.net/en/tools/pdf-sign): Online PDF sign tool — draw, type, or upload a signature, then drag, resize, and download. Everything runs in your browser; nothing is uploaded. - [VS Code Refactory](https://blog.markkulab.net/en/tools/refactory): Refactory is a VS Code refactoring extension: 34 actions plus a 37-rule code-smell inspection layer with a Code Health dashboard, across 18 languages, backed by 534 tests. It learns your repo's conventions: where interfaces live, where DI is registered, whether 'use client' belongs. It ranks files by git churn × complexity so you know what to fix first, and hands any smell to the Claude Code already on your machine. Free to use, and your source never leaves your computer. - [DB-Kit Database Manager](https://blog.markkulab.net/en/tools/db-kit): DB-Kit is a lightweight, cross-platform database manager built with Tauri + Rust + React. Manage MySQL, MariaDB, PostgreSQL, SQL Server, Oracle, SQLite, MongoDB, Redis, Kafka, Elasticsearch and RabbitMQ from one consistent interface: passwords encrypted in the OS keychain, SSH tunnels, full CRUD, a visual query builder, stacked multi-statement result sets, cross-connection data transfer, schema & data compare with sync SQL and schema snapshots, Excel / CSV import & export, visualized execution plans, ER diagrams, scheduled backups, SQL stress testing with p50–p99 latency percentiles, a 15-rule SQL review engine, Review & Run (AI review plus per-statement backups and an auto-generated rollback script), Kafka message browsing with monitoring & alerts, an SSH terminal with SFTP / FTP, Docker / Kubernetes, remote desktop (RDP / VNC / RustDesk), file / folder compare, stored procedure integration tests, six UI languages, a built-in AI assistant (a local CLI or any Anthropic / OpenAI-compatible API; natural-language SQL, AI review and tuning advice) and the dbk CLI. Free and open source (MIT), with installers for Windows, macOS and Linux. - [VS Code Super Mermaid](https://blog.markkulab.net/en/tools/super-mermaid): Super Mermaid is a VS Code extension for beautiful Mermaid diagrams out of the box: auto-colored live preview, mouse pan & zoom, high-res PNG / SVG export, 21 templates and multiple themes. Free and open source (MIT). - [React Super Mermaid](https://blog.markkulab.net/en/tools/react-super-mermaid): react-super-mermaid is an open-source React component library: render beautiful Mermaid diagrams with a single , with built-in colorful / sketch themes, pan & zoom, in-diagram search, and high-res SVG / PNG export. Lightweight, SSR-safe, fully typed. Free and open source (MIT). - [Jira / Confluence Super Mermaid](https://blog.markkulab.net/en/tools/jira-super-mermaid): An Atlassian Forge app: write Mermaid syntax directly inside a Jira issue or a Confluence page and get flowcharts, sequence diagrams, state machines and Gantt charts. 11 diagram types, SVG / PNG export, light and dark themes, full CJK support. Runs on Atlassian: your diagrams live in your own site and the app calls no third-party service. Free, coming soon to the Atlassian Marketplace. - [Mermaid Live Preview](https://blog.markkulab.net/en/tools/mermaid-preview): Write Mermaid in your browser, see it render instantly, and share the whole diagram as a single link. No sign-up, nothing uploaded to a server, and mermaid.live share links work as-is. - [React Intl Phone Number](https://blog.markkulab.net/en/tools/react-intl-phone-number): react-intl-phone-number is an open-source React component: framework-agnostic and antd-free, with E.164 in/out, a searchable flag / country-code dropdown, configurable validation levels (strict / mobile-strict / loose), themeable CSS, and i18n — phone logic powered by google-libphonenumber. Lightweight and fully typed. Free and open source (MIT). - [Uptime Kuma Cluster](https://blog.markkulab.net/en/tools/uptime-kuma-cluster): Turn single-node Uptime Kuma into a highly available cluster: OpenResty + Lua smart load balancing, shared MariaDB state, health checks and automatic failover, plus cluster-management REST APIs. One Docker Compose command to start. Free and open source (MIT). - [AI Podcast Cut](https://blog.markkulab.net/en/tools/ai-podcast-cut): Drop in a recording and it removes fillers and stutters, levels loudness segment by segment, and sends a second agent to review every cut. Cut points snap to word boundaries and zero crossings, every splice gets a fade, and sentence-end breaths are preserved. Desktop app for Windows, macOS and Linux. MIT licensed; the Windows installer bundles ffmpeg. - [AI Video Cut](https://blog.markkulab.net/en/tools/ai-video-cut): Pick something in your video: type it (face, license plate, phone screen, logo), click points or drag a box, or let Claude Code / Codex pick it. AI Video Cut tracks it frame by frame, then mosaics or blurs it, recolors it, pins stickers and text to it, swaps a screen or poster for your own image or video, or removes it using background that other frames actually captured. Plus sequence editing, local captions and an AI assistant. Free, open-source (MIT) Windows desktop app (macOS / Linux experimental) that runs locally, nothing uploaded. - [open-pos restaurant POS](https://blog.markkulab.net/en/tools/open-pos): One computer and one receipt printer is enough to open the shop. Your data lives on your own disk, no subscription, no lock-in, MIT licensed. Money is integer New Taiwan dollars with tax split by the statutory formula, so sales + tax always equals the total. Printing goes straight over ESC/POS on TCP 9100, with no vendor driver. Tauri + Rust + SQLite desktop app, v1.0 in development. - [.NET e-commerce API template](https://blog.markkulab.net/en/tools/dotnet-ecommerce-api): An e-commerce backend API template built on .NET 10: Controller → Service → Repository layering, Autofac convention-based registration, SqlSugar for MySQL / MariaDB and SQL Server, Redis cache and message queue, Hangfire jobs, JWT auth, and an extensible payment provider module (ECPay, Newebpay). Apache 2.0. - [Open Token Monitor team AI usage](https://blog.markkulab.net/en/tools/open-token-monitor): Collect token usage and equivalent cost from every employee's AI coding tools into a hub you host yourself: PostgreSQL, org roster, dashboards compared by company and department, and a reports API. Electron and Rust/Tauri clients upload every 30 minutes. MIT licensed, one Docker command. - [ai-security-monitor defensive security](https://blog.markkulab.net/en/tools/ai-security-monitor): Your website is live. Who checks its certificates, security settings and known vulnerabilities? A self-hosted Security Hub brings website scans and host monitoring together, deduplicates findings and compares scans after fixes. ### Daily podcasts - [Mark's Tech Insights — Daily AI News](https://blog.markkulab.net/en/category/tech-news): Daily curated AI and tech trends. Catch the latest developments via audio summaries — covering AI applications, software architecture, DevOps, and engineering practice. — RSS: https://blog.markkulab.net/feed.xml - [AI股市蝦聊](https://blog.markkulab.net/en/category/ai-stock-chat): Every trading day, an AI-analyzed take on the Taiwan stock market, delivered as a two-host conversation covering the session and the next-day outlook. — RSS: https://blog.markkulab.net/ai-stock-chat/feed.xml - [開源好物日報](https://blog.markkulab.net/en/category/open-source-weekly): A daily two-host pick of free open-source tools surfaced from real Hacker News, GitHub, and Reddit buzz — what pain they solve and the fastest way to get started. — RSS: https://blog.markkulab.net/open-source-weekly/feed.xml - [AI 運動日報](https://blog.markkulab.net/en/category/sports-weekly): Two hosts talk NBA, MLB and world sport every day — scores, records and the stories behind them, from a Taiwanese fan perspective. — RSS: https://blog.markkulab.net/sports-weekly/feed.xml - [AI 國際新聞快報](https://blog.markkulab.net/en/category/world-news): A daily two-host briefing that makes sense of the past 24 hours in world news — geopolitics, the global economy, conflict and security, disasters and climate — from a Taiwanese perspective, neutral and fully sourced. — RSS: https://blog.markkulab.net/world-news/feed.xml - [地端 AI 實驗室](https://blog.markkulab.net/en/category/local-ai-lab): Every day, a two-host look at open-weight models and LoRAs you can actually run on your own machine: what they are for, whether your GPU can handle them, and how they compare — sourced from official model cards. — RSS: https://blog.markkulab.net/local-ai-lab/feed.xml - [AI 行銷日報](https://blog.markkulab.net/en/category/ai-marketing): Every day, two hosts break down the latest in AI for marketing, advertising, SEO/GEO and content marketing: practical, never sponsored, every story sourced. — RSS: https://blog.markkulab.net/ai-marketing/feed.xml - [AI 資安日報](https://blog.markkulab.net/en/category/ai-security): Every day, two hosts cover the latest where AI meets cybersecurity: LLM and AI agent vulnerabilities, prompt injection, model supply-chain risks and AI-powered attacks, from a defense-first angle with every story sourced. — RSS: https://blog.markkulab.net/ai-security/feed.xml ### Newsletter [Subscribe to the newsletter](https://blog.markkulab.net/en/subscribe) — Be the first to know about new posts. No spam, unsubscribe anytime.