---
title: "Six-Month Retrospective: API Gateway (Kong) Architecture Evaluation and Validation for a Mid-to-Large System"
description: "Over the past six months, I helped a mid-to-large system project complete its first-phase API Gateway architecture evaluation and validation. This article documents how we used Kong for routing and key authentication for a geospatial service, integrated with Prometheus, Grafana, and the ELK Stack for full-stack monitoring."
canonical_url: "https://blog.markkulab.net/en/post/kong-api-gateway-architecture-verification"
author: "Mark Ku"
author_url: "https://blog.markkulab.net/en/author/mark-ku"
site: "Mark Ku's Tech Notes"
date_published: "2025-12-14 10:00:00 +0800"
category: "DevOps"
tags: ["kong", "api gateway", "kubernetes", "prometheus", "grafana", "elk", "devops", "architecture"]
language: "en"
license: "CC BY 4.0"
license_url: "https://creativecommons.org/licenses/by/4.0/"
attribution: "when reusing or quoting, credit the author and link back to the original"
---

# Six-Month Retrospective: API Gateway (Kong) Architecture Evaluation and Validation for a Mid-to-Large System

Over the past six months, I participated in the first-phase build of a mid-to-large system project, mainly responsible for API Gateway architecture evaluation and PoC validation.

It wasn't just standing up services. The real challenge was: how do you design a single external entry point for a pile of services with different characteristics, while balancing security, observability, and high availability?
![Architecture diagram](https://blog.markkulab.net/content/markku/posts/kong-api-gateway-architecture-verification/images/./architecture.png)

## Project background and goals

The client owns a large number of internal API services and wants to **expose them externally and commercialize them** through digital transformation. The core goal: build a **self-service API rental platform** where developers can quickly apply, pay-as-you-go, and use the APIs. The first phase was to evaluate solutions and architecture before formal system development — to validate whether the requirements could be met.

## Pain points before adoption

| Aspect | Problem |
|------|------|
| **Process efficiency** | Paper-based applications, multi-layer approvals, IP binding — key issuance took ages |
| **Usage tracking** | No unified monitoring; no idea who's using how much, can't track |
| **Security & stability** | Lack of unified authentication, no rate limiting, hard to localize issues |
| **System integration** | Subsystems acted independently; high integration cost for developers |

## Value after adoption

What the API Gateway delivers:

- **Efficiency boost**: API Key application time dropped from days to minutes
- **Transparent billing**: Real-time usage stats supporting tiered plans
- **Unified entry point**: One Gateway for all APIs — lower learning curve
- **Automated ops**: Rate Limiting, ACL run automatically; centralized monitoring of anomalies

## 1. Core architecture: why Kong?

In the evaluation phase, we needed something that could handle high concurrency, support a rich plugin ecosystem, and integrate natively with Kubernetes. We ended up with **Kong API Gateway** paired with **Redis** and **PostgreSQL** as the core stack.


According to our architecture plan:
* **Traffic entry**: All external requests are handled uniformly by Kong.
![kong](https://blog.markkulab.net/content/markku/posts/kong-api-gateway-architecture-verification/images/kong.png)
* **State management**: **Redis** handles Kong's cache and Rate Limiting state.
* **Configuration storage**: PostgreSQL stores routes and plugin configs for persistence.
* **GitOps flow**: GitLab CI/CD + ArgoCD sync configs to the K8s environment.
![argocd](https://blog.markkulab.net/content/markku/posts/kong-api-gateway-architecture-verification/images/argocd.png)


### Infrastructure
For this architecture validation environment, we deployed on **Google Cloud Platform (GCP)** virtual machines, using **RKE (Rancher Kubernetes Engine)** to set up a **Kubernetes Cluster** ourselves for testing. RKE is Rancher's K8s install/management tool, focused on standardized, fast K8s cluster deployment on existing hosts. This let us flexibly simulate on-premises network restrictions and resource scheduling scenarios.

### Automated deployment strategy (Helm & Helmfile)
We used **Helmfile** to manage multiple Helm Releases (Kong, Redis, Postgres, etc.). To make the architecture more robust and meet security requirements, we adopted a **state-separation** deployment strategy:

*   **Stateless services automated**: Application-layer components (Kong Gateway, Kuma) are fully sync'd by GitOps.
*   **Stateful and sensitive data manual**: For data security and persistence, **Secrets** and **PersistentVolumes (PV/PVC)** are extracted out of Helm Charts. These resources are deployed manually by ops to ensure CI/CD pipeline mistakes don't lose data or leak keys.

### Performance
Per the official [benchmark report](https://developer.konghq.com/gateway/performance/benchmarks/), Kong Gateway can stably handle 100k+ RPS on a single 16-core machine. Even under HTTPS encryption with multi-plugin logic, it maintains single-digit ms latency — suitable for microservice architectures with strict performance demands.

## 2. Implementing API management

The first goal of API Gateway architecture validation: bring existing subsystem APIs under unified management.

### Routing and service config
We used **Kong Manager** for standardized configuration:

*   **Gateway Services**: unified definition of backend service forwarding targets, covering multiple API protocols.
*   **Working around plugin limits**: Kong's mechanism doesn't allow the same plugin type on a single route twice (e.g., you can't apply two different Rate Limiting rules on one Route). We used **Route Splitting** to solve it. For the same Backend Service, we split into multiple Routes by URL pattern, then attached targeted traffic-control plugins to each — enabling finer-grained control.

    To implement differentiated rate limiting, we adopted Route Splitting.
    **Example**: suppose the backend has an `ecommerce-service`. We can split into multiple routes by Route Prefix:

    | Route Name | Path Prefix | Rate Limit | Purpose |
    |------------|-------------|------------|------|
    | `product-list` | `/api/v1/products/*` | 1000 req/min | Product list query |
    | `order-create` | `/api/v1/orders/*` | 100 req/min | Order creation (compute-heavy) |
    | `payment-process` | `/api/v1/payments/*` | 50 req/min | Payment processing (sensitive) |

    This not only lets us set differentiated limits per API, but also **uses the Route Prefix to separately track usage by API category** — convenient for later usage analysis, billing, and resource planning.

*   **Routes and Rewrite**: precise path forwarding rules. For legacy APIs, we leveraged the **Request Transformer** plugin with regex for URI Rewrite (e.g., forwarding `~/(wms|wmts)/(?<path>api/item/1)$`), ensuring seamless old/new system integration.

> 💡 **Important takeaway from this process**:
> When validating an API Gateway architecture, **prioritize API normalization** strongly.
> If backend API URL structures are messy and naming is inconsistent, you can technically use the Gateway's Regex Rewrite to forward — but route configs become extremely complex and hard to maintain. Better to suffer briefly than chronically: clean up the API conventions first, and integration becomes far easier.

### Authentication choice: why Key Auth?
Considering the user mix and compatibility, we weighed JWT, HMAC, and API Key — and went with "Header-based Key Auth."

- Not JWT: requires expiry/refresh handling and client state; integration and ops cost is high.
- Not HMAC: must implement signing/timestamps and crypto; cross-language SDK maintenance is high.
- Chose Key Auth: low barrier (key carried in Header directly), high compatibility (natively supported by common tools and frameworks), and stackable security (IP whitelist, ACL, Rate Limiting).

### User-call flow

Here's the actual Gateway operation steps:

1. Initiate request: user calls the platform with an API Key.
2. Authentication and authorization (Kong): validate the Key's validity and permissions.
3. Rate limit: check whether the plan quota is exceeded.
4. Service proxy: forward the request to the backend subsystem.
5. Logging and response: async log write, return data.

### Tiered Rate Limiting
To allocate compute resources reasonably, our traffic-shaping strategy:

1.  **Rate Limit**: separate "basic" and "high-frequency" channels for general APIs vs heavy-load services, fitting different use cases.
2.  **Bandwidth Limit**: for large file transfers, since Kong (even the paid version) doesn't natively support fine-grained "data-size-based" control, I additionally **wrote a custom plugin** to implement bandwidth limiting.

To support multi-node horizontal scaling, we switched the Rate Limiting policy to **Redis Mode**. All counters and state are stored uniformly in **Redis**, ensuring data consistency across nodes and avoiding errors from per-node counting.

![custom-plugin](https://blog.markkulab.net/content/markku/posts/kong-api-gateway-architecture-verification/images/custom-plugin-1.png)
![custom-plugin](https://blog.markkulab.net/content/markku/posts/kong-api-gateway-architecture-verification/images/custom-plugin-2.png)

### Custom Bandwidth Limiting
Beyond basic Rate Limiting, for large-file or high-bandwidth services we developed a custom Kong Lua plugin:
* **Multi-dimension bandwidth control**: supports six time dimensions from "second" to "year," and auto-converts MB to Bytes for precise computation.
* **High-reliability architecture**: implements **Redis cache optimization** and **fault tolerance**. When Redis fails, the plugin auto-degrades to a local counting strategy, preventing single point of failure from impacting API availability.

### Fine-grained access control (ACL & Consumer Groups)
For paid and sensitive data, we used the **ACL (Access Control Lists)** plugin to build a strict permission model:
* Created Consumers like `backend_service`, `mobile_app`.
* Added Consumers to specific groups (e.g., `business-tier-gold`).

### Automated onboarding (Swagger-based URL Sync)

After solving core routing and permission config, the next challenge: how to onboard many subsystems quickly and in a standardized way. To avoid manual errors from reading Swagger docs and keying configs by hand, we designed a **URL Sync** mechanism.

The core: "spec-as-source-of-truth, in real time." Instead of uploading files manually, the system reads the Swagger / OpenAPI Specification (OAS) URL the subsystem provides.

The automated onboarding flow has four key steps:

- Spec read (Swagger Sync): admin only needs to set the subsystem's Swagger/OAS URL, and the system auto-fetches the latest API spec.
- Kong conversion (Spec to Config): system parses the Swagger and auto-creates corresponding Routes and Plugins in Kong, ensuring Gateway routing rules match Swagger definitions exactly.
- Auto-monitoring: at onboarding time, the system auto-triggers Uptime Kuma to set up a Health Check, achieving "monitor as soon as the service is live."
- External publishing: after the above config, the service is officially live and proxied through the Gateway.

## 3. Building comprehensive observability

Solid monitoring is the bedrock of stable Gateway operation. This half-year we invested heavily in integrating the **ELK Stack** and **Prometheus** ecosystems, achieving full observability from "log query" to "metric dashboards."

### Log analysis (ELK Stack & Custom Lua)

Kong's logging plugin already includes a `client_ip` field, but to get more advanced info (like the original IP after multiple proxy layers, the `X-Forwarded-For` Header, etc.), you can use the **UDP Log** plugin's `custom_fields_by_lua` to inject custom fields:

```lua
custom_fields_by_lua = {
  remote_addr = "return kong.client.get_forwarded_ip()",
  real_ip = "return ngx.var.realip_remote_addr",
  x_forwarded_for = "return kong.request.get_header('x-forwarded-for')"
}
```

This config gives us more complete source-IP info for precise traffic-source and behavior analysis — providing key data for business decisions. We also deployed Filebeat in **Sidecar mode**, reading access/error logs from `/var/log/kong` and forwarding them to Elasticsearch, building `kong-logs-YYYY.MM.DD` indexes.
![log-server](https://blog.markkulab.net/content/markku/posts/kong-api-gateway-architecture-verification/images/log-server.png)


### Metrics monitoring (Prometheus & Grafana)
With Kong's Prometheus plugin, we collected detailed traffic metrics and built dedicated dashboards in **Grafana**:
![prometheus](https://blog.markkulab.net/content/markku/posts/kong-api-gateway-architecture-verification/images/prometheus.png)
* **Kong API Gateway Dashboard**: real-time display of Request Rate, Latency, Bandwidth, etc.
![grafana-kong-monitor](https://blog.markkulab.net/content/markku/posts/kong-api-gateway-architecture-verification/images/grafana-kong-monitor.png)
* **Kubernetes Node hardware monitoring**: via Node Exporter integration, view each K8s node's hardware status (CPU, memory, Disk I/O, Network Traffic) directly in Grafana — easy to grasp infrastructure health.
![grafana-hardware-monitor](https://blog.markkulab.net/content/markku/posts/kong-api-gateway-architecture-verification/images/grafana-hardware-monitor.png)


### Service availability monitoring (Uptime Kuma Clustering)

**Uptime Kuma** is an open-source self-hosted monitoring tool similar to Uptime Robot, but you can deploy it entirely on your own server. It provides an intuitive Web UI, supports multiple monitor types (HTTP, TCP, Ping, DNS, etc.), and can send alerts via Telegram, Slack, Email. For teams that need to monitor lots of internal services without depending on external SaaS, it's a great choice.

To overcome native Uptime Kuma's performance bottleneck under heavy monitoring load (around 800 APIs), we restructured the architecture:
* **Automated extension (RESTful API Extension)**: native Uptime Kuma lacks an external API, so to achieve "monitor as soon as service is live" automation, we extended a RESTful API ourselves so that future API onboarding triggers monitoring automatically.
* **Database extracted**: replaced the default SQLite with **MariaDB**, deployed independently as `kuma-mariadb` service, supporting higher concurrent reads/writes.
* **Clustering**: multiple Uptime Kuma instances connect to the same database for load distribution.

I wrote up the implementation details in another article: [Building a Uptime Kuma Cluster System with Vibe Coding: from Single-Node to High-Availability Monitoring Platform](https://blog.markkulab.net/implement-uptime-kuma-cluster-vibe-coding/).
![Uptime Kuma dashboard displaying API endpoint health and event logs](https://blog.markkulab.net/content/markku/posts/kong-api-gateway-architecture-verification/images/custom-uptime-kuma.png)


## 4. Setting up alerting
![alert-manager](https://blog.markkulab.net/content/markku/posts/kong-api-gateway-architecture-verification/images/alert-manager-2.png)
![alert-manager](https://blog.markkulab.net/content/markku/posts/kong-api-gateway-architecture-verification/images/alert-manager-1.png)

The last mile of monitoring is "alerting." We defined **20 alert rules** in Prometheus and **Alert Manager**, split into Critical, Warning, and Info severities.

Here are several key Critical alert rules (PromQL):

### 1. Service completely unavailable (KongServiceDown)
Triggers when the Kong namespace has no Up Pods:
```promql
sum(up{job="kong-metrics", namespace="kong"}) == 0
```

### 2. High error rate (KongHighErrorRate)
Triggers when 5xx error rate exceeds 5% — usually indicating backend service issues:
```promql
(sum(rate(kong_http_requests_total{code=~"5.."}[5m])) by (instance)
 /
 sum(rate(kong_http_requests_total[5m])) by (instance)
) > 0.05
```

### 3. High latency (KongHighLatency)
Triggers when P95 request latency exceeds 2 seconds (Warning level):
```promql
histogram_quantile(0.95, 
  sum(rate(kong_latency_bucket[5m])) by (le, instance)
) > 2000
```

### 4. System resource alerts
* **Disk space**: `NodeDiskSpaceCritical` (free space < 5%)
* **Memory**: `NodeMemoryCritical` (usage > 95%)

## 5. Validation and outcomes

The final outcome of these six months: successfully helping the project **pass the first-phase system acceptance**.

During acceptance, we used the open-source API testing tool **Hoppscotch** (formerly Postwoman) for intensive functional validation against multiple critical services. Test results showed the API Gateway not only effectively intercepts unauthorized malicious requests, but its fine-grained traffic-control mechanism also kept services stable under high system load — meeting all the performance metrics specified in the contract.

## 6. Reflections
Kong OSS itself doesn't provide group rules, advanced traffic control, or route-level-only control mechanisms, and there's no native concept of API subscription. So if you want to use Kong to implement API subscription tiers or more complex group permissions and traffic rules, you typically need a lot of customization. Otherwise, the design easily ends up not matching needs and you have to bend to the existing architecture and logic.

---

## About this article and its author

Originally published on [Mark Ku's Tech Notes](https://blog.markkulab.net/en/post/kong-api-gateway-architecture-verification)

License: [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) — when reusing or quoting, credit the author and link back to the original

### About the author

**[Mark Ku](https://blog.markkulab.net/en/author/mark-ku)** — Software Solution Provider

- 10+ years senior software engineer, now an AI Builder
- Focused on large-platform architecture — North-American e-commerce, AI SaaS subscription billing
- Combining AI Agents and automation to build evolvable product foundations

### Free tools built by the author

All of these are free to use:

- [Free PDF Sign Tool](https://blog.markkulab.net/en/tools/pdf-sign): Online PDF sign tool — draw, type, or upload a signature, then drag, resize, and download. Everything runs in your browser; nothing is uploaded.
- [VS Code Refactory](https://blog.markkulab.net/en/tools/refactory): Refactory is a VS Code refactoring extension: 34 actions plus a 37-rule code-smell inspection layer with a Code Health dashboard, across 18 languages, backed by 534 tests. It learns your repo's conventions: where interfaces live, where DI is registered, whether 'use client' belongs. It ranks files by git churn × complexity so you know what to fix first, and hands any smell to the Claude Code already on your machine. Free to use, and your source never leaves your computer.
- [DB-Kit Database Manager](https://blog.markkulab.net/en/tools/db-kit): DB-Kit is a lightweight, cross-platform database manager built with Tauri + Rust + React. Manage MySQL, MariaDB, PostgreSQL, SQL Server, Oracle, SQLite, MongoDB, Redis, Kafka, Elasticsearch and RabbitMQ from one consistent interface: passwords encrypted in the OS keychain, SSH tunnels, full CRUD, a visual query builder, stacked multi-statement result sets, cross-connection data transfer and compare/sync, Excel / CSV import & export, visualized execution plans, ER diagrams, scheduled backups, SQL stress testing with p50–p99 latency percentiles, a 15-rule SQL review engine, Kafka message browsing with monitoring & alerts, a bilingual UI (Traditional Chinese / English), a built-in AI assistant (natural-language SQL, AI review and tuning advice) and the dbk CLI. Free and open source (MIT), with installers for Windows, macOS and Linux.
- [VS Code Super Mermaid](https://blog.markkulab.net/en/tools/super-mermaid): Super Mermaid is a VS Code extension for beautiful Mermaid diagrams out of the box: auto-colored live preview, mouse pan & zoom, high-res PNG / SVG export, 21 templates and multiple themes. Free and open source (MIT).
- [React Super Mermaid](https://blog.markkulab.net/en/tools/react-super-mermaid): react-super-mermaid is an open-source React component library: render beautiful Mermaid diagrams with a single <MermaidViewer>, with built-in colorful / sketch themes, pan & zoom, in-diagram search, and high-res SVG / PNG export. Lightweight, SSR-safe, fully typed. Free and open source (MIT).
- [Jira / Confluence Super Mermaid](https://blog.markkulab.net/en/tools/jira-super-mermaid): An Atlassian Forge app: write Mermaid syntax directly inside a Jira issue or a Confluence page and get flowcharts, sequence diagrams, state machines and Gantt charts. 11 diagram types, SVG / PNG export, light and dark themes, full CJK support. Runs on Atlassian: your diagrams live in your own site and the app calls no third-party service. Free, coming soon to the Atlassian Marketplace.
- [Mermaid Live Preview](https://blog.markkulab.net/en/tools/mermaid-preview): Write Mermaid in your browser, see it render instantly, and share the whole diagram as a single link. No sign-up, nothing uploaded to a server, and mermaid.live share links work as-is.
- [React Intl Phone Number](https://blog.markkulab.net/en/tools/react-intl-phone-number): react-intl-phone-number is an open-source React component: framework-agnostic and antd-free, with E.164 in/out, a searchable flag / country-code dropdown, configurable validation levels (strict / mobile-strict / loose), themeable CSS, and i18n — phone logic powered by google-libphonenumber. Lightweight and fully typed. Free and open source (MIT).
- [Uptime Kuma Cluster](https://blog.markkulab.net/en/tools/uptime-kuma-cluster): Turn single-node Uptime Kuma into a highly available cluster: OpenResty + Lua smart load balancing, shared MariaDB state, health checks and automatic failover, plus cluster-management REST APIs. One Docker Compose command to start. Free and open source (MIT).
- [Special Education](https://blog.markkulab.net/en/education): Learning materials crafted for special education students

### Daily podcasts

- [Mark's Tech Insights — Daily AI News](https://blog.markkulab.net/en/category/tech-news): Daily curated AI and tech trends. Catch the latest developments via audio summaries — covering AI applications, software architecture, DevOps, and engineering practice. — RSS: https://blog.markkulab.net/feed.xml
- [AI股市蝦聊](https://blog.markkulab.net/en/category/ai-stock-chat): Every trading day, an AI-analyzed take on the Taiwan stock market, delivered as a two-host conversation covering the session and the next-day outlook. — RSS: https://blog.markkulab.net/ai-stock-chat/feed.xml
- [開源好物週報](https://blog.markkulab.net/en/category/open-source-weekly): A weekly two-host pick of free open-source tools surfaced from real Hacker News, GitHub, and Reddit buzz — what pain they solve and the fastest way to get started. — RSS: https://blog.markkulab.net/open-source-weekly/feed.xml

### Newsletter

[Subscribe to the newsletter](https://blog.markkulab.net/en/subscribe) — Be the first to know about new posts. No spam, unsubscribe anytime.