About Me: From Cloud-Native Data Planes to Intelligent Semantic Routing

I am an engineer working on AI infrastructure, LLM Gateway, and intelligent semantic routing. My path started in cloud-native networking and gateway data planes, then moved toward unified model services, MaaS platforms, model routing, and LLM traffic governance.

If I had to summarize my current focus in one sentence, it would be this: when an LLM call evolves from a simple API request into high-cost, high-latency, multi-model, multi-tenant, observable, and governed production traffic, how should the infrastructure layer understand that request and make better decisions?

Background

I have a master’s background in Network Engineering and have spent my early career inside infrastructure systems. Before moving into AI infrastructure, I worked on cloud-native networking, Kubernetes Service, Ingress, Gateway, traffic entry, and production data-plane stability. That work gave me a practical intuition for data planes, control planes, policy propagation, service discovery, load balancing, rollout, rollback, and production troubleshooting.

My work later moved toward AI infrastructure and LLM Gateway. I now work on unified model service and MaaS platform capabilities, including model API access, request governance, tenant isolation, quota control, observability, and reliability. Compared with ordinary HTTP services, LLM requests are usually more expensive, slower, longer-lived, harder to retry, and much more dependent on platform-level governance.

These two phases are not separate to me. They form one continuous line: whether the traffic is traditional microservice traffic or LLM inference traffic, production systems need a reliable, observable, and evolvable data plane. The difference is that LLM traffic adds semantics, cost, quality, safety, context, and model capability as new decision variables.

Why I Care About LLM Gateway

An LLM application often starts simply: the application calls a model API and returns the result. Once it moves into production, the system becomes more complicated very quickly.

You need to handle different model providers and inference backends. You need streaming, tool calling, timeout, retry, circuit breaking, and fallback. You need authentication, rate limiting, quota, metering, auditing, and tenant isolation. You also need to understand why a request is slow, expensive, failed, unsafe, or routed to a particular backend.

That is where LLM Gateway matters. It is not just a reverse proxy. It is the infrastructure boundary between LLM applications and model services. The application sends a request to the gateway; the gateway governs the traffic, understands the request, selects the backend, enforces policy, collects feedback, and makes system behavior observable.

Much of my current work lives on this boundary: model API access, request governance, tenant isolation, quotas and SLOs, auditing and metering, model routing, semantic cache, safety checks, and feedback loops.

How I Understand Intelligent Semantic Routing

I am also a core committer of vllm-project/semantic-router, where I focus on Mixture-of-Models, AI Gateway, and production LLM traffic governance.

In traditional gateways, routing usually depends on path, host, headers, weights, or backend health. In LLM Gateway, many of the most important routing signals live inside the request semantics and model capabilities. Is this request coding, math reasoning, extraction, summarization, or casual chat? Does it need long context? Does it contain sensitive information? Can a smaller model answer it? Can it hit a semantic cache? Should the system prefer lower cost, lower latency, higher quality, or safer behavior?

So intelligent semantic routing is not merely forwarding a request to a model. It means extracting signals from prompts, context, request parameters, tenant metadata, historical feedback, model capabilities, cost, latency, safety level, and cache state, then making a decision that can be explained, observed, replayed, and improved.

The decision may include:

• Selecting the best model or model combination for the task.
• Balancing cost, latency, quality, and reliability.
• Deciding whether safety checks, refusal, rewriting, or human review are needed.
• Checking whether semantic cache, prefix cache, or other context reuse mechanisms apply.
• Falling back or retrying when backends fail, become rate limited, or degrade in quality.
• Feeding user feedback, error rate, latency, cost, and quality signals into future routing decisions.

This is why I like semantic-router: it is not a narrow algorithm project. It is a place to move LLM application-layer decisions into the infrastructure layer. It connects model serving, gateway data planes, safety governance, online learning, caching systems, and observability.

Engineering View

I tend to think about LLM systems as complete request paths rather than isolated models.

One request may pass through authentication, quota checks, semantic understanding, safety detection, model selection, cache decisions, backend inference, streaming response, usage accounting, and feedback collection. Every layer can influence the final experience, and every layer can become the source of a production issue.

That is why I often ask:

• Should this capability live in the application, the gateway, or the inference backend?
• Can this policy be explained, observed, replayed, and rolled out gradually?
• Is it optimizing average cost, tail latency, quality, reliability, or cache hit rate?
• Does it still work under multi-tenant traffic, long-lived streaming requests, retries, and failures?
• How does it compose with Kubernetes, Gateway API, Envoy, vLLM, and SGLang?

After moving from cloud-native gateways into AI Gateway work, one idea became very clear to me: LLM infrastructure is not a completely new continent. It reuses many of the control-plane, data-plane, and observability patterns built over the past decade of cloud-native systems. The request itself has simply become more semantic.

What I Will Write About

This site is my long-term technical archive. It records how I understand AI infrastructure, LLM Gateway, intelligent semantic routing, and cloud-native data planes. It is not meant to be only a static resume. It will gradually collect writing, research notes, project summaries, and public resume material.

I expect to write mainly about:

• LLM Gateway, AI Gateway, and model-serving infrastructure.
• Semantic Router, Mixture-of-Models, and model routing strategies.
• Envoy, Gateway API, ext_proc, and cloud-native data planes.
• vLLM, SGLang, KV cache, and prefix caching as inputs to routing and scheduling.
• Agent runtime, context engineering, tool calling, and observability.

What I want this site to show is not only what I have done, but how I think about these systems and how I connect open-source research, production engineering, and implementation details.