Back to Projects

LLM Gateway / AI Gateway Research

#AI Gateway #Gateway API #Envoy

This research thread studies how AI Gateway systems evolve from protocol forwarding layers into intelligent decision layers for production LLM applications.

Key topics:

  • Model routing, token-aware rate limiting, semantic cache, safety detection, multi-tenant isolation, and observability.
  • Envoy ext_proc, Gateway API, InferencePool, InferenceModel, and the resource models that connect gateways with inference backends.
  • How vLLM, SGLang, PagedAttention, continuous batching, RadixAttention, prefix caching, and KV cache reuse influence gateway-level routing decisions.

The goal is to connect cloud-native traffic governance with model-serving realities: high-cost requests, streaming responses, long contexts, vendor and self-hosted backends, and feedback-driven routing policies.