vllm-project/semantic-router

vllm-project/semantic-router is an intelligent semantic routing project in the official vLLM ecosystem. It sits in the AI Gateway layer and targets Mixture-of-Models, model selection, request governance, safety policy, semantic cache, and production observability.

My focus is on the decision layer and production data-plane view:

• Request signal extraction from prompt, context, parameters, tenant metadata, and feedback.
• Model capability matching across task type, context length, cost, latency, quality, safety, and cache affinity.
• Gateway-level safety governance for prompt injection, jailbreak, PII, and content-safety workflows.
• Semantic cache and prefix / KV-cache-aware routing as cost, latency, and throughput levers.
• Feedback learning and evaluation loops that make routing behavior observable and improvable.

I treat semantic routing as infrastructure: the important question is not only which model is selected, but whether the decision is explainable, measurable, replayable, and safe to run in production.