Pooja Dhavale Enterprises building on AI don’t always have the luxury of sending their data to an external model provider. Data residency laws, compliance rules, and cost at scale often push platform teams toward running models inside their own infrastructure.
The good news: Envoy AI Gateway treats a self-hosted Small Language Model (SLM) exactly like any external provider — same configuration, same controls, same routing and authentication model.
In this post, we walk through how we proved that by running TinyLlama via Ollama inside Azure Kubernetes Service (AKS), and validated the two pillars of provider support: API schema support and upstream auth support.
The Problem: When You Can’t Use an External Provider
Many AI applications start by calling a hosted API. But for a large class of workloads, that isn’t an option:
- Data residency laws: Regulations require data to stay within specific geographic boundaries. External providers are simply off the table.
- Compliance requirements: Healthcare, finance, and defence sectors often prohibit sending sensitive data to third-party APIs.
- Cost at scale: External provider costs grow linearly with usage. At high volume, self-hosting becomes significantly more cost effective.
- Full model ownership: Platform teams want control over model versions, fine-tuning, and updates without vendor lock-in.
The key insight is that Envoy AI Gateway treats self-hosted models identically to external providers — the same config, the same controls. Whether the model is OpenAI behind an API key or TinyLlama running in your own cluster, the gateway configuration looks the same.
Architecture: TinyLlama + Ollama + Envoy AI Gateway inside AKS
The setup runs entirely inside a Kubernetes cluster, with no external calls and no data leaving the cluster.
The request path is:
curl (your machine) → Envoy AI Gateway → AIGatewayRoute (slm-route) → AIServiceBackend (tinyllama) → Ollama + TinyLlama (inside AKS)
TinyLlama is a compact 1.1B-parameter model — the Ollama build is only around 638MB — which makes it ideal for demonstrating self-hosted inference on modest hardware. Ollama serves it with an OpenAI-compatible API, and the gateway sits in front, handling routing and credential injection.
Two Pillars of Provider Support
Supporting a new provider in Envoy AI Gateway comes down to two things: does the gateway understand the provider’s API format, and can it authenticate to that provider securely? We proved both.
Pillar 1 — API Schema Support
What it means: The gateway understands the provider’s API format and routes requests correctly.
How we proved it: We sent an OpenAI-format request to TinyLlama via Ollama through the gateway. The schema config is simply:
schema:
name: OpenAI
version: v1The response came back in full OpenAI format:
{ "object": "chat.completion",
"model": "tinyllama",
"choices": [...],
"usage": { "total_tokens": 156 } }Result: A full OpenAI-format response, with token usage tracked automatically — the same behavior you’d get from any hosted provider.
Pillar 2 — Auth Support
What it means: The gateway injects credentials into upstream requests automatically. The application never holds the key.
How we proved it: An echo server confirmed the Authorization header was injected by the gateway — the app itself sent none. The policy config uses BackendSecurityPolicy:
type: APIKey
secretRef:
name: ollama-api-keyThe echo server showed the injected header:
"headers": {
"authorization": "Bearer my-secret-ollama-key-123"
}Result: The key was injected from a Kubernetes Secret, the app sent zero credentials, and Ollama accepted and responded. This is the same upstream-authentication mechanism the gateway uses for AWS Bedrock, Azure OpenAI, and every other provider.
Sample Demo Results
Setup: AKS (1 node, 2 vCPU, 8GB RAM) • TinyLlama 1.1B via Ollama • API key secured • no GPU • no external calls.
Pillar 1 — Schema proof. A real inference response, in OpenAI format, with token accounting:
{"id":"chatcmpl-618",
"object":"chat.completion",
"model":"tinyllama",
"choices":[{"message":{"role":"assistant","content":"..."}}],
"usage":{"prompt_tokens":43,
"completion_tokens":113,
"total_tokens":156}}Pillar 2 — Auth proof (echo server). The Authorization header, injected by the gateway from a K8s Secret:
"headers": {
"content-type": "application/json",
"x-tenant-id": "alice",
"authorization": "Bearer my-secret-ollama-key-123"
← injected by gateway
}What this demonstrates:
- OpenAI-format request produced an OpenAI-format response, with token usage tracked.
- The Authorization header was injected from a K8s Secret — the app sent zero credentials.
- A 637MB model ran CPU-only, with no GPU, no external API, and no data leaving the cluster.
- The same YAML config as any external provider — three resources, no code changes.
Why This Matters for Platform Teams
The real win here is uniformity. A self-hosted SLM and a hosted frontier model are configured, secured, and observed the same way through Envoy AI Gateway. That means:
- Sensitive workloads can stay entirely inside your own cluster, satisfying data-residency and compliance needs.
- Credentials are centralized and injected by the gateway, never embedded in application code.
- You can mix self-hosted and external providers behind one unified, OpenAI-compatible API.
- Moving from a hosted model to a self-hosted one — or the reverse — requires no application changes.
Final Thoughts
Self-hosted Small Language Models are no longer a special case. With Envoy AI Gateway, a model like TinyLlama running on Ollama inside AKS is just another supported provider — routed by API schema and secured by upstream authentication, exactly like OpenAI, AWS Bedrock, or Azure OpenAI.
For teams building on Envoy AI Gateway — whether connecting to external providers or running their own models in-cluster — IMESH provides enterprise-grade support, architecture guidance, troubleshooting, and best-practice recommendations to help you deploy and operate AI gateways with confidence.
Need help running self-hosted models behind Envoy AI Gateway? Connect with the experts.
