Real-Time TTS Inference on Amazon EKS with KServe + vLLM-Omni

Run text-to-speech on your EKS cluster using the OmniVoice (600+ languages) model with KServe and the vLLM-Omni serving engine — a real-time TTS stack with streaming PCM delivery over WebSocket, below 0.1 RTF. A warm-floor + Spot-burst GPU strategy keeps a guaranteed baseline online while bursting cheaply onto diversified Spot capacity. Full control over latency, cost, and model behavior instead of per-character API metering.

What you'll build

A production-grade TTS inference stack on Amazon EKS. The stack is made up of:

• a FastAPI WebSocket gateway that authenticates clients, sends V2 infer requests to a shared predictor pool spanning both tiers, and streams PCM audio back to the browser
• an always-on floor InferenceService (tts-floor — on-demand g6e.xlarge / L40S, 25 fixed replicas, sized to 3k rpm peak)
• an elastic burst InferenceService (tts-burst — Spot g6e, HPA-scaled 1→16)
• a browser inference dashboard app with 600+ language support, voice-design control, streaming playback, and live inference stats
• a Karpenter node strategy: guaranteed on-demand GPU floor pool + diversified Spot burst pool + Bottlerocket CPU pool for the gateway
• Prometheus + Grafana with DCGM GPU metrics, gateway TTFA/RTF histograms, and a pre-built dashboard
• an S3 model-weights bucket loaded into each pod at startup by KServe's storage initializer — no StatefulSet, no PVC, no per-instance state

Features

• 🔥 Warm floor + Spot burst — A fixed on-demand baseline absorbs steady traffic with zero cold starts; the burst tier scales 1→16 on predictor_backlog_streams (demand minus ready floor capacity) across g6e Spot in 2 sizes × 4 AZs.
• ⏱️ Fast GPU cold starts — GPU nodes run containerd and kubelet ephemeral on local NVMe instance store, so the ~9 GB serving image pulls at multi-GB/s: ~5–6 min to pod-ready, measured (node boot + weights pull + image pull + engine start; burst pods above the 3k rpm floor only — the warm floor keeps serving throughout).
• 🚀 Parallel image pull on burst nodes — The EKS AL2023 AMI's built-in SOCI parallel pull (FastImagePull feature gate, one line in the GPU EC2NodeClass) cuts the 9 GB pull from ~3 min to ~1m55s and total cold start to ~4m20s, measured.
• 🗣️ Streaming delivery, sentence-level pipelining — The engine synthesizes each utterance whole (OmniVoice is non-streaming); the stack streams delivery: text splits into sentences that synthesize while earlier audio plays, making TTFA ≈ first-sentence synthesis time (~430 ms p50 warm; RTF p50 ~0.07).
• 💾 Stateless model weights — KServe's storage initializer pulls weights from S3 into /mnt/models at pod start (~30 s); no Hugging Face dependency at runtime, no PVCs to manage.
• 🚦 Admission control, on by default — A bounded-concurrency gate with a time-limited FIFO queue returns clean "busy" signals instead of unbounded hangs; sized by WARM_CAPACITY (ready pods × streams/GPU), disable via the ADMISSION_ENABLED kill switch.
• 📊 Metrics that drive operations — TTFA/RTF histograms, DCGM GPU telemetry, live /stats JSON, and a pre-built Grafana dashboard; the burst HPA scales on the same series, and engine-agnostic names survive engine swaps.
• 🔒 Locked-down by default — Default-deny NetworkPolicies with surgical allows, IAM via Pod Identity (no node-role fallback), TLS-terminated NLB pinned to allowlisted source IPs, non-root images, Bottlerocket on the internet-facing tier.
• 🛠️ In-cluster image builds — The serving image builds with Kaniko on a CPU node straight from your local checkout (kubectl cp, no git push, no local Docker daemon for the 9 GB image).

Project structure

sample-tts-on-eks-kserve/
├── app/
│   ├── gateway/          # FastAPI WebSocket relay + Dockerfile + .dockerignore
│   ├── client/           # Browser inference dashboard/demo client (single HTML file; baked into the gateway image)
│   └── vllm-omni-adapter/# KServe V2 ⇄ /v1/audio/speech adapter + Dockerfile (FROM AWS vLLM-Omni DLC)
├── k8s/
│   ├── namespace/        # Namespace (tts) + ServiceAccounts
│   ├── karpenter/        # gpu-floor + gpu-burst + cpu NodePools (GPU nodes on NVMe instance store)
│   ├── storage/          # gp3 StorageClass (Kaniko build PVC only) — predictors are stateless (S3 weights into NVMe-backed ephemeral, no PVCs)
│   ├── kserve/           # vllm-omni-omnivoice runtime + floor/burst InferenceServices + burst HPA + shared predictor pool Service
│   ├── build/            # Kaniko in-cluster serving-image build (PVC + Job)
│   ├── gateway/          # Deployment/ConfigMap/Service, NLB Service, HPA, SecretProviderClass
│   ├── network-policy/   # Default-deny + surgical allow rules
│   └── observability/    # DCGM exporter, ServiceMonitors, Grafana dashboard, burst rules
├── terraform/
│   ├── cluster/          # VPC, EKS, Karpenter IAM, ECR, Secrets Manager, S3 model weights
│   ├── kserve/           # cert-manager + KServe v0.17.1 (separate workspace for CRD ordering)
│   └── addons/           # Karpenter, AWS LBC, kube-prometheus-stack, Secrets Store CSI, NVIDIA plugin, Prometheus Adapter
└── scripts/              # build-serving-image.sh, configure-nlb-tls.sh, smoke-test.sh, generate-contact-json.sh

Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                         BROWSER (HTTPS/WSS)                         │
│         Inference dashboard · voice design · live inference stats   │
└─────────────────────────────────────────────────────────────────────┘
                                  │
                                  ▼
┌─────────────────────────────────────────────────────────────────────┐
│                NLB (TLS via ACM · source-IP allowlist)              │
│      internal by default — run configure-nlb-tls.sh to expose       │
└─────────────────────────────────────────────────────────────────────┘
                                  │
                                  ▼
┌─────────────────────────────────────────────────────────────────────┐
│                  GATEWAY (FastAPI · Bottlerocket CPU)               │
│        WS /ws/synthesize · GET / · /health · /metrics · /stats      │
│      sentence streaming · admission gate · per-utterance conns      │
└─────────────────────────────────────────────────────────────────────┘
                                  │
                                  ▼
┌─────────────────────────────────────────────────────────────────────┐
│       omnivoice-tts-lb (shared pool Service, floor+burst pods)      │
│        burst capacity added by HPA on predictor_backlog_streams     │
└─────────────────────────────────────────────────────────────────────┘
               │                                    │
               ▼                                    ▼
┌───────────────────────────────┐   ┌───────────────────────────────┐
│    tts-floor (KServe ISVC)    │   │    tts-burst (KServe ISVC)    │
│  on-demand g6e.xlarge (L40S)  │   │  Spot g6e · HPA 1→16          │
│  25 fixed replicas · warm     │   │  overflow only                │
│  vLLM-Omni :8000 + V2 :8080   │   │  vLLM-Omni :8000 + V2 :8080   │
└───────────────────────────────┘   └───────────────────────────────┘
        │              │                     │
        ▼              ▼                     ▼
┌─────────────┐ ┌─────────────┐ ┌─────────────────────────────────┐
│ S3 weights  │ │ NVMe local  │ │   Prometheus + Grafana + DCGM   │
│ storage     │ │ instance    │ │   TTFA/RTF histograms · GPU     │
│ initializer │ │ store       │ │   metrics · burst HPA signal    │
└─────────────┘ └─────────────┘ └─────────────────────────────────┘

1. Quickstart

Follow the Quickstart one-time setup to deploy the full TTS inference stack and run smoke tests.

2. Access the app

Follow the Access the app steps to retrieve an API key, connect locally, or over internet-facing HTTPS to run the TTS inference dashboard app.

3. (Optional) Access observability dashboards

Follow the (Optional) Access observability dashboards steps to load the pre-built Grafana dashboard and query Prometheus directly.

Cleanup

1. Remove workloads (lets Karpenter drain GPU nodes):

   kubectl delete -f k8s/observability/ -f k8s/network-policy/ \
     -f k8s/gateway/ -f k8s/kserve/ --ignore-not-found
   # If the optional Grafana step was run:
   kubectl delete configmap grafana-dashboard-kserve -n monitoring --ignore-not-found

2. Destroy KServe and cert-manager:

   terraform -chdir=terraform/kserve destroy

3. Destroy addons:

   terraform -chdir=terraform/addons destroy

4. Destroy AWS infrastructure:

   terraform -chdir=terraform/cluster destroy

Security

See CONTRIBUTING for more information.

License

This library is licensed under the MIT-0 License. See the LICENSE file.