Epic: SLO-aware LLM inference service system on SAGE

[kms12425-ctrl]

Background

AI inference clusters are expensive, but GPU utilization in production inference often stays around 20%-40%. The main reasons are dynamic peak provisioning, small models running on oversized GPU/NPU devices, and static one-model-per-GPU allocation that prevents memory sharing across services.

Scope

• Define request-level SLO and multi-model workload contracts.
• Design a GPU/NPU memory abstraction for model weights and KV cache with on-demand allocation, release, and reuse.
• Develop multi-model SLO-aware scheduling, spillover, and admission control.
• Evaluate the integrated system on heterogeneous clusters with cost and utilization targets.

Deliverables

• Integrated prototype system (SAGE + vLLM-HUST).
• Experiment reports and reproducibility artifacts.
• Final report and final acceptance package.
• One accepted CCF-A paper and two patent application packages.

Acceptance Criteria

• GPU/NPU VRAM utilization >90% in target scenarios.
• End-to-end inference cost reduced by >70%.
• P95 latency reduced by >20% vs FIFO/load-aware/static-allocation baselines.
• SLO violation rate reduced by >30%.
• Support multi-model, multi-instance, heterogeneous-node scheduling and spillover.

Dependencies

None

Notes

SAGE is the control plane; vLLM-HUST is the execution plane.

Tracked Issues

• Survey, architecture, and workload definition #1505 Survey, architecture, and workload definition
• Request schema and SLO metadata contract #1507 Request schema and SLO metadata contract
• vLLM-HUST runtime metrics collection and reporting adapter #1508 vLLM-HUST runtime metrics collection and reporting adapter
• Backend container registration and runtime state model in SAGE #1509 Backend container registration and runtime state model in SAGE
• Online request profiling and risk estimation #1510 Online request profiling and risk estimation
• SLO-aware scheduler, spillover, and admission control #1511 SLO-aware scheduler, spillover, and admission control
• Experiment harness, baselines, and reproducibility artifacts #1512 Experiment harness, baselines, and reproducibility artifacts
• Midterm/final acceptance package and benchmark reporting #1513 Midterm/final acceptance package and benchmark reporting