Production LLM serving across multiple inference stacks for large-scale multi-turn, long-context chat. APC-aware KV-cache lookup in LMCache (16–25% lower p50 TTFT), profiler-driven attention-kernel tuning on AMD GPUs (4.3× faster prefill kernel, 42–48% lower P99 TTFT at equal load), and a production traffic replay tool that replays requests in real arrival time with session-aware multi-turn replay, matches production metrics within the pod's own variance, and underpins every later optimization.

Supervised by Prof. Udit Gupta, mentored by Yueying (Lisa) Li. Designed a measurement-driven placement planner for multi-stage speech serving that sets stage placement, memory split, replica count, and GPU-sharing mode from two short probes per candidate plan (+40% on Qwen3-Omni 30B, +60% on Qwen3-TTS 1.7B over shipped configurations). Designed a prediction-free, tail-aware scheduler (γ-Boost) with KV-cache-aware preemption that cut tail TTFT/TTLT by 37–60% on Azure production traces (ICML 2026). Developed PTA (Prefill → Think → Answer) disaggregated serving with per-stage DVFS, KV-cache, and GPU allocation under power caps, cutting P90 time-between-answer-tokens by 49–71% versus prefill–decode disaggregation, and a Flex service tier with a KKT-based online solver that turns bounded latency slack into per-stage settings within milliseconds (78.3% vs. 47.6% goodput at a 30% power cap).

Request-level admission control with stepwise latency profiling and SLA-aware priority scheduling, holding 100% goodput under 2–10× QPS bursts. CUDA attention-kernel optimization for GQA and DeepSeek FlashMLA, profiled with Nsight Compute, for about 20% and 15% throughput gains.

Supervised by Prof. Qizhen Zhang at the Far Data Lab. Co-designed dpBento to benchmark heterogeneous DPUs for data processing offload. Studied RocksDB internals and explored DPU offload for compute-storage disaggregation.