{"notes":[{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2404.02015v2"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"duan|muxserve_flexible_multiplexing_for_efficient_multiple_llm_serving"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Jiangfei_Duan:","https://dblp.org/search/pid/api?q=author:Runyu_Lu:","https://dblp.org/search/pid/api?q=author:Haojie_Duanmu:","https://dblp.org/search/pid/api?q=author:Xiuhong_Li:","~Xingcheng_Zhang3","~Dahua_Lin1","https://dblp.org/search/pid/api?q=author:Ion_Stoica:","~Hao_Zhang2"]},"html":{"value":"https://doi.org/10.48550/arXiv.2404.02015"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2404-02015,\n  publtype={informal},\n  author={Jiangfei Duan and Runyu Lu and Haojie Duanmu and Xiuhong Li and Xingcheng Zhang and Dahua Lin and Ion Stoica and Hao Zhang},\n  title={MuxServe: Flexible Multiplexing for Efficient Multiple LLM Serving},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2404.02015},\n  url={https://doi.org/10.48550/arXiv.2404.02015}\n}\n"},"abstract":{"value":"Large language models (LLMs) have demonstrated remarkable performance, and organizations are racing to serve LLMs of varying sizes as endpoints for use-cases like chat, programming and search. However, efficiently serving multiple LLMs poses significant challenges for existing approaches due to varying popularity of LLMs. In the paper, we present MuxServe, a flexible spatial-temporal multiplexing system for efficient multiple LLM serving. The key insight behind is to colocate LLMs considering their popularity to multiplex memory resources, and leverage the characteristics of prefill and decoding phases to separate and flexibly colocate them to multiplex computation resources. MuxServe formally formulates the multiplexing problem, and proposes a novel placement algorithm and adaptive batch scheduling strategy to identify optimal colocations and maximize utilization. MuxServe designs a unified resource manager to enable flexible and efficient multiplexing. Evaluation results show that MuxServe can achieves up to $1.8\\times$ higher throughput or processes $2.9\\times$ more requests within $99\\%$ SLO attainment. The code is available at: \\url{https://github.com/hao-ai-lab/MuxServe}."},"title":{"value":"MuxServe: Flexible Multiplexing for Efficient Multiple LLM Serving"},"authors":{"value":["Jiangfei Duan","Runyu Lu","Haojie Duanmu","Xiuhong Li","Xingcheng Zhang","Dahua Lin","Ion Stoica","Hao Zhang"]}},"tmdate":1731581222607,"pdate":1704067200000,"tcdate":1727762152806,"writers":["~"],"signatures":["~Hao_Zhang2"],"forum":"xPUxUX1JDN","license":"CC BY-SA 4.0","number":128216,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1731581222607,"domain":"DBLP.org","id":"xPUxUX1JDN","version":2},{"content":{"venue":{"value":"ICML 2024 Poster"},"abstract":{"value":"Large language models (LLMs) have demonstrated remarkable performance, and organizations are racing to serve LLMs of varying sizes as endpoints for use-cases like chat, programming and search. However, efficiently serving multiple LLMs poses significant challenges for existing approaches due to varying popularity of LLMs. In the paper, we present MuxServe, a flexible spatial-temporal multiplexing system for efficient multiple LLM serving. The key insight behind is to colocate LLMs considering their popularity to multiplex memory resources, and leverage the characteristics of prefill and decoding phases to separate and flexibly colocate them to multiplex computation resources. MuxServe formally formulates the multiplexing problem, and proposes a novel placement algorithm and adaptive batch scheduling strategy to identify optimal colocations and maximize utilization. MuxServe designs a unified resource manager to enable flexible and efficient multiplexing. Evaluation results show that MuxServe can achieves up to $1.8\\times$ higher throughput or processes $2.9\\times$ more requests within $99\\%$ SLO attainment. The code is available at: https://github.com/hao-ai-lab/MuxServe."},"_bibtex":{"value":"@inproceedings{\nduan2024muxserve,\ntitle={MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple {LLM} Serving},\nauthor={Jiangfei Duan and Runyu Lu and Haojie Duanmu and Xiuhong Li and Xingcheng ZHANG and Dahua Lin and Ion Stoica and Hao Zhang},\nbooktitle={Forty-first International Conference on Machine Learning},\nyear={2024},\nurl={https://openreview.net/forum?id=R0SoZvqXyQ}\n}"},"title":{"value":"MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving"},"pdf":{"value":"/pdf/36d7e72e212982ce5ede445e0cf902f49e381240.pdf"},"venueid":{"value":"ICML.cc/2024/Conference"},"paperhash":{"value":"duan|muxserve_flexible_spatialtemporal_multiplexing_for_multiple_llm_serving"},"authorids":{"value":["~Jiangfei_Duan1","~Runyu_Lu2","~Haojie_Duanmu1","~Xiuhong_Li1","~Xingcheng_ZHANG2","~Dahua_Lin1","~Ion_Stoica1","~Hao_Zhang2"]},"authors":{"value":["Jiangfei Duan","Runyu Lu","Haojie Duanmu","Xiuhong Li","Xingcheng ZHANG","Dahua Lin","Ion Stoica","Hao Zhang"]}},"tmdate":1719287258517,"pdate":1714610390438,"tcdate":1706795589195,"writers":["ICML.cc/2024/Conference","ICML.cc/2024/Conference/Submission5489/Authors"],"signatures":["ICML.cc/2024/Conference/Submission5489/Authors"],"forum":"R0SoZvqXyQ","license":"CC BY 4.0","number":5489,"cdate":1706795589195,"readers":["everyone"],"invitations":["ICML.cc/2024/Conference/-/Submission","ICML.cc/2024/Conference/-/Post_Submission","ICML.cc/2024/Conference/-/Edit","ICML.cc/2024/Conference/Submission5489/-/Camera_Ready_Revision"],"mdate":1719287258517,"odate":1717693064704,"domain":"ICML.cc/2024/Conference","id":"R0SoZvqXyQ","version":2},{"content":{"venue":{"value":"ICML 2024"},"venueid":{"value":"dblp.org/conf/ICML/2024"},"paperhash":{"value":"duan|muxserve_flexible_spatialtemporal_multiplexing_for_multiple_llm_serving"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Jiangfei_Duan:","https://dblp.org/search/pid/api?q=author:Runyu_Lu:","https://dblp.org/search/pid/api?q=author:Haojie_Duanmu:","https://dblp.org/search/pid/api?q=author:Xiuhong_Li:","~Xingcheng_Zhang3","~Dahua_Lin1","https://dblp.org/search/pid/api?q=author:Ion_Stoica:","~Hao_Zhang2"]},"html":{"value":"https://openreview.net/forum?id=R0SoZvqXyQ"},"_bibtex":{"value":"@inproceedings{DBLP:conf/icml/DuanLDLZLS024,\n  author={Jiangfei Duan and Runyu Lu and Haojie Duanmu and Xiuhong Li and Xingcheng Zhang and Dahua Lin and Ion Stoica and Hao Zhang},\n  title={MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving},\n  year={2024},\n  cdate={1704067200000},\n  url={https://openreview.net/forum?id=R0SoZvqXyQ},\n  booktitle={ICML},\n  crossref={conf/icml/2024}\n}\n"},"title":{"value":"MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving"},"authors":{"value":["Jiangfei Duan","Runyu Lu","Haojie Duanmu","Xiuhong Li","Xingcheng Zhang","Dahua Lin","Ion Stoica","Hao Zhang"]}},"tmdate":1731581222280,"pdate":1704067200000,"tcdate":1727762153021,"writers":["~"],"signatures":["~Hao_Zhang2"],"forum":"eTttZiJw3a","license":"CC BY-SA 4.0","number":128224,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1731581222280,"domain":"DBLP.org","id":"eTttZiJw3a","version":2},{"content":{"comment":{"value":"Thank you for your response and for pointing out the need for further evaluation. We conducted additional experiments as below.\n\n(1) We conducted a real-world experiment comparing typical autoscaling (SLLM-GPU) and WarmServe under stable workloads. The experimental setup is identical to the paper (16 GPUs, α=0.5). Results (P99 TTFT in seconds) are shown below:\n\n|CV|RPS|SLLM-GPU|WarmServe|\n|---:|----:|--------:|----------:|\n|0.1|25|5.660|0.267|\n|0.1|35|5.734|0.487|\n|0.5|25|5.776|0.306|\n|0.5|35|6.160|0.510|\n\nEven under stable workloads, SLLM-GPU exhibits P99 TTFT exceeding 5 seconds, whereas WarmServe maintains sub-second latency.\n\n(2) We extended the large-scale simulation experiments to higher RPS and CV combinations. Results (P99 TTFT in seconds) are shown below:\n\n|CV|RPS|SLLM-GPU|Cold Start%|WarmServe|\n|---:|----:|--------------:|----:|-------------:|\n|0.1|800|5.8|0.073%|0.26|\n|0.1|960|5.7|0.10%|0.43|\n|0.1|1120|5.8|0.094%|0.49|\n|0.1|1280|53|0.052%|0.61|\n|0.1|1440|170|0.019%|0.87|\n|0.5|800|5.8|0.083%|0.31|\n|0.5|960|6.1|0.075%|0.43|\n|0.5|1120|15|0.099%|0.52|\n|0.5|1280|48|0.043%|0.67|\n|0.5|1440|180|0.019%|0.93|\n|1.0|800|5.8|0.10%|0.44|\n|1.0|960|5.8|0.12%|0.50|\n|1.0|1120|5.9|0.094%|0.73|\n|1.0|1280|47|0.058%|0.65|\n|1.0|1440|170|0.029%|1.1|\n|3.0|800|6.2|0.32%|1.1|\n|3.0|960|7.6|0.21%|1.0|\n|3.0|1120|11|0.18%|1.1|\n|3.0|1280|56|0.071%|1.1|\n|3.0|1440|170|0.054%|3.9|\n\nUnder higher loads, SLLM-GPU suffers significantly elevated TTFT (up to 170s). This is because all models have numerous requests in the waiting queue, forcing frequent instance switching (cold start) across models. Each cold start blocks subsequent requests, creating cascading queueing delays. The decrease in cold start percentage at higher RPS is due to more requests being served during instance lifetime.\n\n(3) We also note that our simulator can closely match real-world system behavior.\nFor example, under 16 GPUs with RPS=15 and α=0.5, the P99 TTFT comparison is as follows:\n\n| System | Sim | Real | Error |\n|---|---|---|---|\n| WarmServe | 0.24 | 0.25 | -4% |\n| SLLM-GPU | 7.2 | 7.2 | -0% |\n| MuxServe | 35 | 34 | +2.9% |\n\nAll errors are within 5%, confirming the accuracy of our simulation.\n\n---\n\nWe hope that these additional results help address your remaining concerns. \nWe are grateful for the effort you have invested in reviewing our work,\nand we would sincerely appreciate your consideration in raising the score."}},"parentInvitations":"ICML.cc/2026/Conference/-/Reply_Rebuttal_Comment","tmdate":1782367327186,"tcdate":1775292451364,"writers":["ICML.cc/2026/Conference","ICML.cc/2026/Conference/Submission3516/Authors"],"signatures":["ICML.cc/2026/Conference/Submission3516/Authors"],"forum":"DVHpvumD60","number":1,"license":"CC BY 4.0","cdate":1775292451364,"readers":["everyone"],"invitations":["ICML.cc/2026/Conference/Submission3516/Official_Review1/Rebuttal1/Rebuttal_Acknowledgement1/-/Reply_Rebuttal_Comment","ICML.cc/2026/Conference/-/Edit"],"mdate":1782367327186,"domain":"ICML.cc/2026/Conference","replyto":"7fqIcCQ24l","id":"bfTCpNsLHd","forumContent":{"venue":{"value":"ICML 2026 regular"},"keywords":{"value":["LLM Inference","Multi-Model Serving","Prewarming"]},"_bibtex":{"value":"@inproceedings{\nlou2026warmserve,\ntitle={WarmServe: Enabling One-for-Many {GPU} Prewarming for Multi-{LLM} Serving},\nauthor={Chiheng Lou and Sheng Qi and Rui Kang and Yong Zhang and Chen Sun and pengcheng wang and Xuanzhe Liu and Xin Jin},\nbooktitle={Forty-third International Conference on Machine Learning},\nyear={2026},\nurl={https://openreview.net/forum?id=DVHpvumD60}\n}"},"title":{"value":"WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Serving"},"paperhash":{"value":"lou|warmserve_enabling_oneformany_gpu_prewarming_for_multillm_serving"},"originally_submitted_PDF":{"value":"/pdf/93c7d59de02fa64daee20fcaae868904639befd3.pdf"},"TLDR":{"value":"We introduce WarmServe, a multi-LLM serving system that drastically reduces cold-start latency by using workload forecasts to proactively load models."},"primary_area":{"value":"deep_learning->large_language_models"},"abstract":{"value":"Deploying multiple models within shared GPU clusters is a key strategy to improve resource efficiency in large language model (LLM) serving.\nExisting multi-LLM serving systems improve GPU utilization at the cost\nof degraded inference performance, particularly time-to-first-token (TTFT).\nWe attribute this degradation to the lack of awareness regarding future workload characteristics.\nIn contrast, recent analyses have shown the strong periodicity and long-term predictability of real-world LLM serving workloads.\nIn this paper, we propose \\textit{one-for-many GPU prewarming}, which proactively loads parameters from multiple models onto GPUs based on workload forecasts.\nThese prewarmed weights enable the system to promptly instantiate serving instances upon encountering request bursts.\nWe design and implement WarmServe, a multi-LLM serving system incorporating three key techniques:\n(1) a model placement algorithm that optimizes prewarming decisions to minimize cross-model prewarming interference,\n(2) a KV cache reservation strategy that repurposes idle KV cache space on running GPUs for prewarming new models,\nand (3) an efficient GPU memory switching mechanism for tensor management.\nEvaluation on real-world datasets shows that WarmServe reduces tail TTFT by up to 50.8$\\times$ compared to the state-of-the-art autoscaling-based system,\nwhile supporting up to 2.5$\\times$ higher request throughput than the GPU-sharing system."},"link_to_code":{"value":"https://github.com/LLMServe/WarmServe"},"pdf":{"value":"/pdf/65f24a25be88037deaff37d0c441476f812ad8c1.pdf"},"lay_summary":{"value":"Modern AI platforms host dozens of specialized large language models — some answer chat questions, others write code, and others tackle complex reasoning. Cocurrently running all of them efficiently is expensive because each one demands powerful, scarce GPU hardware while user demand swings wildly throughout the day. Today's systems face an uncomfortable trade-off: keeping every model permanently loaded wastes hardware during quiet periods, but loading models on demand forces users to wait many seconds when traffic suddenly spikes. Sharing one GPU among several active models avoids the wait but cripples each model's working memory and hurts performance.\n\nWe start from a simple observation: although minute-to-minute traffic looks chaotic, daily and weekly usage patterns are remarkably predictable. We use these forecasts to \"prewarm\" GPUs — loading the weights of several likely-needed models in advance, so a serving instance can be activated almost instantly when a burst arrives. We further developed techniques that decide which models to prewarm together, sneak preparations into GPUs that are about to free up, and switch between models without disruption.\n\nOur system, WarmServe, responds up to 50× faster to traffic spikes and serves up to 2.5× more requests than today's leading systems, making AI services cheaper to run and faster for users."},"venueid":{"value":"ICML.cc/2026/Conference"},"authorids":{"value":["~Chiheng_Lou1","~Sheng_Qi1","~Rui_Kang2","~Yong_Zhang30","~Chen_Sun15","~pengcheng_wang14","~Xuanzhe_Liu3","~Xin_Jin10"]},"authors":{"value":["Chiheng Lou","Sheng Qi","Rui Kang","Yong Zhang","Chen Sun","pengcheng wang","Xuanzhe Liu","Xin Jin"]}},"version":2},{"content":{"decision":{"value":"Accept (poster)"},"comment":{"value":"This paper proposes HyGen, an efficient serving system for large language models using online-offline request co-location. It uses a latency predictor and an SLO-aware profiler, coupled with SLO-aware offline scheduling policies designed to maximize throughput and ensure fairness.\n\nStrengths:\n1. This paper introduces a new and important problem, i.e., efficient co-location of online and offline workloads.\n\nWeaknesses:\n1. As suggested by cK9i, there are many questions regarding how this new framework applies to different models and compute resource constraints and how this new framework addresses some extreme edge cases.\n\nThese gaps persist despite the rebuttal and are essential to assessing HyGen’s broader applicability and reliability.\n\nHowever, I believe this paper marks a significant step froward in this particular research direction.\n\nFor the remaining concerns raised by cK9i, see the author's final remarks.\n\nP1. Our data-driven latency predictor, T_batch = f(Sp, Sd, S_p^2, S_d^2, Np, Nd), adapts flexibly to new architectures. For hybrid models (e.g., Mamba-Transformer), it learns a reduced S_p^2 coefficient (Transformer cost) while capturing Mamba’s linear cost via Sp, consistent with Flash-Linear-Attention GitHub repo's report. For MoE models, due to a fixed activated expert size per token, MoE-FFN costs are captured by Sp and Sd. This again highlights the flexibility and effectiveness of our design.\n\nP2. The “warm-up” concern reflects a misunderstanding. Latency predictor and budget are fixed via one-time, pre-deployment profiling; No runtime stabilization and no SLO risk at startup. The system is immediately ready.\n\nP3. HyGen uses a multi-layer strategy for traffic bursts. First, its latency budget is robust to in-distribution bursts, as it is derived from realistic traces containing such bursts. On spikes, online queues fill, offline requests are preempted/blocked (Fig. 8). Our evaluations on realistic workloads (Fig. 4), with large fluctuations (e.g., request load varies up to 8x in minutes), show HyGen outperforms prior methods. Second, it remains resilient to out-of-distribution events, maintaining SLOs even with 20% MAPE (Fig. 14). Third, for lasting workload shifts, it adapts using feedback from online deployment, asynchronously to serving. Predictor retraining is highly efficient (80k samples in 15 ms).\n\nP4. Baselines were transparently optimized. The Sarathi-offline ceiling baseline tunes chunk size via hyperparameter search (Line 252), achieving 12% throughput improvement over default.\n\nP5. Our problem scope is orthogonal to the cited hybrid systems. Multi-LoRA methods (e.g., Punica) batch requests from different LoRAs, while multi-model methods (e.g., MuxServe) multiplex models across space. Instead, HyGen batches online and offline requests, addressing distinct optimization problems that are not interchangeable.\n\nP6. The profiler’s practicality comes from its data-driven design: it measures performance on target hardware under the actual online request distribution, capturing hardware and memory constraints. Our evaluations show that HyGen efficiently adjusts latency budgets when model, hardware, workload, or SLO changes (Figures 4, 8, 14, and our A5000 experiment replied in Concern 1), as noted in P3."},"title":{"value":"Paper Decision"}},"parentInvitations":"NeurIPS.cc/2025/Conference/-/Decision","nonreaders":[],"tmdate":1761717972342,"tcdate":1758113222620,"writers":["NeurIPS.cc/2025/Conference","NeurIPS.cc/2025/Conference/Program_Chairs"],"signatures":["NeurIPS.cc/2025/Conference/Program_Chairs"],"forum":"cQxLCVa9u7","number":1,"license":"CC BY 4.0","cdate":1758113222620,"readers":["everyone"],"invitations":["NeurIPS.cc/2025/Conference/Submission15013/-/Decision","NeurIPS.cc/2025/Conference/-/Edit"],"mdate":1761717972342,"domain":"NeurIPS.cc/2025/Conference","replyto":"cQxLCVa9u7","id":"jvNHtcRYBq","forumContent":{"TLDR":{"value":"Colocating online and offline LLM inference requests in the same inference engine."},"venue":{"value":"NeurIPS 2025 poster"},"keywords":{"value":["LLM Inference","Request Scheduling","Machine Learning Systems"]},"primary_area":{"value":"infrastructure"},"abstract":{"value":"Large language models (LLMs) have facilitated a wide range of applications with distinct service-level objectives (SLOs), from latency-sensitive online tasks like interactive chatbots to throughput-oriented offline workloads like data synthesis. The existing deployment model, which dedicates machines to each workload, simplifies SLO management but often leads to poor resource utilization. This paper introduces HyGen, an interference-aware LLM serving system that enables efficient co-location of online and offline workloads while preserving SLOs. HyGen incorporates two key innovations: (1) performance control mechanisms, including a latency predictor to estimate batch execution time and an SLO-aware profiler to quantify latency interference, and (2) SLO-aware offline scheduling policies that maximize serving throughput and prevent starvation. Our evaluation on production workloads shows that HyGen achieves up to 3.9-5.8× throughput gains over online and hybrid serving baselines, while ensuring latency SLOs. The code of HyGen is publicly available at https://github.com/UIUC-MLSys/HyGen."},"_bibtex":{"value":"@inproceedings{\nsun2025hygen,\ntitle={HyGen: Efficient {LLM} Serving via Elastic Online-Offline Request Co-location},\nauthor={Ting Sun and Penghan Wang and Fan Lai},\nbooktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},\nyear={2025},\nurl={https://openreview.net/forum?id=cQxLCVa9u7}\n}"},"title":{"value":"HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-location"},"pdf":{"value":"/pdf/40f47181fc0983cc4ba51054de34b5d6a4e75e14.pdf"},"venueid":{"value":"NeurIPS.cc/2025/Conference"},"paperhash":{"value":"sun|hygen_efficient_llm_serving_via_elastic_onlineoffline_request_colocation"},"authorids":{"value":["~Ting_Sun2","~Penghan_Wang1","~Fan_Lai1"]},"authors":{"value":["Ting Sun","Penghan Wang","Fan Lai"]}},"version":2}],"count":5}