{"notes":[{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2510.09665v2"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"cheng|lmcache_an_efficient_kv_cache_layer_for_enterprisescale_llm_inference"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Yihua_Cheng:","~Yuhan_Liu3","https://dblp.org/search/pid/api?q=author:Jiayi_Yao:","https://dblp.org/search/pid/api?q=author:Yuwei_An:","~Xiaokun_Chen1","~Shaoting_Feng1","https://dblp.org/search/pid/api?q=author:Yuyang_Huang:","https://dblp.org/search/pid/api?q=author:Samuel_Shen:","https://dblp.org/search/pid/api?q=author:Kuntai_Du:","https://dblp.org/search/pid/api?q=author:Junchen_Jiang:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2510.09665"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2510-09665,\n publtype={informal},\n author={Yihua Cheng and Yuhan Liu and Jiayi Yao and Yuwei An and Xiaokun Chen and Shaoting Feng and Yuyang Huang and Samuel Shen and Kuntai Du and Junchen Jiang},\n title={LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference},\n year={2025},\n month={October},\n cdate={1759276800000},\n journal={CoRR},\n volume={abs/2510.09665},\n url={https://doi.org/10.48550/arXiv.2510.09665}\n}\n"},"abstract":{"value":"KV cache has traditionally been stored in GPU memory to accelerate the decoding phase of large language model (LLM) inference. However, it is increasingly necessary to move KV caches outside GPU devices, to enable cache reuse across different queries and inference engines. Our real-world usage statistics confirm this trend: over time, the total KV cache stored by users has grown rapidly, far exceeding the capacity of GPU memory. Despite this need, there lacks an efficient solution for offloading and transferring KV caches. We present LMCACHE, the first and so far the most efficient open-source KV caching solution, which extracts and stores KV caches generated by modern LLM engines (vLLM and SGLang) out of the GPU memory and shares them across engines and queries. LMCACHE supports both cache offloading (prefix reuse across queries) and prefill-decode (PD) disaggregation (cross-engine/GPU cache transfer). LMCACHE's high performance and wide adoption stem from the following contributions: (1) highly optimized KV cache data movement powered by batched data movement operations, compute and I/O pipelining; (2) a modular KV cache connector component, decoupling LMCACHE from the rapid evolution of inference engines; (3) a first-class control API for flexible cache orchestration across GPU, CPU, storage, and network layers. Our evaluation shows that combining LMCACHE with vLLM achieves up to 15x improvement in throughput across workloads such as multi-round question answering and document analysis. Large-scale adoption of LMCACHE in enterprise settings provides us valuable insights, for example, fetching KV cache from remote storage has unsurprisingly benefits to prefill delay, and that context truncation, which is a widely applied technique in industry, can greatly reduce prefix cache hit ratio by half. The source code of LMCACHE is at: https://github.com/LMCache/LMCache."},"title":{"value":"LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference"},"authors":{"value":["Yihua Cheng","Yuhan Liu","Jiayi Yao","Yuwei An","Xiaokun Chen","Shaoting Feng","Yuyang Huang","Samuel Shen","Kuntai Du","Junchen Jiang"]}},"tmdate":1785335354166,"pdate":1767139200000,"externalIds":["dblp:journals/corr/abs-2510-09665"],"tcdate":1769503793396,"writers":["~"],"signatures":["~Yuhan_Liu3"],"forum":"M1BI86jGg6","license":"CC BY-SA 4.0","number":807000,"cdate":1759276800000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1785335354166,"domain":"DBLP.org","id":"M1BI86jGg6","version":2},{"content":{"comment":{"value":"**W2 Recalibration when model structure changes; limited end-to-end serving results** \nWe would like to point out that the calibration completes within minutes on modern GPUs, and could be performed with CPUs. Recalibration would likely constitute a fraction of the cost and time required to alter the weights of the model, or reconfigure how the computation is split over accelerators, for instance, when the number of tensor- or pipeline-parallel instances changes.\nWe provide additional end-to-end measurements in a simplified, multi-user scenario. To this end, we have run vLLM with LMCache managing the KV cache on a workload in which, for 64K of initial input tokens, the users ask questions that are between 16 and 100 tokens and receive 100-token answers, with no delays in between conversation turns. We have used Llama 3.3 70B in FP8 precision, split over 2x H100 80GB GPUs using tensor parallelism. LMCache has been configured to use 128GiB of host (CPU DRAM) memory per GPU, which is equivalent to using 1TiB in an 8-GPU server. We adjust the load of the server by changing the number of clients that concurrently hold conversations.\n\n| vLLM num of concurrent clients | Response latency of vanilla (s) | Response latency of KVTC 16x (s) | TTFT of vanilla (s) | TTFT of KVTC 16x (s) |\n| -----------------------------: | ------------------------------: | -------------------------------: | -------: | -----------------------: |\n| 1 | 2.5 | 2.5 | 0.2 | 0.2 |\n| 2 | 2.9 | 2.9 | 0.3 | 0.3 |\n| 4 | 8.6 | 9.1 | 1.2 | 1.7 |\n| 6 | 12.7 | 13.9 | 2.0 | 2.7 |\n| 8 | 17.3 | 18.0 | 2.8 | 3.5 |\n| 10 | 21.4 | 24.2 | 3.4 | 4.5 |\n| 12 | 155.7 | 27.9 | 136.6 | 5.6 |\n| 14 | 180.9 | 31.3 | 159.0 | 6.4 |\n| 16 | 208.2 | 37.1 | 181.6 | 7.3 |\n\nWith 12 or more clients, the dedicated amount of host memory becomes too little to hold the KV caches and forces recomputation, which is reflected in spiking latency. In a more realistic scenario, there would be more concurrent users taking pauses between conversation turns. We note that in this test, the KV cache is compressed with KVTC only for storage in host memory; it does not take advantage of compressing the KV cache for storage in GPU HBM, which would bring additional latency benefits, as its implementation is more involved.\n\n[LMCache] Cheng et al., 2025, LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference. \n[vLLM] Woosuk Kwon et al., 2023, Efficient Memory Management for Large Language Model Serving with PagedAttention."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763761616738,"tcdate":1763761291864,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission19321/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission19321/Authors"],"forum":"aNVKROYpLB","number":3,"license":"CC BY 4.0","cdate":1763761291864,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission19321/-/Official_Comment"],"mdate":1763761616738,"domain":"ICLR.cc/2026/Conference","replyto":"j2gKWLupAX","id":"ER3gZkJtiG","forumContent":{"TLDR":{"value":"We present KVTC, a lightweight transform coder that allows for extended retention of transformer KV-cache via compression."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["transformer","kv cache","compression"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Serving large language models (LLMs) at scale necessitates efficient key-value (KV) cache management. KV caches can be reused across conversation turns via shared-prefix prompts that are common in iterative code editing and chat. However, stale caches consume scarce GPU memory, require offloading, or force recomputation. We present KVTC, a lightweight transform coder that compresses KV caches for compact on-GPU and off-GPU storage. Drawing on classical media compression, KVTC combines PCA-based feature decorrelation, adaptive quantization, and entropy coding. It requires only a brief initial calibration and leaves model parameters unchanged. By exploiting redundancies in KV caches, KVTC achieves up to 20x compression while maintaining reasoning and long-context accuracy, and 40x or higher for specific use cases. We test KVTC with Llama 3, Mistral NeMo, and R1-Qwen 2.5 models across benchmarks\nincluding AIME25, GSM8K, LiveCodeBench, LongBench, MATH-500, MMLU, Qasper and RULER.\nIt consistently outperforms inference-time baselines such as token eviction, quantization, and SVD-based methods, while achieving higher compression ratios. These results support KVTC as a practical building block for memory-efficient LLM serving with reusable KV caches."},"_bibtex":{"value":"@inproceedings{\nstaniszewski2026kv,\ntitle={{KV} Cache Transform Coding for Compact Storage in {LLM} Inference},\nauthor={Konrad Staniszewski and Adrian {\\L}a{\\'n}cucki},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=aNVKROYpLB}\n}"},"title":{"value":"KV Cache Transform Coding for Compact Storage in LLM Inference"},"pdf":{"value":"/pdf/1cef9774f0f0cf7bb9e4b167882e3ad3ef8cde16.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"staniszewski|kv_cache_transform_coding_for_compact_storage_in_llm_inference"},"authorids":{"value":["~Konrad_Staniszewski1","~Adrian_Łańcucki1"]},"authors":{"value":["Konrad Staniszewski","Adrian Łańcucki"]}},"version":2},{"content":{"summary":{"value":"The paper proposes SGLANG-LSM, a disk-resident KV-cache backend for large language model (LLM) serving that replaces the traditional file-per-object layout with a Log-Structured Merge (LSM) tree index and an append-only tensor log. It further introduces an adaptive controller that dynamically tunes LSM parameters (e.g., tiering/leveling, size ratio) based on observed read/write ratios, and implements runtime services for batch tensor codecs and background file merging. Experiments conducted on a single machine (NVIDIA A30 GPU, 64 GB RAM, 8 TB SSD) report a 143% increase in cache-hit rate and up to 24% reduction in time-to-first-token (TTFT) compared to SGLang’s existing memory- or file-based backends, under synthetic workloads."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"1. Performance scaling beyond 16k tokens.\nThe TTFT trends for 4k and 8k sequences are counterintuitive: even when cache hit rates are significantly higher than baselines (e.g., 8k at request ID ≈ 250), the TTFT reduction is marginal. Prior work that overlaps KV-cache computation and disk loading (Jin et al., 2024) shows that early tokens' KVCache recomputation is faster than loading, which might be the reason why SGLang-LSM didn't excel in those cases. It will make more sense for SGLang-LSM to evaluate on a longer context (e.g., 32k or 64k)?\n2. Scalability across nodes and tiers.\nThe current experiments are single-node and local-disk only. How would the system extend to remote disks, multi-node environments, or tiered storage hierarchies (e.g., GPU → CPU → SSD → network store)? Such results are critical to substantiate the “scalability” claim.\n3. Correlation between LSM and hit rate.\nThe motivation for using LSM is unclear. If we want to deal with too many small files and increase the capacity, we can concatenate the KVCache and store an index file separately; this would be much easier. I wonder why LSM can increase the hit rate."},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. The paper correctly identifies a practical scalability problem: existing file-per-object KV-cache backends create millions of small files, leading to metadata overhead and poor I/O locality.\n2. The application of LSM-tree storage to KV-cache management is relatively new.\n3. The reported 24% TTFT reduction represents a measurable and relevant performance improvement for latency-sensitive LLM inference."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Questionable necessity of LSM for this workload.\nModern KV-cache management systems (e.g., LMCache (Cheng et al., 2025)) already compute hash-chained prefix identifiers and perform O(1) lookups over content-addressed append logs. Since most KV-cache objects are immutable and written once, the motivation for adopting a full LSM-tree (with compaction and sorted order maintenance) is not well justified.\n2. Incomplete evaluation details.\nThe evaluation setup raises questions: the authors report handling 7 million files even though the hardware configuration (A30 + 64 GB RAM) is relatively small. The paper does not specify the KV-cache page size, or the prefix-sharing ratio, all of which are necessary to assess scalability and reproducibility.\n3. Misleading title and framing.\nThe title “10× Better Scalability” is not supported by the presented results.\n4. Weak and outdated baselines.\nThe baseline SGLang (file)—is too naive to substantiate strong claims of improvement. The paper does not compare against state-of-the-art tiered KV-cache systems such as LMCache+vLLM."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762927120885,"tcdate":1761859638177,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission17124/Reviewer_izj8"],"signatures":["ICLR.cc/2026/Conference/Submission17124/Reviewer_izj8"],"forum":"QqpbjtPbIc","number":2,"license":"CC BY 4.0","cdate":1761859638177,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission17124/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762927120885,"domain":"ICLR.cc/2026/Conference","replyto":"QqpbjtPbIc","id":"uW87LtkaBC","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["LLM serving","database"]},"primary_area":{"value":"infrastructure, software libraries, hardware, systems, etc."},"abstract":{"value":"Large language models (LLMs) rely on Key-Value (KV) cache to reduce time-\nto-first-token (TTFT) latency, but existing disk-based KV cache systems using\nfile-per-object layouts suffer from severe scalability bottlenecks due to file system\nmetadata overhead, I/O inefficiency, and poor spatial locality. This paper presents\nSGLANG-LSM, a database-inspired system that leverages Log-Structured Merge-\ntree (LSM-tree) architectures for scalable KV cache management. SGLANG-LSM\nimplements a layered system design with three coordinated components: (1) a\nprefix-preserving storage engine that maintains token sequence locality while\nefficiently storing large KV cache tensors through key-value separation, (2) an\nadaptive controller that dynamically optimizes LSM-tree configurations based on\nshifting workload characteristics, and (3) runtime services including batch opera-\ntions and automatic resource management for production deployment. Evaluation\non large-scale dynamic workloads demonstrates that SSGLANG-LSM significantly\nimproves cache hits by up to 143% and reduces TTFT by up to 24% compared to\nstate-of-the-art systems, representing the first systematic application of database\nstorage architectures to large-scale LLM cache management."},"_bibtex":{"value":"@misc{\nyu2026on,\ntitle={On 10X Better Scalability: {KV} Stores Scale Up {KV} Cache},\nauthor={Weiping Yu and Ye Jiarui and He Mengke and Junfeng Liu and Siqiang Luo},\nyear={2026},\nurl={https://openreview.net/forum?id=QqpbjtPbIc}\n}"},"title":{"value":"On 10X Better Scalability: KV Stores Scale Up KV Cache"},"pdf":{"value":"/pdf/6ab6f1d8bd3e7b9071c53623817244d7a366ffec.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"yu|on_10x_better_scalability_kv_stores_scale_up_kv_cache"},"authorids":{"value":["~Weiping_Yu1","~Ye_Jiarui1","~He_Mengke1","~Junfeng_Liu6","~Siqiang_Luo1"]},"authors":{"value":["Weiping Yu","Ye Jiarui","He Mengke","Junfeng Liu","Siqiang Luo"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2409.13761v2"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"cheng|do_large_language_models_need_a_content_delivery_network"},"authorids":{"value":["~Yihua_Cheng2","~Kuntai_Du1","https://dblp.org/search/pid/api?q=author:Jiayi_Yao:","https://dblp.org/search/pid/api?q=author:Junchen_Jiang:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2409.13761"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2409-13761,\n publtype={informal},\n author={Yihua Cheng and Kuntai Du and Jiayi Yao and Junchen Jiang},\n title={Do Large Language Models Need a Content Delivery Network?},\n year={2024},\n cdate={1704067200000},\n journal={CoRR},\n volume={abs/2409.13761},\n url={https://doi.org/10.48550/arXiv.2409.13761}\n}\n"},"abstract":{"value":"As the use of large language models (LLMs) expands rapidly, so does the range of knowledge needed to supplement various LLM queries. Thus, enabling flexible and efficient injection of new knowledge in LLM inference is critical. Three high-level options exist: (i) embedding the knowledge in LLM's weights (i.e., fine-tuning), (ii) including the knowledge as a part of LLM's text input (i.e., in-context learning), or (iii) injecting the KV caches of the new knowledge to LLM during prefill. This paper argues that, although fine-tuning and in-context learning are popular, using KV caches as the medium of knowledge could simultaneously enable more modular management of knowledge injection and more efficient LLM serving with low cost and fast response. To realize these benefits, we envision a Knowledge Delivery Network (KDN), a new system component in LLM services that dynamically optimizes the storage, transfer, and composition of KV cache across LLM engines and other compute and storage resources. We believe that, just like content delivery networks (CDNs), such as Akamai, enabled the success of the Internet ecosystem through their efficient data delivery, KDNs will be critical to the success of LLM applications through their efficient knowledge delivery. We have open-sourced a KDN prototype at https://github.com/LMCache/LMCache."},"title":{"value":"Do Large Language Models Need a Content Delivery Network?"},"authors":{"value":["Yihua Cheng","Kuntai Du","Jiayi Yao","Junchen Jiang"]}},"tmdate":1732914551459,"pdate":1704067200000,"tcdate":1732747983771,"writers":["~"],"signatures":["~Kuntai_Du1"],"forum":"nrGe7K13EN","license":"CC BY-SA 4.0","number":251803,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1732914551459,"domain":"DBLP.org","id":"nrGe7K13EN","version":2},{"content":{"comment":{"value":"**W2 Latency/TTFT overhead when KV cache is read from HBM (cache hit)?** \nThe three building blocks of KVTC, which are projection, quantization and entropy-coding, are memory-bound for long contexts. In the current implementation, in which these are invoked as separate kernels, the cost of decompression would be equal or less to the cost of generating three tokens at the same batch size. However, fusing three stages into a single kernel would speed up decompression roughly 3x. We would like to emphasize that the point of using KVTC to compress KV cache for its residency in HBM is to free up HBM memory for other conversation, and in the end increase the number of concurrently served users and throughput of the node. KVTC can be applied on-demand, even to old caches, so as long as there is enough free memory in HBM, the KV caches can remain uncompressed. In other words, if a KV cache in HBM needs to be decompressed with KVTC, it means that KV Cache Manager chose to compress it in the past, and the node benefited from higher throughput or the cache has been prevented from being offloaded/evicted.\n\nIn addition, we provide the results of end-to-end TTFT measurements in a simplified, multi-user scenario. To this end, we have run vLLM with LMCache managing the KV cache on a workload in which, for 64K of initial input tokens, the users ask questions that are between 16 and 100 tokens and receive 100-token answers, with no delays in between conversation turns. We have used Llama 3.3 70B in FP8 precision, split over 2x H100 80GB GPUs using tensor parallelism. LMCache has been configured to use 128GiB of host (CPU DRAM) memory per GPU, which is equivalent to using 1TiB in an 8-GPU server. We adjust the load of the server by changing the number of clients that concurrently hold conversations.\n\n| vLLM num of concurrent clients | Response latency of vanilla (s) | Response latency of KVTC 16x (s) | TTFT of vanilla (s) | TTFT of KVTC 16x (s) |\n| -----------------------------: | ------------------------------: | -------------------------------: | -------: | -----------------------: |\n| 1 | 2.5 | 2.5 | 0.2 | 0.2 |\n| 2 | 2.9 | 2.9 | 0.3 | 0.3 |\n| 4 | 8.6 | 9.1 | 1.2 | 1.7 |\n| 6 | 12.7 | 13.9 | 2.0 | 2.7 |\n| 8 | 17.3 | 18.0 | 2.8 | 3.5 |\n| 10 | 21.4 | 24.2 | 3.4 | 4.5 |\n| 12 | 155.7 | 27.9 | 136.6 | 5.6 |\n| 14 | 180.9 | 31.3 | 159.0 | 6.4 |\n| 16 | 208.2 | 37.1 | 181.6 | 7.3 |\n\nIn this experiment, the cache has always been compressed, regardless of the remaining memory. As the number of concurrently served clients increases, KV caches no longer fit in the memory and need to be recalculated. For KVTC, that is not the case; the TTFT increases are noticeable, but acceptable in comparison to full response time or recalculation.\n\n[LMCache] Cheng et al., 2025, LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference. \n[vLLM] Woosuk Kwon et al., 2023, Efficient Memory Management for Large Language Model Serving with PagedAttention."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763762555101,"tcdate":1763762555101,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission19321/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission19321/Authors"],"forum":"aNVKROYpLB","number":14,"license":"CC BY 4.0","cdate":1763762555101,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission19321/-/Official_Comment"],"mdate":1763762555101,"domain":"ICLR.cc/2026/Conference","replyto":"EqFzwXpRAf","id":"WznatoS2ll","forumContent":{"TLDR":{"value":"We present KVTC, a lightweight transform coder that allows for extended retention of transformer KV-cache via compression."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["transformer","kv cache","compression"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Serving large language models (LLMs) at scale necessitates efficient key-value (KV) cache management. KV caches can be reused across conversation turns via shared-prefix prompts that are common in iterative code editing and chat. However, stale caches consume scarce GPU memory, require offloading, or force recomputation. We present KVTC, a lightweight transform coder that compresses KV caches for compact on-GPU and off-GPU storage. Drawing on classical media compression, KVTC combines PCA-based feature decorrelation, adaptive quantization, and entropy coding. It requires only a brief initial calibration and leaves model parameters unchanged. By exploiting redundancies in KV caches, KVTC achieves up to 20x compression while maintaining reasoning and long-context accuracy, and 40x or higher for specific use cases. We test KVTC with Llama 3, Mistral NeMo, and R1-Qwen 2.5 models across benchmarks\nincluding AIME25, GSM8K, LiveCodeBench, LongBench, MATH-500, MMLU, Qasper and RULER.\nIt consistently outperforms inference-time baselines such as token eviction, quantization, and SVD-based methods, while achieving higher compression ratios. These results support KVTC as a practical building block for memory-efficient LLM serving with reusable KV caches."},"_bibtex":{"value":"@inproceedings{\nstaniszewski2026kv,\ntitle={{KV} Cache Transform Coding for Compact Storage in {LLM} Inference},\nauthor={Konrad Staniszewski and Adrian {\\L}a{\\'n}cucki},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=aNVKROYpLB}\n}"},"title":{"value":"KV Cache Transform Coding for Compact Storage in LLM Inference"},"pdf":{"value":"/pdf/1cef9774f0f0cf7bb9e4b167882e3ad3ef8cde16.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"staniszewski|kv_cache_transform_coding_for_compact_storage_in_llm_inference"},"authorids":{"value":["~Konrad_Staniszewski1","~Adrian_Łańcucki1"]},"authors":{"value":["Konrad Staniszewski","Adrian Łańcucki"]}},"version":2},{"content":{"comment":{"value":"I appreciate the authors for their detailed and helpful response. I have also read all other reviewers' comments in detail. \n\nOverall, I like the research direction this work opens (leveraging both I/O + compute for efficient KV cache management). I can foresee related ideas like I/O+compute for loading from heterogeneous storage, e.g. CXL memory, serverless storage, etc., or better policy design under the framework of Cake. It might require some additional effort if you're sending something related to OSDI/SOSP though. \n\nI decide to raise my score to 4. The main reason is because (1) my concerns are addressed, and (2) the authors actually integrate Cake into LMCache and vLLM --- So hopefully the impact of the idea will go beyond the paper itself."}},"tmdate":1750257316291,"tcdate":1744102582343,"writers":["ICML.cc/2025/Conference","ICML.cc/2025/Conference/Submission1121/Reviewer_opvL"],"signatures":["ICML.cc/2025/Conference/Submission1121/Reviewer_opvL"],"forum":"WOyOtaO6lQ","number":1,"license":"CC BY 4.0","cdate":1744102582343,"readers":["everyone"],"invitations":["ICML.cc/2025/Conference/Submission1121/Official_Review3/Rebuttal1/-/Rebuttal_Comment","ICML.cc/2025/Conference/-/Edit"],"mdate":1750257316291,"domain":"ICML.cc/2025/Conference","replyto":"I09jvdZuF4","id":"l3nzUoXw1t","forumContent":{"venue":{"value":"ICML 2025 poster"},"TLDR":{"value":"Cake is a KV cache loading system that optimally balances computation and I/O resources through a bidirectional scheduling strategy, reducing Time to First Token (TTFT) by 2.6× on average and improving long-context LLM inference efficiency."},"keywords":{"value":["Prefix caching System Optimization","KV Cache","Low-Latency LLM Inference"]},"primary_area":{"value":"general_machine_learning->hardware_and_software"},"abstract":{"value":"Large Language Models (LLMs) are increasingly deployed in large-scale online services, enabling sophisticated applications. However, the computational overhead of generating key-value (KV) caches in the prefill stage presents a major bottleneck, particularly for long-context inputs. Prefix caching mitigates this issue by storing KV caches for reuse, reducing redundant computation. Despite its advantages, prefix caching suffers from high latency due to the limited I/O bandwidth of storage devices, constraining inference efficiency. To address this challenge, we introduce Cake, a novel KV cache loading system that optimally utilizes both computational and I/O resources in parallel. Cake employs a bidirectional scheduling strategy that dynamically balances KV cache computation and loading, ensuring efficient resource utilization. Additionally, Cake incorporates an adaptive scheduling mechanism that seamlessly integrates with non-prefix caching requests, improving system throughput and adapting to fluctuating resource availabilty. Through extensive evaluations across various hardware configurations, datasets, and storage conditions, Cake achieves on average 2.6× reduction in Time to First Token (TTFT) compared to compute-only and I/O-only methods. Our findings highlight Cake as an effective and practical solution for optimizing long-context LLM inference, bridging the gap between computation and I/O efficiency in large-scale AI deployments."},"_bibtex":{"value":"@inproceedings{\njin2025compute,\ntitle={Compute or Load {KV} Cache? Why Not Both?},\nauthor={Shuowei Jin and Xueshen Liu and Qingzhao Zhang and Zhuoqing Mao},\nbooktitle={Forty-second International Conference on Machine Learning},\nyear={2025},\nurl={https://openreview.net/forum?id=WOyOtaO6lQ}\n}"},"title":{"value":"Compute or Load KV Cache? Why Not Both?"},"pdf":{"value":"/pdf/367a534dac0cfcac16a8506794609c1bc6a4a15c.pdf"},"lay_summary":{"value":"Large Language Models (LLMs) have become vital tools in various online applications, helping users with tasks like answering questions, summarizing documents, and chatting naturally. However, using these models with lengthy text inputs (like an entire book or a long conversation) can be slow, as they need to perform extensive calculations before providing their first response.\n\nPrevious approaches typically relied solely on loading pre-calculated data from storage, but these methods alone have limitations and often fail to sufficiently reduce the latency experienced by users, especially when storage bandwidth is limited. In this work, we introduce a new approach called \"Cake,\" which smartly combines recomputing calculations and loading pre-stored data simultaneously. Cake efficiently manages resources by recomputing certain data while loading other parts from storage, significantly reducing the waiting time even under constrained storage bandwidth.\n\nBy dynamically adapting to available computing power and storage bandwidth, Cake effectively handles real-world conditions, enhancing the responsiveness of LLMs. Our extensive experiments demonstrate that Cake consistently accelerates the initial response time, making large language models faster and more practical for everyday applications."},"venueid":{"value":"ICML.cc/2025/Conference"},"paperhash":{"value":"jin|compute_or_load_kv_cache_why_not_both"},"authorids":{"value":["~Shuowei_Jin1","~Xueshen_Liu1","~Qingzhao_Zhang1","~Zhuoqing_Mao1"]},"authors":{"value":["Shuowei Jin","Xueshen Liu","Qingzhao Zhang","Zhuoqing Mao"]}},"version":2},{"content":{"comment":{"value":"Thank you for the careful and encouraging review.\n\nWe agree with the critical clarification. The revised abstract will state directly that the broad SGLang, LMCache, and vLLM production overlays are write-through compatibility/controller-overhead measurements, not MeritKV-caused speedups. This is already the scope used in the introduction, runtime section, and conclusion.\n\nThe main text already says that semantic mode is not deployable. We will keep that statement and say explicitly that approximate semantic reuse is unsafe without an independent correctness guard.\n\nWe now have measured LRU/LFU capacity-pressure baselines and a separate explicit-state cache-continuation validation, and we will include them in the revision. We will also distinguish speculative-compute waste, storage/cache-efficiency waste, and historical controller feedback more clearly. The capacity-pressure workload will remain described as a controlled trace, not a production request log.\n\nWe will provide exact section and table references when the revision is uploaded. We appreciate the clear separation between critical and optional requests."}},"parentInvitations":"TMLR/-/Official_Comment","tmdate":1786712050534,"tcdate":1786712050534,"writers":["TMLR","TMLR/Paper11074/Authors"],"signatures":["TMLR/Paper11074/Authors"],"forum":"y3j6fucWQP","number":4,"license":"CC BY 4.0","cdate":1786712050534,"readers":["everyone"],"invitations":["TMLR/Paper11074/-/Official_Comment"],"mdate":1786712050534,"domain":"TMLR","replyto":"F4BIznqllA","id":"KJ7VMJsnEA","forumContent":{"submission_length":{"value":"Regular submission (no more than 12 pages of main content)"},"venue":{"value":"Accepted by TMLR"},"code":{"value":"https://github.com/Kushalk0677/meritkv"},"abstract":{"value":"Key-value (KV) cache reuse lowers prefill latency, but lookup, cache manipulation, and speculative precompute can cost more than the computation they save, so a serving system can be slower than no cache even at a high hit rate. We argue that KV reuse should be a per-request economic decision rather than an unconditional mode, and present MeritKV, a lightweight admission-control layer that scores each candidate reuse by its net utility (saved prefill time minus reuse cost minus expected waste) and executes it only when that estimate is positive, bypassing otherwise. The gate is backend-agnostic and composes on top of existing prefix caches, while a single waste-feedback loop makes admission self-correcting as workloads change. Across controlled studies on five Hugging Face models (GPT-2, TinyLlama, Qwen2.5, Gemma, and Phi-3; 124M–3.8B parameters) and ten datasets, MeritKV matches no-cache performance on low-reuse traffic (0.99–1.01×) and observes 1.31–1.61× in custom-HF diagnostic configurations on structured prompts. In a 360-cell admission-policy comparison, it improves speedup from 1.085× to 1.110× while reducing the waste ratio from 24.0% to 5.6% relative to MeritKV-Sem, the reactive-plus-speculative variant without the policy controller. We additionally evaluate twelve models up to 32B and write-through SGLang, LMCache, and vLLM overlays, which establish compatibility and controller overhead, not acceleration. Under capacity pressure, five-seed native SGLang traces on Gemma-4-31B and Qwen2.5-32B show skip-write admission eliminating measured evictions and reducing mean latency/energy by 6.8–7.7%/7.6–8.5% versus native admit-all LRU; Qwen P95 regresses by 1.65% versus write-through, and native LFU remains stronger on that trace. A controlled multi-round trace reduces latency by 11.6–16.7% and improves recovery by 41–61 percentage points versus write-through; recovery remains 15.2–28.2 percentage points above controlled-bank LFU."},"_bibtex":{"value":"@article{\nkhemani2026meritkv,\ntitle={Merit{KV}: Utility-Gated Admission Control for {KV}-Cache Reuse in {LLM} Serving},\nauthor={Kushal Khemani and Evan J. Leri and Sparsh Mittal},\njournal={Transactions on Machine Learning Research},\nissn={2835-8856},\nyear={2026},\nurl={https://openreview.net/forum?id=y3j6fucWQP},\nnote={}\n}"},"video":{"value":"https://www.youtube.com/watch?v=X5VAFEl17zE"},"title":{"value":"MeritKV: Utility-Gated Admission Control for KV-Cache Reuse in LLM Serving"},"changes_since_last_submission":{"value":"Uploading the deanonymized camera-ready version, including author details, affiliations, acknowledgements, declarations, and the public code link, along with minor editorial and reference corrections."},"pdf":{"value":"/pdf/204645d44d3e621a14a925967acfa5e1b573cab6.pdf"},"venueid":{"value":"TMLR"},"paperhash":{"value":"khemani|meritkv_utilitygated_admission_control_for_kvcache_reuse_in_llm_serving"},"authorids":{"value":["~Kushal_Khemani1","~Evan_J._Leri1","~Sparsh_Mittal1"]},"assigned_action_editor":{"value":"~Binhang_Yuan1"},"authors":{"value":["Kushal Khemani","Evan J. Leri","Sparsh Mittal"]}},"version":2},{"content":{"summary":{"value":"This paper introduces Cake, a hybrid KV cache loading system that leverages both I/O resources (for loading) and computational resources (for re-computation). The authors observe that both I/O-only approaches (e.g. LMCache) and compute-only approaches (e.g. vLLM) fall short in practice in terms of minimizing TTFT when loading from local or remote disks. Inspired by this, Cake uses both I/O and computation in parallel when loading from disks (which are high-capacity but low-bandwidth). Evaluations show that Cake can reduce TTFT by 2.6x on average for long-context prefix caching scenarios."},"experimental_designs_or_analyses":{"value":"The paper features solid and comprehensive evaluation of the proposed method, Cake."},"supplementary_material":{"value":"N/A"},"other_comments_or_suggestions":{"value":"N/A"},"overall_recommendation":{"value":4},"other_strengths_and_weaknesses":{"value":"N/A"},"relation_to_broader_scientific_literature":{"value":"This paper lies within the broad area of KV cache management systems for language model inference. The authors focus primarily on acceleration of KV cache loading from disks, which can be helpful to cloud providers in the case of prefix-cache loading or prompt-cache loading."},"claims_and_evidence":{"value":"Most of the claims made by the authors are supported by citations or experiment results. Please see \"Questions For Authors\" for details."},"code_of_conduct":{"value":"Affirmed."},"theoretical_claims":{"value":"N/A"},"methods_and_evaluation_criteria":{"value":"The paper features solid and comprehensive evaluation of the proposed method, Cake."},"questions_for_authors":{"value":"Thank you for submitting this paper to NeurIPS. The core idea of this paper, leveraging I/O and compute in parallel when loading KV cache from disks, is well presented and makes sense. Experiments are also solid and comprehensive. \n\nI have two major questions/concerns. \n1. In reality, I/O (network) and compute capability can vary dramatically from time to time, even with the same set of local/remote hardware, depending on if you have competing workloads, network variance, temperature of hardware, etc. Is your system assuming a fixed config of I/O and compute capability, pre-measured given the hardware stack? If this is the case, do you have real-time profiling that adjusts the capabilities so that your scheduling works well for the real-world scenario? Otherwise it seems very likely that sub-optimal scheduling decisions will occur. \n2. Do you think your system would work for CPU->GPU loading with minor adaptation? If not, what do you think is the major challenge (or maybe loading is all you need in that case)? Is it because disk->CPU/GPU might be the only case where re-computation can be sometimes more efficient (in terms of TTFT) than direct loading?"},"essential_references_not_discussed":{"value":"As far as I know, there is no missing related work (in the area of KV cache loading / prefix caching systems) that remain undiscussed in this paper."}},"nonreaders":[],"tmdate":1750238912514,"tcdate":1741939458529,"writers":["ICML.cc/2025/Conference","ICML.cc/2025/Conference/Submission1121/Reviewer_opvL"],"signatures":["ICML.cc/2025/Conference/Submission1121/Reviewer_opvL"],"forum":"WOyOtaO6lQ","number":3,"license":"CC BY 4.0","cdate":1741939458529,"readers":["everyone"],"invitations":["ICML.cc/2025/Conference/Submission1121/-/Official_Review","ICML.cc/2025/Conference/-/Edit"],"mdate":1750238912514,"domain":"ICML.cc/2025/Conference","replyto":"WOyOtaO6lQ","id":"HFLZTzNUSa","forumContent":{"venue":{"value":"ICML 2025 poster"},"TLDR":{"value":"Cake is a KV cache loading system that optimally balances computation and I/O resources through a bidirectional scheduling strategy, reducing Time to First Token (TTFT) by 2.6× on average and improving long-context LLM inference efficiency."},"keywords":{"value":["Prefix caching System Optimization","KV Cache","Low-Latency LLM Inference"]},"primary_area":{"value":"general_machine_learning->hardware_and_software"},"abstract":{"value":"Large Language Models (LLMs) are increasingly deployed in large-scale online services, enabling sophisticated applications. However, the computational overhead of generating key-value (KV) caches in the prefill stage presents a major bottleneck, particularly for long-context inputs. Prefix caching mitigates this issue by storing KV caches for reuse, reducing redundant computation. Despite its advantages, prefix caching suffers from high latency due to the limited I/O bandwidth of storage devices, constraining inference efficiency. To address this challenge, we introduce Cake, a novel KV cache loading system that optimally utilizes both computational and I/O resources in parallel. Cake employs a bidirectional scheduling strategy that dynamically balances KV cache computation and loading, ensuring efficient resource utilization. Additionally, Cake incorporates an adaptive scheduling mechanism that seamlessly integrates with non-prefix caching requests, improving system throughput and adapting to fluctuating resource availabilty. Through extensive evaluations across various hardware configurations, datasets, and storage conditions, Cake achieves on average 2.6× reduction in Time to First Token (TTFT) compared to compute-only and I/O-only methods. Our findings highlight Cake as an effective and practical solution for optimizing long-context LLM inference, bridging the gap between computation and I/O efficiency in large-scale AI deployments."},"_bibtex":{"value":"@inproceedings{\njin2025compute,\ntitle={Compute or Load {KV} Cache? Why Not Both?},\nauthor={Shuowei Jin and Xueshen Liu and Qingzhao Zhang and Zhuoqing Mao},\nbooktitle={Forty-second International Conference on Machine Learning},\nyear={2025},\nurl={https://openreview.net/forum?id=WOyOtaO6lQ}\n}"},"title":{"value":"Compute or Load KV Cache? Why Not Both?"},"pdf":{"value":"/pdf/367a534dac0cfcac16a8506794609c1bc6a4a15c.pdf"},"lay_summary":{"value":"Large Language Models (LLMs) have become vital tools in various online applications, helping users with tasks like answering questions, summarizing documents, and chatting naturally. However, using these models with lengthy text inputs (like an entire book or a long conversation) can be slow, as they need to perform extensive calculations before providing their first response.\n\nPrevious approaches typically relied solely on loading pre-calculated data from storage, but these methods alone have limitations and often fail to sufficiently reduce the latency experienced by users, especially when storage bandwidth is limited. In this work, we introduce a new approach called \"Cake,\" which smartly combines recomputing calculations and loading pre-stored data simultaneously. Cake efficiently manages resources by recomputing certain data while loading other parts from storage, significantly reducing the waiting time even under constrained storage bandwidth.\n\nBy dynamically adapting to available computing power and storage bandwidth, Cake effectively handles real-world conditions, enhancing the responsiveness of LLMs. Our extensive experiments demonstrate that Cake consistently accelerates the initial response time, making large language models faster and more practical for everyday applications."},"venueid":{"value":"ICML.cc/2025/Conference"},"paperhash":{"value":"jin|compute_or_load_kv_cache_why_not_both"},"authorids":{"value":["~Shuowei_Jin1","~Xueshen_Liu1","~Qingzhao_Zhang1","~Zhuoqing_Mao1"]},"authors":{"value":["Shuowei Jin","Xueshen Liu","Qingzhao Zhang","Zhuoqing Mao"]}},"version":2},{"content":{"summary":{"value":"This paper proposes SGLANG-LSM, an LSM-tree–backed disk KV-cache store for SGLang that replaces the current file-per-object backend, aiming to improve scalability and TTFT via a prefix-preserving index + tensor-log separation and an adaptive compaction controller. \n\nHowever, reviewers raised several strong concerns that remain decisive:\n* the “10× scalability” framing appears unsupported; \n* the choice of LSM (vs simpler/indexed append-log or read-optimized structures) is not convincingly justified; \n* it is unclear why cache hit rate increases if the eviction/replacement policy is unchanged; \n* the evaluation uses weak/outdated baselines while the community already has multi-tier KV cache layers (e.g., LMCache, AIBrix KVCache)."},"reviewer_concerns":{"value":"There is no rebuttal, so the main reviewer concerns are still outstanding:\n\n* Over-claiming / misleading framing: the title “10× Better Scalability”. \n* LSM motivation is weak: reviewers questioned why an LSM-tree (write-optimized) is the right tool when the serving goal is fast reads, and why alternatives (e.g., B-trees / simpler index+log approaches) are not considered. \n* Cache-hit definition / source of gains is unclear:\n* Evaluation is narrow + baselines too weak: experiments are single-node, synthetic, and mainly compare against SGLang(file)/HiCacheFile"},"reviewer_scores":{"value":"N/A, since no rebuttal is provided.."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Meta_Review","nonreaders":[],"tmdate":1770619556721,"tcdate":1768200993232,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission17124/Senior_Area_Chairs","ICLR.cc/2026/Conference/Submission17124/Area_Chair_HR1k"],"signatures":["ICLR.cc/2026/Conference/Submission17124/Area_Chair_HR1k"],"forum":"QqpbjtPbIc","number":1,"license":"CC BY 4.0","cdate":1768200993232,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission17124/-/Meta_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1770619556721,"domain":"ICLR.cc/2026/Conference","replyto":"QqpbjtPbIc","id":"2efW1LoIyX","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["LLM serving","database"]},"primary_area":{"value":"infrastructure, software libraries, hardware, systems, etc."},"abstract":{"value":"Large language models (LLMs) rely on Key-Value (KV) cache to reduce time-\nto-first-token (TTFT) latency, but existing disk-based KV cache systems using\nfile-per-object layouts suffer from severe scalability bottlenecks due to file system\nmetadata overhead, I/O inefficiency, and poor spatial locality. This paper presents\nSGLANG-LSM, a database-inspired system that leverages Log-Structured Merge-\ntree (LSM-tree) architectures for scalable KV cache management. SGLANG-LSM\nimplements a layered system design with three coordinated components: (1) a\nprefix-preserving storage engine that maintains token sequence locality while\nefficiently storing large KV cache tensors through key-value separation, (2) an\nadaptive controller that dynamically optimizes LSM-tree configurations based on\nshifting workload characteristics, and (3) runtime services including batch opera-\ntions and automatic resource management for production deployment. Evaluation\non large-scale dynamic workloads demonstrates that SSGLANG-LSM significantly\nimproves cache hits by up to 143% and reduces TTFT by up to 24% compared to\nstate-of-the-art systems, representing the first systematic application of database\nstorage architectures to large-scale LLM cache management."},"_bibtex":{"value":"@misc{\nyu2026on,\ntitle={On 10X Better Scalability: {KV} Stores Scale Up {KV} Cache},\nauthor={Weiping Yu and Ye Jiarui and He Mengke and Junfeng Liu and Siqiang Luo},\nyear={2026},\nurl={https://openreview.net/forum?id=QqpbjtPbIc}\n}"},"title":{"value":"On 10X Better Scalability: KV Stores Scale Up KV Cache"},"pdf":{"value":"/pdf/6ab6f1d8bd3e7b9071c53623817244d7a366ffec.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"yu|on_10x_better_scalability_kv_stores_scale_up_kv_cache"},"authorids":{"value":["~Weiping_Yu1","~Ye_Jiarui1","~He_Mengke1","~Junfeng_Liu6","~Siqiang_Luo1"]},"authors":{"value":["Weiping Yu","Ye Jiarui","He Mengke","Junfeng Liu","Siqiang Luo"]}},"version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2508.18572v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"xie|strata_hierarchical_context_caching_for_long_context_language_model_serving"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Zhiqiang_Xie:","https://dblp.org/search/pid/api?q=author:Ziyi_Xu:","https://dblp.org/search/pid/api?q=author:Mark_Zhao:","https://dblp.org/search/pid/api?q=author:Yuwei_An:","~Vikram_Sharma_Mailthody1","https://dblp.org/search/pid/api?q=author:Scott_A._Mahlke:","https://dblp.org/search/pid/api?q=author:Michael_Garland:","https://dblp.org/search/pid/api?q=author:Christos_Kozyrakis:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2508.18572"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2508-18572,\n publtype={informal},\n author={Zhiqiang Xie and Ziyi Xu and Mark Zhao and Yuwei An and Vikram Sharma Mailthody and Scott A. Mahlke and Michael Garland and Christos Kozyrakis},\n title={Strata: Hierarchical Context Caching for Long Context Language Model Serving},\n year={2025},\n month={August},\n cdate={1754006400000},\n journal={CoRR},\n volume={abs/2508.18572},\n url={https://doi.org/10.48550/arXiv.2508.18572}\n}\n"},"abstract":{"value":"Large Language Models (LLMs) with expanding context windows face significant performance hurdles. While caching key-value (KV) states is critical for avoiding redundant computation, the storage footprint of long-context caches quickly exceeds GPU memory capacity, forcing production systems to adopt hierarchical caching across memory hierarchies. However, transferring large cached contexts back to the GPU introduces severe performance bottlenecks: fragmented I/O from paged layouts prevents full bandwidth utilization, and existing schedulers fail to account for cache-loading delays, leaving systems loading-bound rather than compute-bound. We present Strata, a hierarchical context caching framework designed for efficient long context LLM serving. Strata introduces GPU-assisted I/O to combat KV cache fragmentation, decoupling GPU and CPU memory layouts and employs cache-aware request scheduling to balance compute with I/O latency and overlapping unavoidable stalls with complementary tasks. Built on SGLang and deployed in production, Strata achieves up to 5x lower Time-To-First-Token (TTFT) compared to vLLM + LMCache and 3.75x speedup over NVIDIA TensorRT-LLM on long-context benchmarks, without degrading short-context performance."},"title":{"value":"Strata: Hierarchical Context Caching for Long Context Language Model Serving"},"authors":{"value":["Zhiqiang Xie","Ziyi Xu","Mark Zhao","Yuwei An","Vikram Sharma Mailthody","Scott A. Mahlke","Michael Garland","Christos Kozyrakis"]}},"tmdate":1767629580740,"pdate":1735689600000,"externalIds":["dblp:journals/corr/abs-2508-18572"],"tcdate":1767629575784,"writers":["~"],"signatures":["~Vikram_Sharma_Mailthody1"],"forum":"SE70USzeEf","license":"CC BY-SA 4.0","number":716478,"cdate":1754006400000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1767629580740,"domain":"DBLP.org","id":"SE70USzeEf","version":2},{"content":{"summary":{"value":"In this paper, the authors introduce SGLANG-LSM, a database-inspired system that applies Log-Structured Merge-tree (LSM-tree) storage architectures to manage large language model (LLM) Key-Value (KV) caches) at scale.\nTraditional disk-based KV cache systems (e.g., in SGLang), each token’s cache tensor is stored as a separate file. This approach leads to scalability issues due to metadata overhead, I/O inefficiency, and poor spatial locality. \nTo address this, the authors design a prefix-preserving LSM storage engine that maintains token sequence locality while using key-value separation to efficiently store large tensors (start tensor files as a log).\nAn adaptive controller dynamically tunes LSM parameters such as compaction and size ratios based on changing workload patterns, balancing performance between read- and write-heavy phases. Complementary runtime services further enhance scalability through batch operations, tensor compression, and automatic file merging. Experiments across GLM-4-8B, GLM-4-32B, and Llama-3-8B models show up to 143% higher cache hit rates and 24% lower time-to-first-token latency."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"Please see weakness."},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":1},"contribution":{"value":2},"strengths":{"value":"S1. The paper presents an interesting integration of LSM-tree storage architectures (e.g., RocksDB) into LLM serving (e.g., SGLang). This cross-domain design bridges disk storage and AI systems for improved scalability.\n\nS2. The experiments results seems to be good;"},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"This paper raises a lot of questions about the design choices and evaluation scope of SGLANG-LSM despite its strong technical contributions.\n\nW1. Since the LSM-tree design is primarily optimized for high write throughput, it is unclear why it was chosen over read-optimized alternatives such as B-trees, especially when the main goal of LLM serving is to accelerate cache retrieval. The authors should better motivate why enabling high write throughput is critical for this workload.\n\nW2. The paper proposes a tensor-log append-only storage layer with periodic merging, yet this design might reintroduce the same spatial locality issues that the authors attribute to file-per-object layouts.\n\nW3. It is unclear how SGLANG-LSM achieves substantially higher cache hit rates than existing approaches when the cache eviction policy remains unchanged. The authors should explicitly define the notion of “cache hit” and explain whether structural improvements alone can increase it without modifying replacement strategies.\n\nW4. The evaluation primarily compares against SGLang HiCacheFile, which the SGLang authors describe it as a simple file-based backend. To strengthen claims of scalability and performance, the system should be compared with stronger baselines such as AIbrix KVCache or LMCache, which are more representative of current production systems."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762927120531,"tcdate":1761974342437,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission17124/Reviewer_5yY9"],"signatures":["ICLR.cc/2026/Conference/Submission17124/Reviewer_5yY9"],"forum":"QqpbjtPbIc","number":3,"license":"CC BY 4.0","cdate":1761974342437,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission17124/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762927120531,"domain":"ICLR.cc/2026/Conference","replyto":"QqpbjtPbIc","id":"IUf5k5AQbA","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["LLM serving","database"]},"primary_area":{"value":"infrastructure, software libraries, hardware, systems, etc."},"abstract":{"value":"Large language models (LLMs) rely on Key-Value (KV) cache to reduce time-\nto-first-token (TTFT) latency, but existing disk-based KV cache systems using\nfile-per-object layouts suffer from severe scalability bottlenecks due to file system\nmetadata overhead, I/O inefficiency, and poor spatial locality. This paper presents\nSGLANG-LSM, a database-inspired system that leverages Log-Structured Merge-\ntree (LSM-tree) architectures for scalable KV cache management. SGLANG-LSM\nimplements a layered system design with three coordinated components: (1) a\nprefix-preserving storage engine that maintains token sequence locality while\nefficiently storing large KV cache tensors through key-value separation, (2) an\nadaptive controller that dynamically optimizes LSM-tree configurations based on\nshifting workload characteristics, and (3) runtime services including batch opera-\ntions and automatic resource management for production deployment. Evaluation\non large-scale dynamic workloads demonstrates that SSGLANG-LSM significantly\nimproves cache hits by up to 143% and reduces TTFT by up to 24% compared to\nstate-of-the-art systems, representing the first systematic application of database\nstorage architectures to large-scale LLM cache management."},"_bibtex":{"value":"@misc{\nyu2026on,\ntitle={On 10X Better Scalability: {KV} Stores Scale Up {KV} Cache},\nauthor={Weiping Yu and Ye Jiarui and He Mengke and Junfeng Liu and Siqiang Luo},\nyear={2026},\nurl={https://openreview.net/forum?id=QqpbjtPbIc}\n}"},"title":{"value":"On 10X Better Scalability: KV Stores Scale Up KV Cache"},"pdf":{"value":"/pdf/6ab6f1d8bd3e7b9071c53623817244d7a366ffec.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"yu|on_10x_better_scalability_kv_stores_scale_up_kv_cache"},"authorids":{"value":["~Weiping_Yu1","~Ye_Jiarui1","~He_Mengke1","~Junfeng_Liu6","~Siqiang_Luo1"]},"authors":{"value":["Weiping Yu","Ye Jiarui","He Mengke","Junfeng Liu","Siqiang Luo"]}},"version":2},{"content":{"TLDR":{"value":"HarmonyTransfer introduces Agent Prefill Group abstraction for fan-out requests sharing a collective prefill milestone, prioritizing APGs by remaining transfer and prefill demand to reduce mean completion time and improve deadline attainment."},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["LLM Inference","LLM Agents","KVCache Transfer","Hierarchical Caching","Scheduling"]},"primary_area":{"value":"infrastructure, software libraries, hardware, systems, etc."},"abstract":{"value":"LLM agent applications often contain fan-out stages whose completion depends on\nmultiple requests. Such a stage reaches collective prefill completion only\nwhen every required request has returned its first token.\nServing long-context fan-out stages efficiently requires moving the reusable KVCache from lower-tier storage back onto the GPU. That transfer runs over a\nlink shared by every live stage.\nExisting systems, however,\nschedule KVCache transfers independently for each request, without accounting\nfor the stage's joint completion semantics. We introduce the Agent Prefill Group (APG), a first-class abstraction\nrepresenting the set of requests whose first-token completion jointly\ndetermines a stage's collective prefill completion, and present\nHarmonyTransfer, an APG-aware KVCache transfer scheduling system.\nHarmonyTransfer implements objective-specific APG scheduling policies for\nreducing mean completion time and improving deadline attainment. It dispatches\nKVCache transfers in bounded chunk-level batches, allowing the scheduler to\nmake a new APG scheduling decision at each batch boundary rather than only at\nfull-request transfer completion. Built on LMCache and vLLM, HarmonyTransfer\nreduces mean APG completion time by up to 51.6\\% over the best-performing\nbaseline and improves APG deadline attainment by up to 23.0 percentage points\nin testbed experiments."},"_bibtex":{"value":"@inproceedings{\nanonymous2026harmonytransfer,\ntitle={HarmonyTransfer: {KVC}ache Transfer Scheduling for Efficient {LLM} Agent Serving},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=K8qmVP2B8Q},\nnote={under review}\n}"},"title":{"value":"HarmonyTransfer: KVCache Transfer Scheduling for Efficient LLM Agent Serving"},"pdf":{"value":"/pdf/2f553f8a43aac5192c60ca803d983d6bcf4709dd.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791230741353,"tcdate":1789607860820,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission27395/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission27395/Authors"],"forum":"K8qmVP2B8Q","license":"CC BY 4.0","number":27395,"cdate":1789607860820,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission27395/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791230741353,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"K8qmVP2B8Q","version":2},{"content":{"comment":{"value":"Thank you for the detailed review. Locations for each request:\n\n- *Enforced vs. write-through explicit throughout (esp. abstract/headline).* The abstract now states the SGLang/LMCache/vLLM overlays measure compatibility and controller overhead, not acceleration; §5.3 and Table G.1 carry the same labels per experiment.\n- *Stronger correctness validation of the splice (token/logit with correct position/mask), or separate perf from correctness.* We did both: a new explicit-state continuation check (full mask, logical positions, `cache_position`, float16, SDPA) agrees on 956/960 decoded tokens (Qwen/Gemma 192/192), reported separately in §5.2.2 and Appendix E; the custom-HF timings stay tied to their stated agreement rule, with the Qwen disagreement preserved and no universal-bitwise-equivalence claim.\n- *Stronger admission/eviction baselines (recency/frequency).* Added LRU, LFU, FIFO, and admit-all under the same Gemma-4-31B trace (§5.3.1, Table J.5); MeritKV retains +28.2 / +15.2 pp recovery over LFU with fewer evictions, and we label LRU/LFU as admit-all retention policies to isolate how much comes from the utility model.\n- *Keep approximate semantic reuse outside the deployable contribution unless a correctness guard is added.* §3 and Appendix H state it is unsafe without an independent correctness guard; the divergence sandbox never enters the decision path, so we do not present it as such a guard.\n\nWe also added the enforced native-backend evidence noted during discussion (App. G.1 / Table G.2, and App. J.3 / Tables J.6–J.7), reported with their limits."}},"parentInvitations":"TMLR/-/Official_Comment","tmdate":1787750183960,"tcdate":1787750183960,"writers":["TMLR","TMLR/Paper11074/Authors"],"signatures":["TMLR/Paper11074/Authors"],"forum":"y3j6fucWQP","number":11,"license":"CC BY 4.0","cdate":1787750183960,"readers":["everyone"],"invitations":["TMLR/Paper11074/-/Official_Comment"],"mdate":1787750183960,"domain":"TMLR","replyto":"ipY0KwlCRI","id":"pALGMQs2jN","forumContent":{"submission_length":{"value":"Regular submission (no more than 12 pages of main content)"},"venue":{"value":"Accepted by TMLR"},"code":{"value":"https://github.com/Kushalk0677/meritkv"},"abstract":{"value":"Key-value (KV) cache reuse lowers prefill latency, but lookup, cache manipulation, and speculative precompute can cost more than the computation they save, so a serving system can be slower than no cache even at a high hit rate. We argue that KV reuse should be a per-request economic decision rather than an unconditional mode, and present MeritKV, a lightweight admission-control layer that scores each candidate reuse by its net utility (saved prefill time minus reuse cost minus expected waste) and executes it only when that estimate is positive, bypassing otherwise. The gate is backend-agnostic and composes on top of existing prefix caches, while a single waste-feedback loop makes admission self-correcting as workloads change. Across controlled studies on five Hugging Face models (GPT-2, TinyLlama, Qwen2.5, Gemma, and Phi-3; 124M–3.8B parameters) and ten datasets, MeritKV matches no-cache performance on low-reuse traffic (0.99–1.01×) and observes 1.31–1.61× in custom-HF diagnostic configurations on structured prompts. In a 360-cell admission-policy comparison, it improves speedup from 1.085× to 1.110× while reducing the waste ratio from 24.0% to 5.6% relative to MeritKV-Sem, the reactive-plus-speculative variant without the policy controller. We additionally evaluate twelve models up to 32B and write-through SGLang, LMCache, and vLLM overlays, which establish compatibility and controller overhead, not acceleration. Under capacity pressure, five-seed native SGLang traces on Gemma-4-31B and Qwen2.5-32B show skip-write admission eliminating measured evictions and reducing mean latency/energy by 6.8–7.7%/7.6–8.5% versus native admit-all LRU; Qwen P95 regresses by 1.65% versus write-through, and native LFU remains stronger on that trace. A controlled multi-round trace reduces latency by 11.6–16.7% and improves recovery by 41–61 percentage points versus write-through; recovery remains 15.2–28.2 percentage points above controlled-bank LFU."},"_bibtex":{"value":"@article{\nkhemani2026meritkv,\ntitle={Merit{KV}: Utility-Gated Admission Control for {KV}-Cache Reuse in {LLM} Serving},\nauthor={Kushal Khemani and Evan J. Leri and Sparsh Mittal},\njournal={Transactions on Machine Learning Research},\nissn={2835-8856},\nyear={2026},\nurl={https://openreview.net/forum?id=y3j6fucWQP},\nnote={}\n}"},"video":{"value":"https://www.youtube.com/watch?v=X5VAFEl17zE"},"title":{"value":"MeritKV: Utility-Gated Admission Control for KV-Cache Reuse in LLM Serving"},"changes_since_last_submission":{"value":"Uploading the deanonymized camera-ready version, including author details, affiliations, acknowledgements, declarations, and the public code link, along with minor editorial and reference corrections."},"pdf":{"value":"/pdf/204645d44d3e621a14a925967acfa5e1b573cab6.pdf"},"venueid":{"value":"TMLR"},"paperhash":{"value":"khemani|meritkv_utilitygated_admission_control_for_kvcache_reuse_in_llm_serving"},"authorids":{"value":["~Kushal_Khemani1","~Evan_J._Leri1","~Sparsh_Mittal1"]},"assigned_action_editor":{"value":"~Binhang_Yuan1"},"authors":{"value":["Kushal Khemani","Evan J. Leri","Sparsh Mittal"]}},"version":2},{"content":{"comment":{"value":"Thank you again. The revision is uploaded; locations for the points raised:\n\n**Critical.**\n- *Abstract states production results are write-through only.* The abstract now says the SGLang/LMCache/vLLM overlays \"establish compatibility and controller overhead, not acceleration.\"\n- *Semantic reuse is unsafe.* Stated in §3 (execution boundary) and Appendix H: approximate semantic reuse is unsafe without an independent correctness guard and is excluded from the deployable contribution.\n\n**Would strengthen.**\n- *Stronger baselines than FIFO.* Added five-seed admit-all, FIFO, LRU, LFU, and MeritKV under the same Gemma-4-31B trace (§5.3.1, Table J.5). Against LFU, MeritKV keeps +28.2 / +15.2 pp final-round recovery and 18.0 / 11.4 fewer evictions at 28 / 56 GB; we identify LRU/LFU as admit-all retention policies and note that LFU recovers much of the advantage.\n- *Real workload traces.* We added native-backend traces (App. J.3) but continue to describe every capacity-pressure workload as a controlled trace, not a production request log.\n- *Fix the splice.* The custom path is now scoped as diagnostic; the explicit-state check agrees on 956/960 tokens (Qwen/Gemma 192/192), reported separately (§5.2.2, App. E). Qwen splice speedups remain excluded.\n- *Define waste terms.* §3–§4 and Appendices B–C now distinguish the dimensionless waste ratio W, the millisecond expected-waste cost \\hat{C}_w = W·C_spec, the storage/displacement cost, and the historical EWMA conservatism signal (no longer described as a causal cost of consuming a resident hit)."}},"parentInvitations":"TMLR/-/Official_Comment","tmdate":1787750052047,"tcdate":1787750052047,"writers":["TMLR","TMLR/Paper11074/Authors"],"signatures":["TMLR/Paper11074/Authors"],"forum":"y3j6fucWQP","number":9,"license":"CC BY 4.0","cdate":1787750052047,"readers":["everyone"],"invitations":["TMLR/Paper11074/-/Official_Comment"],"mdate":1787750052047,"domain":"TMLR","replyto":"F4BIznqllA","id":"0khD44kwgc","forumContent":{"submission_length":{"value":"Regular submission (no more than 12 pages of main content)"},"venue":{"value":"Accepted by TMLR"},"code":{"value":"https://github.com/Kushalk0677/meritkv"},"abstract":{"value":"Key-value (KV) cache reuse lowers prefill latency, but lookup, cache manipulation, and speculative precompute can cost more than the computation they save, so a serving system can be slower than no cache even at a high hit rate. We argue that KV reuse should be a per-request economic decision rather than an unconditional mode, and present MeritKV, a lightweight admission-control layer that scores each candidate reuse by its net utility (saved prefill time minus reuse cost minus expected waste) and executes it only when that estimate is positive, bypassing otherwise. The gate is backend-agnostic and composes on top of existing prefix caches, while a single waste-feedback loop makes admission self-correcting as workloads change. Across controlled studies on five Hugging Face models (GPT-2, TinyLlama, Qwen2.5, Gemma, and Phi-3; 124M–3.8B parameters) and ten datasets, MeritKV matches no-cache performance on low-reuse traffic (0.99–1.01×) and observes 1.31–1.61× in custom-HF diagnostic configurations on structured prompts. In a 360-cell admission-policy comparison, it improves speedup from 1.085× to 1.110× while reducing the waste ratio from 24.0% to 5.6% relative to MeritKV-Sem, the reactive-plus-speculative variant without the policy controller. We additionally evaluate twelve models up to 32B and write-through SGLang, LMCache, and vLLM overlays, which establish compatibility and controller overhead, not acceleration. Under capacity pressure, five-seed native SGLang traces on Gemma-4-31B and Qwen2.5-32B show skip-write admission eliminating measured evictions and reducing mean latency/energy by 6.8–7.7%/7.6–8.5% versus native admit-all LRU; Qwen P95 regresses by 1.65% versus write-through, and native LFU remains stronger on that trace. A controlled multi-round trace reduces latency by 11.6–16.7% and improves recovery by 41–61 percentage points versus write-through; recovery remains 15.2–28.2 percentage points above controlled-bank LFU."},"_bibtex":{"value":"@article{\nkhemani2026meritkv,\ntitle={Merit{KV}: Utility-Gated Admission Control for {KV}-Cache Reuse in {LLM} Serving},\nauthor={Kushal Khemani and Evan J. Leri and Sparsh Mittal},\njournal={Transactions on Machine Learning Research},\nissn={2835-8856},\nyear={2026},\nurl={https://openreview.net/forum?id=y3j6fucWQP},\nnote={}\n}"},"video":{"value":"https://www.youtube.com/watch?v=X5VAFEl17zE"},"title":{"value":"MeritKV: Utility-Gated Admission Control for KV-Cache Reuse in LLM Serving"},"changes_since_last_submission":{"value":"Uploading the deanonymized camera-ready version, including author details, affiliations, acknowledgements, declarations, and the public code link, along with minor editorial and reference corrections."},"pdf":{"value":"/pdf/204645d44d3e621a14a925967acfa5e1b573cab6.pdf"},"venueid":{"value":"TMLR"},"paperhash":{"value":"khemani|meritkv_utilitygated_admission_control_for_kvcache_reuse_in_llm_serving"},"authorids":{"value":["~Kushal_Khemani1","~Evan_J._Leri1","~Sparsh_Mittal1"]},"assigned_action_editor":{"value":"~Binhang_Yuan1"},"authors":{"value":["Kushal Khemani","Evan J. Leri","Sparsh Mittal"]}},"version":2},{"content":{"review":{"value":"**Recommendation: Weak accept.** The research question is relevant, but the contribution and results are limited.\n\n**Summary**\n\nThe paper proposes Localized TabICLv2, a method to reduce the inference cost of TabICLv2. Instead of letting each test point attend to the full training set during in-context learning, it retrieves only the most similar training rows per test point and runs ICL on that smaller local context. The main novelty is that the retrieval is performed in the model's learned row representation space rather than in the raw input feature space, which the authors show produces better neighbours for the downstream ICL task. They additionally fine-tune the relevant stages of the model to better adapt to this localized setting. They evaluate on TabArena and report that the localized model retains most of the accuracy of the full model while achieving a substantial speedup, especially in single-query serving.\n\n**Strengths**\n\n- The research question is relevant: inference cost is a real bottleneck for tabular foundation models, and any work that makes them more practical to deploy is useful to the community.\n- The evaluation follows the standard TabArena protocol, which makes the numbers easy to interpret and compare against other work.\n\n**Weaknesses**\n\n- The method is essentially a port of an existing LLM technique (k-NN retrieval to limit attention context) to TabICLv2. The contribution is mostly in the integration rather than in a new mechanism.\n- The scope is narrow: the method is tied to the specific three-stage architecture of TabICLv2, which limits broader applicability.\n- The reported improvement from changing the retrieval space (raw features → learned row embeddings) is marginal, which weakens the main contribution.\n\n**Suggestions**\n\n- A comparison against a random-k baseline (e.g. k=32 randomly drawn training rows) is missing. Without this, it is hard to tell whether the accuracy retention comes from the k-NN retrieval being meaningfully informative or simply from the ICL head being robust to small context sizes. This baseline is cheap to add and would directly support the central claim of the paper.\n- For a fair latency comparison, you might also want to compare against persisting KV-caches to disk and loading them on demand. The current text of \"Limited-Query Latency\" suggests the full attention is computed for every prediction call instead of the KV-cache being loaded. KV-cache disk offloading is now standard in production LLM serving (e.g. vLLM with LMCache, llm-d), where caches survive restarts and are reused across queries. This would be a more realistic deployment baseline for the single-query serving setting.\n- If you are already porting LLM techniques to speed up inference, distilling TabICLv2 into a smaller non-ICL variant is a natural secondary direction worth exploring. It would be informative to know whether retrieval-based localization gives more than distillation for the same accuracy budget."},"confidence":{"value":4},"rating":{"value":6},"title":{"value":"Review of Localized TabICLv2: Scaling Tabular In-Context Learning through k-NN"}},"parentInvitations":"ICML.cc/2026/Workshop/FMSD/-/Official_Review","nonreaders":[],"tmdate":1780077389070,"tcdate":1779299971987,"writers":["ICML.cc/2026/Workshop/FMSD","ICML.cc/2026/Workshop/FMSD/Submission122/Reviewer_T2hJ"],"signatures":["ICML.cc/2026/Workshop/FMSD/Submission122/Reviewer_T2hJ"],"forum":"ddITrUyMTB","number":1,"license":"CC BY 4.0","cdate":1779299971987,"readers":["everyone"],"invitations":["ICML.cc/2026/Workshop/FMSD/Submission122/-/Official_Review","ICML.cc/2026/Workshop/FMSD/-/Edit"],"mdate":1780077389070,"domain":"ICML.cc/2026/Workshop/FMSD","replyto":"ddITrUyMTB","id":"5RC41MR6ax","forumContent":{"TLDR":{"value":"Localized TabICLv2 leverages TabICLv2's learned representation space to retrieve the most relevant training examples for each query, retaining 98.64% of Full TabICLv2 accuracy while substantially reducing inference latency."},"venue":{"value":"FMSD @ ICML 2026 Poster"},"pdf":{"value":"/pdf/ca6dd9c8b3506744dad6be9161eb958e69c6b20d.pdf"},"keywords":{"value":["Structured Data","Tabular Foundation Models","In-Context Learning","Retrieval-Augmented Inference","Efficient Inference"]},"venueid":{"value":"ICML.cc/2026/Workshop/FMSD"},"paperhash":{"value":"guta|localized_tabiclv2_scaling_tabular_incontext_learning_through_knn"},"authorids":{"value":["~Beimnet_Bekele_Guta2"]},"abstract":{"value":"Foundational models for tabular data have made significant progress in recent years, with TabICLv2 reporting state-of-the-art performance on several tabular classification tasks. However, full-context tabular ICL still suffers from attention cost that grows with the training-context size, which limits its ability to handle large datasets efficiently. Localized TabICLv2 introduces a method that reduces the inference cost of TabICLv2 by retrieving only the \\(k\\) nearest training neighbours for each test point, measured by similarity in the model's Stage 2 row-representation space, rather than using the full training context. This requires no architectural changes, and we show that accuracy retention can be improved through additional Stage 2 and Stage 3 fine-tuning. On TabArena classification tasks, the fine-tuned localized model retains 98.64\\% of Full TabICLv2 accuracy and it achieves a median 2.18$\\times$ speedup in batch inference, and reaches approximately 249$\\times$ median speedup in the single-query serving setting."},"_bibtex":{"value":"@inproceedings{\nguta2026localized,\ntitle={Localized Tab{ICL}v2: Scaling Tabular In-Context Learning through k-{NN}},\nauthor={Beimnet Bekele Guta},\nbooktitle={2nd ICML Workshop on Foundation Models for Structured Data},\nyear={2026},\nurl={https://openreview.net/forum?id=ddITrUyMTB}\n}"},"title":{"value":"Localized TabICLv2: Scaling Tabular In-Context Learning through k-NN"},"authors":{"value":["Beimnet Bekele Guta"]}},"version":2},{"content":{"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["RAG","Cache","Graphs","VLMS","Foundation Models"]},"primary_area":{"value":"optimization"},"abstract":{"value":"The document order and meaning of the chunks required by a retrieval-augmented language model often show relationships. We have investigated whether these relationships can be used to prepare key--value (KV) cache states before a chunk is requested. We represent the chunks as a graph by including their top $M$ semantic neighbours as well as the structural links that exist within the document. For ranking the candidates, we use two-hop retrieval and select a number of chunks up to $K$. We compare three methods: fixed weights, a globally learned balance, and an online policy which adjusts the balance after observing ranking misses. The evaluation involves an archived simulated-cache study covering six model configurations, a real experiment carried out with Qwen2.5-1.5B using vLLM and LMCache, and answer-quality experiments with Gemini 3.5 Flash-Lite. For the real access trace derived from Hotpot at $K=10$, offline adaptation increases prediction recall from 80.50\\% to 95.33\\% and reduces the ready-cache time to the first token from 56.68 to 51.36\\,ms as compared to cosine prefetching. With synchronous candidate preparation, the corresponding times are 902.67 and 883.27\\,ms, whereas they are 73.41\\,ms when there is no prefetching. In the separate 60-question quality experiments, offline adaptation raises the HotpotQA F1 from 47.94 to 53.83, while cosine performs better on 2WikiMultiHopQA, with an F1 of 48.18 versus 40.53. Altogether, these results demonstrate the advantage of graph-guided prediction on the tested access trace and show that its impact on answer quality varies according to the dataset."},"_bibtex":{"value":"@inproceedings{\nanonymous2026graphguided,\ntitle={{GRAPH}-{GUIDED} {AND} {ADAPTIVE} {KV}-{CACHE} {PREFETCHING} {FOR} {RETRIEVAL}-{AUGMENTED} {GENERATION}},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=AoKpeNWp9o},\nnote={under review}\n}"},"title":{"value":"GRAPH-GUIDED AND ADAPTIVE KV-CACHE PREFETCHING FOR RETRIEVAL-AUGMENTED GENERATION"},"pdf":{"value":"/pdf/18396154015578a042f9c88eaaecba36cb9afbca.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791234201069,"tcdate":1789766564899,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission49056/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission49056/Authors"],"forum":"AoKpeNWp9o","license":"CC BY 4.0","number":49056,"cdate":1789766564899,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission49056/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791234201069,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"AoKpeNWp9o","version":2},{"content":{"comment":{"value":"Thank you for the careful and encouraging review, and for recognising the paper's measurement boundaries and negative results.\n\nWe agree that the enforced/write-through distinction should be impossible to miss. The revised abstract and headline discussion will say directly that the SGLang, LMCache, and broad vLLM results measure compatibility and controller overhead, not MeritKV-caused acceleration. This matches the scope already stated in the introduction, runtime section, and conclusion.\n\nFor exact-reuse validation, we ran a separate explicit-state continuation check using the complete attention mask, logical positions, cache_position, float16, and SDPA. The splice and equivalent native-cache continuation agree on 956/960 decoded tokens across five models; Qwen2.5 and Gemma each agree on all 192/192 tested tokens. We will report this separately from the original crop-and-replay test, keep the original Qwen disagreement visible, and avoid claiming universal bitwise equivalence. The custom-HF timings will remain tied to their stated agreement criterion.\n\nWe also ran LRU and LFU under the same Gemma-4-31B trace. Compared with LFU, MeritKV improves final-round recovery by 28.2 percentage points at 28 GB and 15.2 points at 56 GB, with 18.0 and 11.4 fewer evictions, respectively. We will identify LRU and LFU correctly as admit-all retention baselines, not selective-admission policies.\n\nThe main text already excludes approximate semantic reuse from the deployable contribution. We will make the reason explicit: it is unsafe without an independent correctness guard.\n\nWe are also preparing an enforced extension of the existing Blackwell write-through integration. If it is completed during the discussion period, we will report the paired enforced-versus-write-through result regardless of direction and limit the conclusion to that configuration. We will give exact manuscript locations when the revision is uploaded."}},"parentInvitations":"TMLR/-/Official_Comment","tmdate":1786711993811,"tcdate":1786711993811,"writers":["TMLR","TMLR/Paper11074/Authors"],"signatures":["TMLR/Paper11074/Authors"],"forum":"y3j6fucWQP","number":3,"license":"CC BY 4.0","cdate":1786711993811,"readers":["everyone"],"invitations":["TMLR/Paper11074/-/Official_Comment"],"mdate":1786711993811,"domain":"TMLR","replyto":"ipY0KwlCRI","id":"VTJpTF7H6Z","forumContent":{"submission_length":{"value":"Regular submission (no more than 12 pages of main content)"},"venue":{"value":"Accepted by TMLR"},"code":{"value":"https://github.com/Kushalk0677/meritkv"},"abstract":{"value":"Key-value (KV) cache reuse lowers prefill latency, but lookup, cache manipulation, and speculative precompute can cost more than the computation they save, so a serving system can be slower than no cache even at a high hit rate. We argue that KV reuse should be a per-request economic decision rather than an unconditional mode, and present MeritKV, a lightweight admission-control layer that scores each candidate reuse by its net utility (saved prefill time minus reuse cost minus expected waste) and executes it only when that estimate is positive, bypassing otherwise. The gate is backend-agnostic and composes on top of existing prefix caches, while a single waste-feedback loop makes admission self-correcting as workloads change. Across controlled studies on five Hugging Face models (GPT-2, TinyLlama, Qwen2.5, Gemma, and Phi-3; 124M–3.8B parameters) and ten datasets, MeritKV matches no-cache performance on low-reuse traffic (0.99–1.01×) and observes 1.31–1.61× in custom-HF diagnostic configurations on structured prompts. In a 360-cell admission-policy comparison, it improves speedup from 1.085× to 1.110× while reducing the waste ratio from 24.0% to 5.6% relative to MeritKV-Sem, the reactive-plus-speculative variant without the policy controller. We additionally evaluate twelve models up to 32B and write-through SGLang, LMCache, and vLLM overlays, which establish compatibility and controller overhead, not acceleration. Under capacity pressure, five-seed native SGLang traces on Gemma-4-31B and Qwen2.5-32B show skip-write admission eliminating measured evictions and reducing mean latency/energy by 6.8–7.7%/7.6–8.5% versus native admit-all LRU; Qwen P95 regresses by 1.65% versus write-through, and native LFU remains stronger on that trace. A controlled multi-round trace reduces latency by 11.6–16.7% and improves recovery by 41–61 percentage points versus write-through; recovery remains 15.2–28.2 percentage points above controlled-bank LFU."},"_bibtex":{"value":"@article{\nkhemani2026meritkv,\ntitle={Merit{KV}: Utility-Gated Admission Control for {KV}-Cache Reuse in {LLM} Serving},\nauthor={Kushal Khemani and Evan J. Leri and Sparsh Mittal},\njournal={Transactions on Machine Learning Research},\nissn={2835-8856},\nyear={2026},\nurl={https://openreview.net/forum?id=y3j6fucWQP},\nnote={}\n}"},"video":{"value":"https://www.youtube.com/watch?v=X5VAFEl17zE"},"title":{"value":"MeritKV: Utility-Gated Admission Control for KV-Cache Reuse in LLM Serving"},"changes_since_last_submission":{"value":"Uploading the deanonymized camera-ready version, including author details, affiliations, acknowledgements, declarations, and the public code link, along with minor editorial and reference corrections."},"pdf":{"value":"/pdf/204645d44d3e621a14a925967acfa5e1b573cab6.pdf"},"venueid":{"value":"TMLR"},"paperhash":{"value":"khemani|meritkv_utilitygated_admission_control_for_kvcache_reuse_in_llm_serving"},"authorids":{"value":["~Kushal_Khemani1","~Evan_J._Leri1","~Sparsh_Mittal1"]},"assigned_action_editor":{"value":"~Binhang_Yuan1"},"authors":{"value":["Kushal Khemani","Evan J. Leri","Sparsh Mittal"]}},"version":2},{"content":{"submission_length":{"value":"Regular submission (no more than 12 pages of main content)"},"venue":{"value":"Accepted by TMLR"},"code":{"value":"https://github.com/Kushalk0677/meritkv"},"abstract":{"value":"Key-value (KV) cache reuse lowers prefill latency, but lookup, cache manipulation, and speculative precompute can cost more than the computation they save, so a serving system can be slower than no cache even at a high hit rate. We argue that KV reuse should be a per-request economic decision rather than an unconditional mode, and present MeritKV, a lightweight admission-control layer that scores each candidate reuse by its net utility (saved prefill time minus reuse cost minus expected waste) and executes it only when that estimate is positive, bypassing otherwise. The gate is backend-agnostic and composes on top of existing prefix caches, while a single waste-feedback loop makes admission self-correcting as workloads change. Across controlled studies on five Hugging Face models (GPT-2, TinyLlama, Qwen2.5, Gemma, and Phi-3; 124M–3.8B parameters) and ten datasets, MeritKV matches no-cache performance on low-reuse traffic (0.99–1.01×) and observes 1.31–1.61× in custom-HF diagnostic configurations on structured prompts. In a 360-cell admission-policy comparison, it improves speedup from 1.085× to 1.110× while reducing the waste ratio from 24.0% to 5.6% relative to MeritKV-Sem, the reactive-plus-speculative variant without the policy controller. We additionally evaluate twelve models up to 32B and write-through SGLang, LMCache, and vLLM overlays, which establish compatibility and controller overhead, not acceleration. Under capacity pressure, five-seed native SGLang traces on Gemma-4-31B and Qwen2.5-32B show skip-write admission eliminating measured evictions and reducing mean latency/energy by 6.8–7.7%/7.6–8.5% versus native admit-all LRU; Qwen P95 regresses by 1.65% versus write-through, and native LFU remains stronger on that trace. A controlled multi-round trace reduces latency by 11.6–16.7% and improves recovery by 41–61 percentage points versus write-through; recovery remains 15.2–28.2 percentage points above controlled-bank LFU."},"_bibtex":{"value":"@article{\nkhemani2026meritkv,\ntitle={Merit{KV}: Utility-Gated Admission Control for {KV}-Cache Reuse in {LLM} Serving},\nauthor={Kushal Khemani and Evan J. Leri and Sparsh Mittal},\njournal={Transactions on Machine Learning Research},\nissn={2835-8856},\nyear={2026},\nurl={https://openreview.net/forum?id=y3j6fucWQP},\nnote={}\n}"},"video":{"value":"https://www.youtube.com/watch?v=X5VAFEl17zE"},"title":{"value":"MeritKV: Utility-Gated Admission Control for KV-Cache Reuse in LLM Serving"},"changes_since_last_submission":{"value":"Uploading the deanonymized camera-ready version, including author details, affiliations, acknowledgements, declarations, and the public code link, along with minor editorial and reference corrections."},"pdf":{"value":"/pdf/204645d44d3e621a14a925967acfa5e1b573cab6.pdf"},"venueid":{"value":"TMLR"},"paperhash":{"value":"khemani|meritkv_utilitygated_admission_control_for_kvcache_reuse_in_llm_serving"},"authorids":{"value":["~Kushal_Khemani1","~Evan_J._Leri1","~Sparsh_Mittal1"]},"assigned_action_editor":{"value":"~Binhang_Yuan1"},"authors":{"value":["Kushal Khemani","Evan J. Leri","Sparsh Mittal"]}},"tmdate":1790141021422,"pdate":1790141021317,"tcdate":1785835255322,"writers":["TMLR"],"signatures":["TMLR/Paper11074/Authors"],"forum":"y3j6fucWQP","number":11074,"license":"CC BY 4.0","cdate":1785835255322,"readers":["everyone"],"invitations":["TMLR/-/Submission","TMLR/-/Edit","TMLR/-/Under_Review","TMLR/Paper11074/-/Revision","TMLR/Paper11074/-/Camera_Ready_Revision","TMLR/-/Accepted"],"mdate":1790141021422,"odate":1785923402903,"domain":"TMLR","id":"y3j6fucWQP","version":2},{"content":{"comment":{"value":"We thank the reviewer for the careful reading and for flagging the scope, harness, and semantic-matcher concerns. The revision addresses all three; pointers below.\n\n**W1 / harness (Qwen2.5 float16, Fig. 2c).** We now treat the custom-HF crop-and-replay path as a controlled cross-model actuator, not a correctness claim: MeritKV contributes the policy, not the splice (§5.2.2). All Qwen custom-splice speedups remain excluded from performance claims, and the original disagreement stays visible. Separately, a new explicit-state continuation check (full attention mask, logical positions, `cache_position`, float16, SDPA) matches the equivalent native-cache continuation on 956/960 decoded tokens (192/192 for Qwen2.5 and Gemma), reported apart from the crop-and-replay test in §5.2.2 and Appendix E (E.4–E.6). We do not use it to relabel the custom-HF timings as exact-output-passing.\n\n**W2 / write-through vs. enforced gap (Table 5).** Agreed. This is now stated explicitly. The abstract, §5.3, and Table G.1 label every regime as enforced, write-through, or diagnostic; the broad SGLang/LMCache/vLLM overlays are described as compatibility and controller-overhead measurements, not MeritKV-caused acceleration.\n\n**W3 / semantic matcher.** Approximate semantic reuse is kept outside the deployable contribution and stated to be unsafe without an independent correctness guard (§3 boundary; Appendix H). It is retained only as a cautionary latency-only study, excluded from all derived results.\n\n**Optional (enforcement hooks).** We built and ran them. Two new native-backend experiments are added:\n- Native SGLang enforcement (App. G.1, Table G.2): a four-arm Blackwell experiment (native / write-through / enforced / forced-recompute) on Qwen2.5-32B and Gemma-4-31B. All 42 requested lookup skips and 42 write skips per model match the executed native counters (including Gemma's SWARadix counters); 414 cells / 102,912 requests pass the bundled verifier. Because only 50 of 12,800 requests per model are bypassed, enforced-vs-write-through mean latency moves by +0.197% (Qwen) / +0.144% (Gemma). We report this as physical enforcement with sparse-bypass parity, not acceleration.\n- Native storage admission (App. J.3, Tables J.6–J.7): five-seed capacity-pressure traces where skip-write admission removes measured evictions and reduces mean latency by 6.8–7.7% and energy by 7.6–8.5% vs. native admit-all LRU, with the limits stated (Qwen P95 +1.65% vs. write-through; native LFU stronger on the Qwen trace).\n\nOn open-sourcing: an anonymized artifact accompanies the submission, and the full codebase including the enforcement hooks will be released publicly upon acceptance."}},"parentInvitations":"TMLR/-/Official_Comment","tmdate":1787750002528,"tcdate":1787750002528,"writers":["TMLR","TMLR/Paper11074/Authors"],"signatures":["TMLR/Paper11074/Authors"],"forum":"y3j6fucWQP","number":8,"license":"CC BY 4.0","cdate":1787750002528,"readers":["everyone"],"invitations":["TMLR/Paper11074/-/Official_Comment"],"mdate":1787750002528,"domain":"TMLR","replyto":"TQiAFEtv9K","id":"hSwZ63UynH","forumContent":{"submission_length":{"value":"Regular submission (no more than 12 pages of main content)"},"venue":{"value":"Accepted by TMLR"},"code":{"value":"https://github.com/Kushalk0677/meritkv"},"abstract":{"value":"Key-value (KV) cache reuse lowers prefill latency, but lookup, cache manipulation, and speculative precompute can cost more than the computation they save, so a serving system can be slower than no cache even at a high hit rate. We argue that KV reuse should be a per-request economic decision rather than an unconditional mode, and present MeritKV, a lightweight admission-control layer that scores each candidate reuse by its net utility (saved prefill time minus reuse cost minus expected waste) and executes it only when that estimate is positive, bypassing otherwise. The gate is backend-agnostic and composes on top of existing prefix caches, while a single waste-feedback loop makes admission self-correcting as workloads change. Across controlled studies on five Hugging Face models (GPT-2, TinyLlama, Qwen2.5, Gemma, and Phi-3; 124M–3.8B parameters) and ten datasets, MeritKV matches no-cache performance on low-reuse traffic (0.99–1.01×) and observes 1.31–1.61× in custom-HF diagnostic configurations on structured prompts. In a 360-cell admission-policy comparison, it improves speedup from 1.085× to 1.110× while reducing the waste ratio from 24.0% to 5.6% relative to MeritKV-Sem, the reactive-plus-speculative variant without the policy controller. We additionally evaluate twelve models up to 32B and write-through SGLang, LMCache, and vLLM overlays, which establish compatibility and controller overhead, not acceleration. Under capacity pressure, five-seed native SGLang traces on Gemma-4-31B and Qwen2.5-32B show skip-write admission eliminating measured evictions and reducing mean latency/energy by 6.8–7.7%/7.6–8.5% versus native admit-all LRU; Qwen P95 regresses by 1.65% versus write-through, and native LFU remains stronger on that trace. A controlled multi-round trace reduces latency by 11.6–16.7% and improves recovery by 41–61 percentage points versus write-through; recovery remains 15.2–28.2 percentage points above controlled-bank LFU."},"_bibtex":{"value":"@article{\nkhemani2026meritkv,\ntitle={Merit{KV}: Utility-Gated Admission Control for {KV}-Cache Reuse in {LLM} Serving},\nauthor={Kushal Khemani and Evan J. Leri and Sparsh Mittal},\njournal={Transactions on Machine Learning Research},\nissn={2835-8856},\nyear={2026},\nurl={https://openreview.net/forum?id=y3j6fucWQP},\nnote={}\n}"},"video":{"value":"https://www.youtube.com/watch?v=X5VAFEl17zE"},"title":{"value":"MeritKV: Utility-Gated Admission Control for KV-Cache Reuse in LLM Serving"},"changes_since_last_submission":{"value":"Uploading the deanonymized camera-ready version, including author details, affiliations, acknowledgements, declarations, and the public code link, along with minor editorial and reference corrections."},"pdf":{"value":"/pdf/204645d44d3e621a14a925967acfa5e1b573cab6.pdf"},"venueid":{"value":"TMLR"},"paperhash":{"value":"khemani|meritkv_utilitygated_admission_control_for_kvcache_reuse_in_llm_serving"},"authorids":{"value":["~Kushal_Khemani1","~Evan_J._Leri1","~Sparsh_Mittal1"]},"assigned_action_editor":{"value":"~Binhang_Yuan1"},"authors":{"value":["Kushal Khemani","Evan J. Leri","Sparsh Mittal"]}},"version":2},{"content":{"comment":{"value":"**We sincerely thank the reviewer for the constructive feedback. We appreciate the recognition of our contributions and would like to address the concerns raised:**\n \n > ### **W1: Evaluations on more challenging RAG scenarios.**\n \nWe conducted experiments on two popular multi-hop QA datasets using Qwen3-4B and Qwen3-8B. These datasets require synthesizing information across multiple retrieved documents to answer complex queries. Results demonstrate that AdaCache consistently outperforms Full Recomputation, Prefix Cache, and CacheBlend across all datasets and models while maintaining comparable generation quality.\n\n| Dataset | Model | Method | EM | TTFT (s) |\n| ----------- |-----------|-----------------|--------|----------|\n| **2WikiMultihopQA** | **Qwen3-4B** | Full Recompute | 42.1 | 1.31 |\n| | | Prefix Caching | 42.1 | 1.16 |\n| | | CacheBlend | 41.5 | 0.74 |\n| | | **AdaCache (Ours)** | 41.9 | **0.63** |\n| | **Qwen3-8B** | Full Recompute | 45.4 | 1.84 |\n| | | Prefix Caching | 45.4 | 1.58 |\n| | | CacheBlend | 45.3 | 1.04 |\n| | | **AdaCache (Ours)** | 45.6 | **0.76** |\n| **HotpotQA** | **Qwen3-4B** | Full Recompute | 40.3 | 1.19 |\n| | | Prefix Caching | 40.3 | 1.11 |\n| | | CacheBlend | 39.8 | 0.73 |\n| | | **AdaCache (Ours)** | 40.1 | **0.68** |\n| | **Qwen3-8B** | Full Recompute | 42.7 | 1.86 |\n| | | Prefix Caching | 42.7 | 1.73 |\n| | | CacheBlend | 42.4 | 0.99 |\n| | | **AdaCache (Ours)** | 42.5 | **0.88** |\n \n \n> ### **W2: Applicability beyond open-weight models.**\n \nWe respectfully clarify that while AdaCache requires KV-cache manipulation, this does not limit its utility to open-weight models exclusively. Popular inference frameworks such as vLLM, SGLang, and LMCache all involve KV-cache management. AdaCache is a system-level optimization that LLM vendors can incorporate without exposing model weights. Closed-source providers can equally deploy AdaCache in their inference infrastructure to reduce serving costs and enhance service quality, just as they currently implement various KV-cache optimizations for efficiency.\n\n\n\n> ### **W3: Comparison with TurboRAG.**\n\nTurboRAG independently pre-computes KV caches for text chunks and reuses them during online serving. However, it breaks cross-chunk attention dependencies, necessitating model retraining on specially constructed datasets to adapt to sparse attention masks. This retraining process is both computationally expensive and time-consuming (e.g., the paper reports using 32 A100 GPUs). Furthermore, each new model architecture requires separate retraining, and without publicly released trained models, reproduction is challenging. \n\nIn contrast, AdaCache is **training-free** and introduces only minimal overhead from confidence computation and profiling. Importantly, our adaptive context augmentation module is orthogonal to TurboRAG's caching strategy, combining both could further reduce KV-cache loading cost.\n\n\n> ### **Q1: Integration with dense retrieval models.**\n\nOur evaluation employs dense retrieval for context augmentation. AdaCache's design is retriever-agnostic and can seamlessly integrate with any dense or sparse retrieval pipeline. In our current implementation, we don't cache chunk embeddings because retrieval latency is significantly smaller compared to LLM inference costs. However, caching embeddings would further reduce end-to-end latency when deploying embedding models.\n\n> ### **Q2: Figure 3 could be further improved for readability.**\n\nThank you for the feedback. We apologize for the readability issues and will improve Figure 3 for clarity in the revised manuscript."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1770918024789,"tcdate":1763783372087,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission20010/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission20010/Authors"],"forum":"Bmvx8ybDzo","number":2,"license":"CC BY 4.0","cdate":1763783372087,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission20010/-/Official_Comment","ICLR.cc/2026/Conference/-/Edit"],"mdate":1770918024789,"domain":"ICLR.cc/2026/Conference","replyto":"JmyRHMK7BY","id":"BtVl8cccDF","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["LLM Inference","Retrieval-Augmented Generation","Caching","Recomputation"]},"primary_area":{"value":"infrastructure, software libraries, hardware, systems, etc."},"abstract":{"value":"Retrieval-Augmented Generation (RAG) significantly enhances Large Language Models by integrating external knowledge sources, but at the cost of substantial computational overhead from extended input sequences. \nCurrent RAG systems exhibit two fundamental inefficiencies: redundant processing of frequently retrieved text chunks across multiple queries, and uniform deep retrieval that over-provisions context regardless of query complexity.\nWe present AdaCache, an adaptive caching framework that addresses these limitations through dual optimization strategies. \nFirst, we introduce a cache-aware partial recomputation mechanism that profiles attention patterns to construct selective cache variants, enabling flexible reuse while preserving cross-chunk dependencies. \nSecond, we develop adaptive context augmentation that dynamically determines optimal retrieval depth via lightweight confidence estimation, avoiding unnecessary overhead on simple queries.\nComprehensive experiments across diverse datasets and LLMs demonstrate that AdaCache delivers substantial improvements in Time-To-First-Token compared to state-of-the-art RAG caching systems, while preserving generation quality."},"_bibtex":{"value":"@inproceedings{\nzihao2026adacache,\ntitle={AdaCache: Adaptive Caching and Context Augmentation for Efficient {LLM} Serving},\nauthor={Zeng Zihao and Siyi Li and Xinyu Yan and Lei Xiao and Wei Yang Bryan Lim},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=Bmvx8ybDzo}\n}"},"title":{"value":"AdaCache: Adaptive Caching and Context Augmentation for Efficient LLM Serving"},"pdf":{"value":"/pdf/47dd33c4c6e73344674ecb0b9e0dd2dd603b8ecc.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"zihao|adacache_adaptive_caching_and_context_augmentation_for_efficient_llm_serving"},"authorids":{"value":["~Zeng_Zihao1","~Siyi_Li9","~Xinyu_Yan5","~Lei_Xiao7","~Wei_Yang_Bryan_Lim1"]},"authors":{"value":["Zeng Zihao","Siyi Li","Xinyu Yan","Lei Xiao","Wei Yang Bryan Lim"]}},"version":2},{"content":{"comment":{"value":"Thank you for the detailed review. Your distinction among consuming an existing hit, speculatively precomputing state, and admitting newly computed state is useful. We will make these actions explicit and label each main experiment by what it actually enforces: reuse, storage admission, both jointly, write-through observation, or diagnostic execution.\n\nThe process-isolated 1.31-1.61x range will remain scoped to the evaluated custom Hugging Face crop-and-replay configurations and their reported output-agreement rule. The broader SGLang, LMCache, and vLLM overlays measure compatibility and controller overhead, not MeritKV-caused acceleration. The Gemma-4-31B capacity-pressure experiment jointly enforces skip-write and skip-lookup, with most of the observed benefit coming from preserving the hot set through selective admission. We will state each of these scopes directly.\n\nSince submission, we have completed two relevant checks: LRU/LFU baselines under the same Gemma-4-31B capacity-pressure trace, and an explicit-state cache-continuation test separate from the original crop-and-replay experiment. We are also preparing an enforced version of the existing Blackwell write-through integration. This last experiment directly tests whether declining a resident native hit can help in that configuration. We will report the result regardless of direction and will not generalise beyond the measured setup.\n\nWe address the first two questions here and continue with the remaining two in the next comment.\n\n### Q1. For an exact prefix already resident in GPU memory, what measured cost can make recomputation preferable to consuming the hit?\n\nThe submitted paper does not establish a positive native-backend case where recomputation is preferable to consuming an exact prefix already resident in GPU memory. Lookup, transfer, assembly, synchronisation, and position-management costs can outweigh saved computation on external cache paths, but that does not imply the same result for a native resident GPU hit. The closest submitted native lookup-skipping experiment is negative in its tested configuration.\n\nThe capacity-pressure result answers a different question: whether admitting low-value state now causes future loss by evicting a recurring hot set. We will separate these two questions more clearly. The Blackwell enforced-versus-write-through comparison directly tests the resident-hit case.\n\n### Q2. Why is historical speculative waste charged to an exact resident hit?\n\nIt is not a direct causal cost of consuming that particular hit. In the evaluated implementation, the EWMA waste term is shared controller state in a harness that couples continuation, speculation, and later storage decisions. It conservatively suppresses activity after recent speculative admissions have been unproductive. We will describe it that way instead of implying that the resident hit itself caused the historical waste. This clarification does not change the measurements.\n\nWe will also clarify that the analytic model's delta_r is a fitted fixed-cost proxy rather than measured end-to-end cache-path overhead, and that analytic k* is different from the measured Blackwell prefix profile"}},"parentInvitations":"TMLR/-/Official_Comment","tmdate":1786712844655,"tcdate":1786712844655,"writers":["TMLR","TMLR/Paper11074/Authors"],"signatures":["TMLR/Paper11074/Authors"],"forum":"y3j6fucWQP","number":5,"license":"CC BY 4.0","cdate":1786712844655,"readers":["everyone"],"invitations":["TMLR/Paper11074/-/Official_Comment"],"mdate":1786712844655,"domain":"TMLR","replyto":"PTNrRsvFoI","id":"frBML5ReVi","forumContent":{"submission_length":{"value":"Regular submission (no more than 12 pages of main content)"},"venue":{"value":"Accepted by TMLR"},"code":{"value":"https://github.com/Kushalk0677/meritkv"},"abstract":{"value":"Key-value (KV) cache reuse lowers prefill latency, but lookup, cache manipulation, and speculative precompute can cost more than the computation they save, so a serving system can be slower than no cache even at a high hit rate. We argue that KV reuse should be a per-request economic decision rather than an unconditional mode, and present MeritKV, a lightweight admission-control layer that scores each candidate reuse by its net utility (saved prefill time minus reuse cost minus expected waste) and executes it only when that estimate is positive, bypassing otherwise. The gate is backend-agnostic and composes on top of existing prefix caches, while a single waste-feedback loop makes admission self-correcting as workloads change. Across controlled studies on five Hugging Face models (GPT-2, TinyLlama, Qwen2.5, Gemma, and Phi-3; 124M–3.8B parameters) and ten datasets, MeritKV matches no-cache performance on low-reuse traffic (0.99–1.01×) and observes 1.31–1.61× in custom-HF diagnostic configurations on structured prompts. In a 360-cell admission-policy comparison, it improves speedup from 1.085× to 1.110× while reducing the waste ratio from 24.0% to 5.6% relative to MeritKV-Sem, the reactive-plus-speculative variant without the policy controller. We additionally evaluate twelve models up to 32B and write-through SGLang, LMCache, and vLLM overlays, which establish compatibility and controller overhead, not acceleration. Under capacity pressure, five-seed native SGLang traces on Gemma-4-31B and Qwen2.5-32B show skip-write admission eliminating measured evictions and reducing mean latency/energy by 6.8–7.7%/7.6–8.5% versus native admit-all LRU; Qwen P95 regresses by 1.65% versus write-through, and native LFU remains stronger on that trace. A controlled multi-round trace reduces latency by 11.6–16.7% and improves recovery by 41–61 percentage points versus write-through; recovery remains 15.2–28.2 percentage points above controlled-bank LFU."},"_bibtex":{"value":"@article{\nkhemani2026meritkv,\ntitle={Merit{KV}: Utility-Gated Admission Control for {KV}-Cache Reuse in {LLM} Serving},\nauthor={Kushal Khemani and Evan J. Leri and Sparsh Mittal},\njournal={Transactions on Machine Learning Research},\nissn={2835-8856},\nyear={2026},\nurl={https://openreview.net/forum?id=y3j6fucWQP},\nnote={}\n}"},"video":{"value":"https://www.youtube.com/watch?v=X5VAFEl17zE"},"title":{"value":"MeritKV: Utility-Gated Admission Control for KV-Cache Reuse in LLM Serving"},"changes_since_last_submission":{"value":"Uploading the deanonymized camera-ready version, including author details, affiliations, acknowledgements, declarations, and the public code link, along with minor editorial and reference corrections."},"pdf":{"value":"/pdf/204645d44d3e621a14a925967acfa5e1b573cab6.pdf"},"venueid":{"value":"TMLR"},"paperhash":{"value":"khemani|meritkv_utilitygated_admission_control_for_kvcache_reuse_in_llm_serving"},"authorids":{"value":["~Kushal_Khemani1","~Evan_J._Leri1","~Sparsh_Mittal1"]},"assigned_action_editor":{"value":"~Binhang_Yuan1"},"authors":{"value":["Kushal Khemani","Evan J. Leri","Sparsh Mittal"]}},"version":2},{"content":{"comment":{"value":"We thank the reviewer for this important challenge. We address both points and the requested clarification below.\n\n**1. The Logical Paradox in Prefix Sharing**\n\nWe provide the requested end-to-end exploit chain under standard API constraints:\n\n1. Attacker crafts a prompt that collides with the victim's prefix hash\n2. Attacker's KV entries get cached under the colliding hash\n3. Victim's request hits the cache and retrieves attacker's entries\n4. Without GeoCache: victim's generation is corrupted (semantic poisoning)\n5. With GeoCache: attacker's entries are in a different coordinate system — victim's queries produce noise — attack fails\n\nThis is an API-level attack requiring no hardware access and no privilege escalation. **CVE-2025-25183** (vLLM, fixed in v0.7.2) and **CVE-2025-1953** (vLLM AIBrix) demonstrated exactly this — hash collisions in prefix caching allowed an attacker to serve poisoned KV entries to a victim through a different prompt. \"Cache Me, Catch You\" (Wu et al., NDSS 2026) documented additional vulnerabilities across six attack vectors in vLLM, SGLang, and GPTCache, showing that cache collision attacks lead to incorrect model outputs, information leakage, and content filter bypasses.\n\nWhile these have been patched (vLLM v0.11 now defaults to SHA-256), three observations: (1) This vulnerability class recurred independently — CVE-2025-25183 (vLLM), CVE-2025-1953 (AIBrix), and additional vulnerabilities across six attack vectors (Wu et al., NDSS 2026). A single patch does not eliminate a systemic pattern. (2) vLLM v0.11 offers xxhash as a faster alternative to SHA-256, and their own documentation warns this \"can cause undefined behavior or even leak private information in multi-tenant environments.\" Operators under latency pressure may choose speed over security. (3) Defense-in-depth is the security standard — no single layer is assumed infallible. GeoCache provides the content protection layer independent of hash implementation correctness.\n\n**2. The Missing Exploit Chain — Physical Access Vectors**\n\nBeyond hash collisions, KV tensors also become physically accessible in production infrastructure:\n\n**(a) GPU memory residuals.** LeftoverLocals (Sorensen & Khlaaf, 2024; CVE-2023-4969) demonstrated that GPU local memory is not cleared between workloads, allowing a subsequent tenant to read residual KV tensors.\n\n**(b) Disaggregated serving.** Production architectures (e.g., LMCache, llm-d, AIBrix) store and transmit KV tensors across nodes on remote DRAM/SSD. A compromised storage node or network interceptor can read tensor content.\n\n**(c) Externalized KV-cache outside TEE.** This is the same threat model as KV-Cloak (NDSS 2026) — the KV-cache sits outside the TEE boundary, readable by the CSP.\n\nLogical access controls (cache_salt, SHA-256 hashing) prevent unauthorized cache *hits* but do not protect tensor *content* once physically accessed. GeoCache protects both layers.\n\n**3. Regarding Timing Padding Sufficiency**\n\nTiming padding addresses existence leakage but does not protect tensor content under either vector above. If an attacker poisons the cache via hash collision, timing padding does not help — the victim still retrieves corrupted KV entries. If an attacker reads tensors from GPU memory, timing padding does not help — the raw content is exposed. GeoCache and timing padding address different layers.\n\nWe will rewrite Section 3.1 to explicitly describe both attack vectors (hash collision and physical access) with CVE references."},"title":{"value":"Author Response to Reviewer"}},"parentInvitations":"TMLR/-/Official_Comment","tmdate":1775998510279,"tcdate":1775998510279,"writers":["TMLR","TMLR/Paper8077/Authors"],"signatures":["TMLR/Paper8077/Authors"],"forum":"CNuH7VtNun","number":4,"license":"CC BY 4.0","cdate":1775998510279,"readers":["everyone"],"invitations":["TMLR/Paper8077/-/Official_Comment"],"mdate":1775998510279,"domain":"TMLR","replyto":"1PrdkCMnlM","id":"ww9v15efFC","forumContent":{"submission_length":{"value":"Long submission (more than 12 pages of main content)"},"venue":{"value":"Rejected by TMLR"},"abstract":{"value":"Key-value (KV) cache sharing across user sessions is a critical optimization for multi-tenant large language model (LLM) serving, reducing time-to-first-token (TTFT) latency by up to 73%. However, recent work has demonstrated that shared KV caches create exploitable vulnerabilities: an adversary can exfiltrate data or mount semantic poisoning attacks by accessing cross-tenant cache states. Existing mitigations either disable sharing entirely (negating performance gains) or employ heuristic content classifiers with inherent false-positive/negative tradeoffs. We propose GeoCache, a principled approach that applies session-specific orthogonal transformations to key and value attention matrices before caching. GeoCache exploits the isometry property of orthogonal operators to provide two complementary guarantees: (1) exact preservation of attention computation for authorized in-session cache reuse, proven by the identity q⊤M⊤MK = q⊤K (Theorem 1); and (2) geometric isolation of cross-session cache entries, with attention scores concentrating near zero under a BDOT-specific exponential bound (Theorem 2). The approach uses a Block-Diagonal Orthogonal Transform (BDOT) achieving O(d_k) per-token overhead and sub-microsecond (0.014 μs) latency per head-token on an NVIDIA T4 GPU. Experiments on Llama-2-7B and Mistral-7B confirm: (a) generation quality is preserved exactly (maximum float32 deviation 5.3 × 10⁻⁵); (b) controlled exfiltration and injection attacks via the forward attention path are substantially mitigated (ROUGE-L reduced from 0.917 to 0.043, 0/19 secret markers leaked; cross-session cosine similarity reduced from 1.0 to ≈ 0); and (c) throughput overhead is negligible. We position GeoCache as providing provable utility preservation together with computational content-level isolation (rather than cryptographic security): a separate algebraic recovery attack (Known-Plaintext Attack) on the BDOT operator is feasible if the attacker collects b linearly independent plaintext-ciphertext pairs per block. We therefore require operator rotation every b/2 tokens for production deployments serving structured prompts (JSON schemas, few-shot exemplars), which empirically reduces algebraic recovery to subspace-only directional leakage insufficient for exact token reconstruction; under freeform queries, hybrid deployment alone suffices. For deployments requiring timing-channel protection, GeoCache composes with constant-time response padding to eliminate existence leakage."},"_bibtex":{"value":"@article{\nanonymous2026geocache,\ntitle={GeoCache: Provably Lossless Inference and Computational Content-Level Isolation for Shared {KV}-Caches in Multi-Tenant {LLM} Inference via Isometric Orthogonal Transformation},\nauthor={Anonymous},\njournal={Submitted to Transactions on Machine Learning Research},\nyear={2026},\nurl={https://openreview.net/forum?id=CNuH7VtNun},\nnote={Rejected}\n}"},"title":{"value":"GeoCache: Provably Lossless Inference and Computational Content-Level Isolation for Shared KV-Caches in Multi-Tenant LLM Inference via Isometric Orthogonal Transformation"},"changes_since_last_submission":{"value":"Summary of Revisions (v3)\n\n1. Claims scoped. Title revised to \"Provably Lossless Inference and Computational Content-Level Isolation.\" Abstract and body reframed; \"eliminated\" and \"mathematically infeasible\" replaced with scoped empirical statements.\n\n2. Theorem 2 corrected. BDOT-specific proof added (Appendix A.4) with block-wise concentration bound depending on b, not dk. Variance analysis corrected for best/worst case.\n\n3. Real-template KPA evaluation. Five production templates on Mistral-7B (Table 10, Appendix D). Structured templates reach cosine 0.86-0.90 without rotation; operator rotation reduces to 0.51-0.64. Rotation reframed from \"additional hardening\" to required for production.\n\n4. Threat model clarified. Hash collisions split into poisoning/exfiltration directions. Plaintext K transience stated. New Limitation 6: confidentiality-to-availability degradation. KPA requires conjunction of ciphertext access AND plaintext knowledge.\n\n5. Public/private baseline added. New Table 8 row and Discussion paragraph. GeoCache positioned as defense-in-depth; simple partitioning acknowledged as sufficient when hash isolation is trusted and physical access excluded.\n\n6. KV-Cloak corrected. Removed inaccurate ~5% overhead claim. Differentiators reduced to three. \"Concurrent\" changed to \"recent work.\"\n\n7. Math/editorial fixes. Latency op count corrected (3→4). Per-head vs per-layer semantics reconciled. Theorem 1 proof clarified. GQA claim removed. Cryptographic Assumption justified via PRG chain.\n\n8. Artifacts. Two new KPA scripts added. README updated.\n\nCore methodology unchanged. Theorem 1 preserved; Theorem 2 proof corrected for BDOT."},"pdf":{"value":"/pdf/6989390b7dd1910414bb94c4012a07f477fe130d.pdf"},"venueid":{"value":"TMLR/Rejected"},"previous_TMLR_submission_url":{"value":"https://openreview.net/forum?id=oX8YpiElYp"},"assigned_action_editor":{"value":"~Yiming_Li1"}},"version":2},{"content":{"comment":{"value":"We thank the reviewer for the thorough and technically detailed feedback. We address each concern below.\n\n**1. Known-Plaintext Attack (KPA) and Security Boundary**\n\nWe agree GeoCache provides computational security via linear obfuscation, not a cryptographic security boundary. The revised manuscript will state this distinction explicitly. GeoCache's linearity is a deliberate design choice: orthogonality enables exact isometry (Theorem 1) with zero quality loss. A cryptographic transform would break q⊤M⊤MK = q⊤K. This trades cryptographic hardness for zero degradation — appropriate when hybrid mode ensures the attacker lacks the plaintext to exploit linearity.\n\nFor **user-specific content** (private prompts, PII, confidential queries), the attacker does not have the plaintext tokens needed to construct 64 linearly independent pairs. The KPA requires knowing what the victim typed — which is the secret being protected.\n\nFor **shared system prompts**, the attacker can generate plaintext KV tensors locally. However, in GeoCache's hybrid mode, shared system prompts are deliberately left **untransformed** — cached in plaintext and shared for TTFT savings. Only user-specific suffixes are protected with operators derived from session-specific identifiers via SHA-256. The KPA on the system prompt portion is moot because those tokens are not transformed.\n\nWe have conducted a formal KPA simulation. Recovery requires exactly N=64 pairs (cosine progresses from 0.41 at N=10 to 0.81 at N=40, reaching 1.0 only at N=64). Under hybrid mode with b=64, the attacker obtains 0 valid pairs (system prompt untransformed, suffix plaintext unknown). Even a template attack with 20 known suffix tokens yields cosine of 0.55 — insufficient for meaningful recovery. Operator rotation every N < b tokens provides additional hardening: with rotation every 32 tokens, cosine remains 0.56 even when all suffix plaintext is known.\n\nThe revised manuscript will include a formal KPA table.\n\n**2. Comparison with vLLM cache_salt**\n\nvLLM's cache_salt prevents cross-tenant logical cache hits by salting the prefix hash. However, hash-based isolation has been publicly broken:\n\n**CVE-2025-25183** (vLLM, fixed in v0.7.2) and **CVE-2025-1953** (vLLM AIBrix) demonstrated that hash collisions allowed attackers to serve poisoned KV entries to victims. \"Cache Me, Catch You\" (Wu et al., NDSS 2026) documented additional vulnerabilities across six attack vectors in vLLM, SGLang, and GPTCache. While patched (vLLM v0.11 defaults to SHA-256), these demonstrate that hash-based isolation is a software policy subject to implementation flaws.\n\nGeoCache provides defense-in-depth: even if a hash collision occurs, the retrieved KV entries are in a different coordinate system and produce meaningless attention scores. Additionally, cache_salt does not protect tensor content when physically accessed — through GPU memory residuals (LeftoverLocals, Sorensen et al. 2024), disaggregated serving architectures (e.g., LMCache, llm-d, AIBrix), or TEE externalization.\n\nThe revised manuscript will include cache_salt as a baseline.\n\n**3. Adaptive Attack**\n\nWe have conducted an adaptive attack analysis where the attacker is fully aware of GeoCache and attempts algebraic recovery. The KPA simulation in Point 1 applies directly: under hybrid mode the attacker obtains 0 valid pairs, and operator rotation with N < b provides additional hardening. This analysis with a reproducible simulation script will be included in the revised manuscript.\n\n**4. Content-Level Attack Evaluation**\n\nWe have conducted controlled attack experiments on Mistral-7B-Instruct-v0.1 following the protocols of Luo et al. (2025):\n\n**Injection attack.** The attacker obtains a victim's KV-cache (via physical access vectors described above) and appends an instruction to extract private content. Without GeoCache: **avg ROUGE-L = 0.917, 19/19 secret markers recovered**. With GeoCache: **avg ROUGE-L = 0.043, 0/19 markers recovered (95.3% reduction)**.\n\n**Exfiltration attack.** The attacker compares candidate Key vectors against the victim's cached Keys via cosine similarity. Without GeoCache: exact match has **cosine = 1.0**, spread = 0.336. With GeoCache: all candidates produce **cosine ≈ -0.01, spread = 0.012**.\n\nThese along with comparisons against cache_salt, CacheSolidarity, and SafeKV will be included in the revised manuscript.\n\n**5. Cross-Tenant Caching**\n\nThe reviewer correctly notes that session-specific operators break cross-tenant deduplication for protected content. GeoCache's hybrid mode addresses this — public prefixes remain shared while private suffixes are protected. In experiments on Mistral-7B with a 130-token shared system prompt, hybrid mode reduces average TTFT by **21.4%** compared to per-session isolation while keeping suffix keys geometrically isolated **(cosine = 0.001)**.\n\nWe share these results while we prepare the full revision with updated analysis and code."},"title":{"value":"Author Response to Reviewer"}},"parentInvitations":"TMLR/-/Official_Comment","tmdate":1776113121170,"tcdate":1776113121170,"writers":["TMLR","TMLR/Paper8077/Authors"],"signatures":["TMLR/Paper8077/Authors"],"forum":"CNuH7VtNun","number":5,"license":"CC BY 4.0","cdate":1776113121170,"readers":["everyone"],"invitations":["TMLR/Paper8077/-/Official_Comment"],"mdate":1776113121170,"domain":"TMLR","replyto":"vEITLaWxQo","id":"MMlwYh2Gzf","forumContent":{"submission_length":{"value":"Long submission (more than 12 pages of main content)"},"venue":{"value":"Rejected by TMLR"},"abstract":{"value":"Key-value (KV) cache sharing across user sessions is a critical optimization for multi-tenant large language model (LLM) serving, reducing time-to-first-token (TTFT) latency by up to 73%. However, recent work has demonstrated that shared KV caches create exploitable vulnerabilities: an adversary can exfiltrate data or mount semantic poisoning attacks by accessing cross-tenant cache states. Existing mitigations either disable sharing entirely (negating performance gains) or employ heuristic content classifiers with inherent false-positive/negative tradeoffs. We propose GeoCache, a principled approach that applies session-specific orthogonal transformations to key and value attention matrices before caching. GeoCache exploits the isometry property of orthogonal operators to provide two complementary guarantees: (1) exact preservation of attention computation for authorized in-session cache reuse, proven by the identity q⊤M⊤MK = q⊤K (Theorem 1); and (2) geometric isolation of cross-session cache entries, with attention scores concentrating near zero under a BDOT-specific exponential bound (Theorem 2). The approach uses a Block-Diagonal Orthogonal Transform (BDOT) achieving O(d_k) per-token overhead and sub-microsecond (0.014 μs) latency per head-token on an NVIDIA T4 GPU. Experiments on Llama-2-7B and Mistral-7B confirm: (a) generation quality is preserved exactly (maximum float32 deviation 5.3 × 10⁻⁵); (b) controlled exfiltration and injection attacks via the forward attention path are substantially mitigated (ROUGE-L reduced from 0.917 to 0.043, 0/19 secret markers leaked; cross-session cosine similarity reduced from 1.0 to ≈ 0); and (c) throughput overhead is negligible. We position GeoCache as providing provable utility preservation together with computational content-level isolation (rather than cryptographic security): a separate algebraic recovery attack (Known-Plaintext Attack) on the BDOT operator is feasible if the attacker collects b linearly independent plaintext-ciphertext pairs per block. We therefore require operator rotation every b/2 tokens for production deployments serving structured prompts (JSON schemas, few-shot exemplars), which empirically reduces algebraic recovery to subspace-only directional leakage insufficient for exact token reconstruction; under freeform queries, hybrid deployment alone suffices. For deployments requiring timing-channel protection, GeoCache composes with constant-time response padding to eliminate existence leakage."},"_bibtex":{"value":"@article{\nanonymous2026geocache,\ntitle={GeoCache: Provably Lossless Inference and Computational Content-Level Isolation for Shared {KV}-Caches in Multi-Tenant {LLM} Inference via Isometric Orthogonal Transformation},\nauthor={Anonymous},\njournal={Submitted to Transactions on Machine Learning Research},\nyear={2026},\nurl={https://openreview.net/forum?id=CNuH7VtNun},\nnote={Rejected}\n}"},"title":{"value":"GeoCache: Provably Lossless Inference and Computational Content-Level Isolation for Shared KV-Caches in Multi-Tenant LLM Inference via Isometric Orthogonal Transformation"},"changes_since_last_submission":{"value":"Summary of Revisions (v3)\n\n1. Claims scoped. Title revised to \"Provably Lossless Inference and Computational Content-Level Isolation.\" Abstract and body reframed; \"eliminated\" and \"mathematically infeasible\" replaced with scoped empirical statements.\n\n2. Theorem 2 corrected. BDOT-specific proof added (Appendix A.4) with block-wise concentration bound depending on b, not dk. Variance analysis corrected for best/worst case.\n\n3. Real-template KPA evaluation. Five production templates on Mistral-7B (Table 10, Appendix D). Structured templates reach cosine 0.86-0.90 without rotation; operator rotation reduces to 0.51-0.64. Rotation reframed from \"additional hardening\" to required for production.\n\n4. Threat model clarified. Hash collisions split into poisoning/exfiltration directions. Plaintext K transience stated. New Limitation 6: confidentiality-to-availability degradation. KPA requires conjunction of ciphertext access AND plaintext knowledge.\n\n5. Public/private baseline added. New Table 8 row and Discussion paragraph. GeoCache positioned as defense-in-depth; simple partitioning acknowledged as sufficient when hash isolation is trusted and physical access excluded.\n\n6. KV-Cloak corrected. Removed inaccurate ~5% overhead claim. Differentiators reduced to three. \"Concurrent\" changed to \"recent work.\"\n\n7. Math/editorial fixes. Latency op count corrected (3→4). Per-head vs per-layer semantics reconciled. Theorem 1 proof clarified. GQA claim removed. Cryptographic Assumption justified via PRG chain.\n\n8. Artifacts. Two new KPA scripts added. README updated.\n\nCore methodology unchanged. Theorem 1 preserved; Theorem 2 proof corrected for BDOT."},"pdf":{"value":"/pdf/6989390b7dd1910414bb94c4012a07f477fe130d.pdf"},"venueid":{"value":"TMLR/Rejected"},"previous_TMLR_submission_url":{"value":"https://openreview.net/forum?id=oX8YpiElYp"},"assigned_action_editor":{"value":"~Yiming_Li1"}},"version":2}],"count":23}