example but that is also slightly confusing in the way it's being presented right now). \n\n3. Presentation needs to be improved: there seems to be unfinished section (for example missing figure in line 205-207)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762915527052,"tcdate":1761808219066,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission477/Reviewer_JCWE"],"signatures":["ICLR.cc/2026/Conference/Submission477/Reviewer_JCWE"],"forum":"LeatkxrBCi","number":4,"license":"CC BY 4.0","cdate":1761808219066,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission477/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762915527052,"domain":"ICLR.cc/2026/Conference","replyto":"LeatkxrBCi","id":"7KVMZa08GG","forumContent":{"TLDR":{"value":"We enable LLMs to communicate directly through their internal KV-Cache representations, rather than generating text"},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["LLM","multi-LLM","multi-agent","communication"]},"supplementary_material":{"value":"/attachment/df5dc60a699a01faa222b6910d118c555dc4afda.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Multi-LLM systems harness the complementary strengths of diverse Large Language Models, achieving performance and efficiency gains that are not attainable by a single model. In existing designs, LLMs communicate through text, forcing internal representations to be transformed into output token sequences. This process both loses rich semantic information and incurs token-by-token generation latency. Motivated by these limitations, we ask: Can LLMs communicate beyond text? Oracle experiments show that enriching the KV-Cache semantics can improve response quality without increasing cache size, supporting KV-Cache as an effective medium for inter-model communication. Thus, we propose Cache-to-Cache (C2C), a new paradigm for direct semantic communication between LLMs. C2C uses a neural network to project and fuse the source model's KV-cache with that of the target model to enable direct semantic transfer. A learnable gating mechanism selects the target layers that benefit from cache communication. Compared with text communication, C2C utilizes the deep, specialized semantics from both models, while avoiding explicit intermediate text generation. Experiments show that C2C achieves 6.4-14.2% higher average accuracy than individual models. It further outperforms the text communication paradigm by approximately 3.1-5.4%, while delivering an average 2.5x speedup in latency. Our code is available at https://github.com/thu-nics/C2C."},"_bibtex":{"value":"@inproceedings{\nfu2026cachetocache,\ntitle={Cache-to-Cache: Direct Semantic Communication Between Large Language Models},\nauthor={Tianyu Fu and Zihan Min and Hanling Zhang and Jichao Yan and Guohao Dai and Wanli Ouyang and Yu Wang},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=LeatkxrBCi}\n}"},"title":{"value":"Cache-to-Cache: Direct Semantic Communication Between Large Language Models"},"pdf":{"value":"/pdf/4b7b9ab45327188a6b1b616d8a71f8d1664f3663.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"fu|cachetocache_direct_semantic_communication_between_large_language_models"},"authorids":{"value":["~Tianyu_Fu3","~Zihan_Min1","~Hanling_Zhang3","~Jichao_Yan1","~Guohao_Dai4","~Wanli_Ouyang1","~Yu_Wang3"]},"authors":{"value":["Tianyu Fu","Zihan Min","Hanling Zhang","Jichao Yan","Guohao Dai","Wanli Ouyang","Yu Wang"]}},"version":2},{"content":{"summary":{"value":"This paper proposes a novel approach to sharing the key-value (KV) cache across different layers in a new dimension, which can lead to more efficient memory usage and improved performance."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"What is the time scalability of the proposed approach? Will the inference time remain acceptable when scaling up to models with over 400 billion parameters? It would be valuable to provide an estimation or analysis to address this concern."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"This idea offers new insights into how memory size can be further reduced, potentially leading to more efficient model deployments and optimized hardware utilization."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1) The paper lacks a comparison with other cache-sharing methods, which would provide a clearer understanding of its advantages.\n\n2) It should consider the scenario when the KV cache is quantized, as quantization is often used during inference to save energy.\n\n3) The paper also lacks a scalability analysis, which is crucial for evaluating how well the proposed method performs as model size and complexity increase."}},"nonreaders":[],"tmdate":1731428905086,"tcdate":1729758730393,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission7482/Reviewer_fUgR"],"signatures":["ICLR.cc/2025/Conference/Submission7482/Reviewer_fUgR"],"forum":"2Akf4BBCKo","number":1,"license":"CC BY 4.0","cdate":1729758730393,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission7482/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428905086,"domain":"ICLR.cc/2025/Conference","replyto":"2Akf4BBCKo","id":"yJgBYw3WNp","forumContent":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Large Language Model","KV Cache","KVSharer"]},"supplementary_material":{"value":"/attachment/f460ce31466303cf912af801d075b73567417681.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The development of large language models (LLMs) has significantly expanded model sizes, resulting in substantial GPU memory requirements during inference. The key and value storage of the attention map in the KV (key-value) cache accounts for more than 80\\% of this memory consumption. Nowadays, most existing KV cache compression methods focus on intra-layer compression within a single Transformer layer but few works consider layer-wise compression. In this paper, we propose a plug-and-play method called \\textit{KVSharer}, which shares the KV cache between layers to achieve layer-wise compression. Rather than intuitively sharing based on higher similarity, we discover a counterintuitive phenomenon: sharing dissimilar KV caches better preserves the model performance. Experiments show that \\textit{KVSharer} can reduce KV cache computation by 30\\%, thereby lowering memory consumption without significantly impacting model performance and it can also achieve at least 1.3 times generation acceleration. Additionally, we verify that \\textit{KVSharer} is compatible with existing intra-layer KV cache compression methods, and combining both can further save memory."},"_bibtex":{"value":"@misc{\nyang2024kvsharer,\ntitle={{KVS}harer: Efficient Inference via Layer-Wise Dissimilar {KV} Cache Sharing},\nauthor={Yifei Yang and zouying cao and Qiguang Chen and Libo Qin and Dongjie Yang and Zhi Chen and hai zhao},\nyear={2024},\nurl={https://openreview.net/forum?id=2Akf4BBCKo}\n}"},"title":{"value":"KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing"},"pdf":{"value":"/pdf/89b8a1b38e6cf82dd43280011e39cb6cefe942b9.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"yang|kvsharer_efficient_inference_via_layerwise_dissimilar_kv_cache_sharing"},"authorids":{"value":["~Yifei_Yang2","~zouying_cao1","~Qiguang_Chen1","~Libo_Qin1","~Dongjie_Yang1","~Zhi_Chen1","~hai_zhao1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Yifei Yang","zouying cao","Qiguang Chen","Libo Qin","Dongjie Yang","Zhi Chen","hai zhao"]}},"version":2},{"content":{"venueid":{"value":"OpenReview.net/Archive"},"paperhash":{"value":"jiang|packkv_reducing_kv_cache_memory_footprint_through_llmaware_lossy_compression"},"authorids":{"value":["bo.jiang@temple.edu","taolue.yang@temple.edu","~Youyuan_Liu1","~Xubin_He1","~Sheng_Di1","~Sian_Jin1"]},"html":{"value":"https://arxiv.org/abs/2512.24449"},"abstract":{"value":"Transformer-based large language models (LLMs) have demonstrated remarkable potential across a wide range of practical applications. However, long-context inference remains a significant challenge due to the substantial memory requirements of the key-value (KV) cache, which can scale to several gigabytes as sequence length and batch size increase. In this paper, we present \\textbf{PackKV}, a generic and efficient KV cache management framework optimized for long-context generation. %, which synergistically supports both latency-critical and throughput-critical inference scenarios. PackKV introduces novel lossy compression techniques specifically tailored to the characteristics of KV cache data, featuring a careful co-design of compression algorithms and system architecture. Our approach is compatible with the dynamically growing nature of the KV cache while preserving high computational efficiency. Experimental results show that, under the same and minimum accuracy drop as state-of-the-art quantization methods, PackKV achieves, on average, \\textbf{153.2}\\% higher memory reduction rate for the K cache and \\textbf{179.6}\\% for the V cache. Furthermore, PackKV delivers extremely high execution throughput, effectively eliminating decompression overhead and accelerating the matrix-vector multiplication operation. Specifically, PackKV achieves an average throughput improvement of \\textbf{75.7}\\% for K and \\textbf{171.7}\\% for V across A100 and RTX Pro 6000 GPUs, compared to cuBLAS matrix-vector multiplication kernels, while demanding less GPU memory bandwidth."},"title":{"value":"PackKV: Reducing KV Cache Memory Footprint through LLM-Aware Lossy Compression"},"authors":{"value":["Bo Jiang","Taolue Yang","Youyuan Liu","Xubin He","Sheng Di","Sian Jin"]}},"tmdate":1790229467083,"pdate":1790226000000,"tcdate":1790229467083,"writers":["bo.jiang@temple.edu","taolue.yang@temple.edu","~Youyuan_Liu1","~Xubin_He1","~Sheng_Di1","~Sian_Jin1"],"signatures":["~Sheng_Di1"],"forum":"d5FjO1O3pf","license":"CC BY 4.0","number":54680,"cdate":1790229467083,"readers":["everyone"],"invitations":["OpenReview.net/Archive/-/Direct_Upload"],"mdate":1790229467083,"domain":"OpenReview.net/Archive","id":"d5FjO1O3pf","version":2},{"content":{"summary":{"value":"This paper introduces **POD** (Proximal Over Distant), a method to boost the efficiency of LLM inference with long contexts. \n\nIt addresses the memory and computation bottlenecks of the decoding by leveraging two insights: (1) Proximal Tokens Importance, where initial and recent tokens are more critical than distant tokens, allowing prioritized computation for these tokens; and (2) Intra-Layer Attention Similarity, where attention patterns between consecutive layers are similar, enabling shared attention scores for distant tokens across layers. POD optimizes inference by grouping layers based on this similarity, using shared attention for distant tokens, and minimizing memory usage in KV caching. Experiments show that POD can reduce KV cache memory by 35% while maintaining comparable accuracy to baseline models, effectively optimizing long-context LLMs without compromising on essential token information."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"1. How does the approach apply to larger-scaled and up-to-date models such LLaMA-3-70B and LLaMA-3.1 models? \n\n2. Does the approach outperform other token-selection-based sparse attention mechanisms other than pure eviction-based baselines?\n\n3. Can authors justify the weakness 1. and elaborate the observation from Figure 1(a)?"},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. The paper creatively combines layer-similarity and post-trianing and proposes a novel approach **POD**, which improves inference efficiency by reducing memory requirements for less important distant tokens through inter-layer attention sharing, achieving a 35% reduction in KV cache while retaining all tokens for performance stability.\n\n2. It provides evaluations across multiple long-context benchmarks and relevant ablation studies. **POD** outperforms baselines in most cases on provided experiments. In addition, the authors described the pipeline and details of **PoD** in a clear way."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The paper didn't explain the motivation in a reasonable way. \n\nFor example, (1) the observation 1 from Figure 1(a) is NOT really valid given the algorithm of StreamingLLM is to evict the tokens except for sink tokens (i.e. initial tokens) and most recent ones. In this case, \"A equals B\" inserted in the middle will be evicted and thus not used by StreamingLLM, which reasonably fails to return correct answer in the input. (2) The trend of similar attention scores shown in Figure 1(d) might not be immediately obvious to readers.\n\n2. The setup of empirical results of the paper is relatively limited. \n\nFor example, (1) the paper only used LLaMA-3-8B model in experiment section, it would be better to how this approach performs when the LLM scales up; (2) the paper should also include some other SOTA token-selection-based approaches such as Quest: https://arxiv.org/abs/2406.10774\n\n3. The training stage of **PoD** may add more computational costs the application of the approach."}},"nonreaders":[],"tmdate":1731429303223,"tcdate":1730129779837,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission13815/Reviewer_iXy9"],"signatures":["ICLR.cc/2025/Conference/Submission13815/Reviewer_iXy9"],"forum":"G1fzW97QKR","number":1,"license":"CC BY 4.0","cdate":1730129779837,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission13815/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731429303223,"domain":"ICLR.cc/2025/Conference","replyto":"G1fzW97QKR","id":"qXQ5OkU58p","forumContent":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"TLDR":{"value":"We save 35% KV cache without compromising the performance by sharing attention scores across layers."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Long context LLM","attention similarity","window attention"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The increasing context window size in Large Language Models (LLMs), such as the GPT and LLaMA series, has improved their ability to tackle complex, long-text tasks, but at the cost of inference efficiency, particularly regarding memory and computational complexity. Existing methods, including selective token retention and window-based attention, improve efficiency but risk discarding important tokens needed for future text generation. In this paper, we propose an approach that enhances LLM efficiency without token loss by reducing the memory and computational load of less important tokens, rather than discarding them. \n We address two challenges: 1) investigating the distribution of important tokens in the context, discovering recent tokens are more important than distant tokens in context, and 2) optimizing resources for distant tokens by sharing attention scores across layers. The experiments show that our method saves $35$% KV cache without compromising the performance."},"_bibtex":{"value":"@misc{\nma2024boosting,\ntitle={Boosting Long-Context {LLM} Inference Efficiency with Intra-Layer Attention Similarity},\nauthor={Da Ma and Lu Chen and Situo Zhang and Yuxun Miao and Su Zhu and Zhi Chen and Hongshen Xu and Hanqi Li and Shuai Fan and Kai Yu},\nyear={2024},\nurl={https://openreview.net/forum?id=G1fzW97QKR}\n}"},"title":{"value":"Boosting Long-Context LLM Inference Efficiency with Intra-Layer Attention Similarity"},"pdf":{"value":"/pdf/29f7f317a634feb298f8df3a215f7f84684c68d0.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"ma|boosting_longcontext_llm_inference_efficiency_with_intralayer_attention_similarity"},"authorids":{"value":["~Da_Ma2","~Lu_Chen3","~Situo_Zhang1","~Yuxun_Miao1","~Su_Zhu1","~Zhi_Chen1","~Hongshen_Xu1","~Hanqi_Li3","~Shuai_Fan1","~Kai_Yu3"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Da Ma","Lu Chen","Situo Zhang","Yuxun Miao","Su Zhu","Zhi Chen","Hongshen Xu","Hanqi Li","Shuai Fan","Kai Yu"]}},"version":2},{"content":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Video Large Language Model","Visual KV Cache Quantization"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Video large language models (VideoLLMs) have demonstrated the capability to process longer video inputs and enable complex reasoning and analysis. However, due to the thousands of visual tokens from the video frames, the key-value (KV) cache can significantly increase memory requirements, becoming a bottleneck for inference speed and memory usage. KV cache quantization is a widely used approach to address this problem. In this paper, we find that 2-bit KV quantization of VideoLLMs can hardly hurt the model performance, while the limit of KV cache quantization in even lower bits has not been investigated. To bridge this gap, we introduce VidKV, a plug-and-play KV cache quantization method to compress the KV cache to lower than 2 bits. Specifically, (1) for key, we propose a mixed-precision quantization strategy in the channel dimension, where we perform 2-bit quantization for anomalous channels and 1-bit quantization combined with FFT for normal channels; (2) for value, we implement 1.58-bit quantization while selectively filtering semantically salient visual tokens for targeted preservation, for a better trade-off between precision and model performance. Importantly, our findings suggest that the value cache of VideoLLMs should be quantized in a per-channel fashion instead of the per-token fashion proposed by prior KV cache quantization works for LLMs. Empirically, extensive results with LLaVA-OV-7B and Qwen2.5-VL-7B on six benchmarks show that VidKV effectively compresses the KV cache to 1.5-bit and 1.58-bit precision with almost no performance drop compared to the FP16 counterparts."},"_bibtex":{"value":"@misc{\ntao2026plugandplay,\ntitle={Plug-and-Play 1.x-Bit {KV} Cache Quantization for Video Large Language Models},\nauthor={Keda TAO and Haoxuan You and Yang Sui and Can Qin and Huan Wang},\nyear={2026},\nurl={https://openreview.net/forum?id=ihZMJuLfbi}\n}"},"title":{"value":"Plug-and-Play 1.x-Bit KV Cache Quantization for Video Large Language Models"},"pdf":{"value":"/pdf/b2b5e2c96293280ce4ac8cb95e7c9ca37063c607.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"tao|plugandplay_1xbit_kv_cache_quantization_for_video_large_language_models"},"authorids":{"value":["~Keda_TAO1","~Haoxuan_You1","~Yang_Sui1","~Can_Qin1","~Huan_Wang3"]},"authors":{"value":["Keda TAO","Haoxuan You","Yang Sui","Can Qin","Huan Wang"]}},"tmdate":1768749619236,"tcdate":1756822089292,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission892/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission892/Authors"],"forum":"ihZMJuLfbi","license":"CC BY 4.0","number":892,"cdate":1756822089292,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/-/Submission","ICLR.cc/2026/Conference/-/Post_Submission","ICLR.cc/2026/Conference/Submission892/-/Full_Submission","ICLR.cc/2026/Conference/Submission892/-/Rebuttal_Revision","ICLR.cc/2026/Conference/-/Withdrawn_Submission"],"mdate":1768749619236,"odate":1759896705795,"domain":"ICLR.cc/2026/Conference","id":"ihZMJuLfbi","version":2},{"content":{"venue":{"value":"CoRR 2026"},"pdf":{"value":"https://arxiv.org/pdf/2608.02691v4"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"yun|outputaware_rotation_for_int2_kvcache_quantization"},"html":{"value":"https://doi.org/10.48550/arXiv.2608.02691"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2608-02691,\n publtype={informal},\n author={Vincent-Daniel Yun and Woosang Lim and Minsoo Cheong and Sunwoo Lee and Murali Annavaram and Sai Praneeth Karimireddy and Sungjoo Yoo},\n title={Output-Aware Rotation for INT2 KV-Cache Quantization},\n year={2026},\n month={August},\n cdate={1785542400000},\n journal={CoRR},\n volume={abs/2608.02691},\n url={https://doi.org/10.48550/arXiv.2608.02691}\n}\n"},"abstract":{"value":"The key-value (KV) cache has become a major memory and bandwidth bottleneck in long-context large language model inference, making ultra-low-bit quantization increasingly important. However, existing rotation-based INT2 methods optimize cache statistics or proxy errors before the complete attention readout, even though the model is ultimately affected by the error propagated through attention and the output projection $W_O$. To address this mismatch, we propose \\textit{OptR}, an output-aware rotation method that minimizes post-$W_O$ attention-output error. OptR decomposes the post-$W_O$ attention-output error into key- and value-induced terms and learns per-head orthogonal corrections through the full INT2 quantization and attention path. OptR further applies an attention-equivalent key reparameterization to reduce large channel-wise offsets without changing the softmax distribution. Across three models and five reasoning and coding benchmarks, OptR consistently improves both QuaRot and OSCAR and strengthens long-context retrieval, while preserving the paged KV-cache format with negligible inference overhead."},"title":{"value":"Output-Aware Rotation for INT2 KV-Cache Quantization"},"authors":{"value":[{"fullname":"Vincent-Daniel Yun","username":""},{"fullname":"Woosang Lim","username":"~Woosang_Lim3"},{"fullname":"Minsoo Cheong","username":""},{"fullname":"Sunwoo Lee","username":""},{"fullname":"Murali Annavaram","username":""},{"fullname":"Sai Praneeth Karimireddy","username":""},{"fullname":"Sungjoo Yoo","username":""}]}},"tmdate":1791790697490,"pdate":1798675200000,"externalIds":["dblp:journals/corr/abs-2608-02691"],"tcdate":1791790693640,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Woosang_Lim3"],"forum":"4GxQCAgarY","license":"CC BY-SA 4.0","number":172582,"cdate":1785542400000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1791790697490,"domain":"OpenReview.net/Public_Article","id":"4GxQCAgarY","version":2},{"content":{"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["Large Language Models","Multi-Agent Systems","KV Cache Reuse","Efficient Inference"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Multi-agent large language model (LLM) systems frequently process messages generated by other agents. When agents share a model, the sender’s key–value (KV) cache can be transferred to avoid redundant prefilling, but the imported cache is context-dependent and may not match the receiver’s context. Moreover, the receiver may need only part of the message during decoding. KV handoff therefore poses two receiver-side resource decisions: where to spend recomputation and which transferred entries to keep in memory. We present AttuneKV, a receiver-aware KV-cache handoff method that obtains both decisions from one lightweight receiver observation. AttuneKV forwards one existing receiver-side token over the imported cache. Its layerwise attention ranks message positions for recomputation, while the attention-output change induced by removing each entry ranks positions for retention. The same observation supports both decisions without forwarding the incoming message positions for selection or maintaining a persistent KV reference pool. At 12,288 incoming tokens, AttuneKV achieves a 2.92× receiver time to first token (TTFT) speedup over dense prefill and a 1.45× speedup over RelayCaching. It also reduces KV-handoff memory by 2.0× and 34× relative to dense prefill and KVCOMM, respectively. Across math problem solving, broad-domain knowledge evaluation, and code generation, AttuneKV remains within 3.92 pp of dense prefill performance."},"_bibtex":{"value":"@inproceedings{\nanonymous2026attunekv,\ntitle={Attune{KV}: Receiver-Aware {KV}-Cache Handoff for Efficient Multi-Agent Inference},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=PfxIWVHXl0},\nnote={under review}\n}"},"title":{"value":"AttuneKV: Receiver-Aware KV-Cache Handoff for Efficient Multi-Agent Inference"},"pdf":{"value":"/pdf/862a38a04b91a4decb85cdd42614aa5b41a0d00c.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791234204254,"tcdate":1789766691711,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission49077/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission49077/Authors"],"forum":"PfxIWVHXl0","license":"CC BY 4.0","number":49077,"cdate":1789766691711,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission49077/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791234204254,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"PfxIWVHXl0","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2412.03213v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"liu|clusterkv_manipulating_llm_kv_cache_in_semantic_space_for_recallable_compression"},"authorids":{"value":["~Guangda_Liu1","~Chengwei_Li1","~Jieru_Zhao1","https://dblp.org/search/pid/api?q=author:Chenqi_Zhang:","~Minyi_Guo1"]},"html":{"value":"https://doi.org/10.48550/arXiv.2412.03213"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2412-03213,\n publtype={informal},\n author={Guangda Liu and Chengwei Li and Jieru Zhao and Chenqi Zhang and Minyi Guo},\n title={ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression},\n year={2024},\n cdate={1704067200000},\n journal={CoRR},\n volume={abs/2412.03213},\n url={https://doi.org/10.48550/arXiv.2412.03213}\n}\n"},"abstract":{"value":"Large Language Models (LLMs) have been widely deployed in a variety of applications, and the context length is rapidly increasing to handle tasks such as long-document QA and complex logical reasoning. However, long context poses significant challenges for inference efficiency, including high memory costs of key-value (KV) cache and increased latency due to extensive memory accesses. Recent works have proposed compressing KV cache to approximate computation, but these methods either evict tokens permanently, never recalling them for later inference, or recall previous tokens at the granularity of pages divided by textual positions. Both approaches degrade the model accuracy and output quality. To achieve efficient and accurate recallable KV cache compression, we introduce ClusterKV, which recalls tokens at the granularity of semantic clusters. We design and implement efficient algorithms and systems for clustering, selection, indexing and caching. Experiment results show that ClusterKV attains negligible accuracy loss across various tasks with 32k context lengths, using only a 1k to 2k KV cache budget, and achieves up to a 2$\\times$ speedup in latency and a 2.5$\\times$ improvement in decoding throughput. Compared to SoTA recallable KV compression methods, ClusterKV demonstrates higher model accuracy and output quality, while maintaining or exceeding inference efficiency."},"title":{"value":"ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression"},"authors":{"value":["Guangda Liu","Chengwei Li","Jieru Zhao","Chenqi Zhang","Minyi Guo"]}},"tmdate":1747274451051,"pdate":1704067200000,"tcdate":1741016812028,"writers":["~"],"signatures":["~Jieru_Zhao1"],"forum":"TlvJijpNYH","license":"CC BY-SA 4.0","number":346469,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1747274451051,"domain":"DBLP.org","id":"TlvJijpNYH","version":2},{"content":{"venue":{"value":"CoRR 2026"},"pdf":{"value":"https://arxiv.org/pdf/2609.10266v2"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"shi|kvsharearena_kvcache_reuse_across_contexts_and_model_checkpoints"},"html":{"value":"https://doi.org/10.48550/arXiv.2609.10266"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2609-10266,\n publtype={informal},\n author={Xi Shi and Mengxin Zheng and Qian Lou},\n title={KVShareArena: KV-Cache Reuse Across Contexts and Model Checkpoints},\n year={2026},\n month={September},\n cdate={1788220800000},\n journal={CoRR},\n volume={abs/2609.10266},\n url={https://doi.org/10.48550/arXiv.2609.10266}\n}\n"},"abstract":{"value":"Reusing key-value (KV) caches speeds up LLM inference by avoiding repeated computation on shared text. Standard prefix caching reuses a KV cache only when the LLM is the same and all preceding text is identical, but real workloads often break both conditions: RAG systems place different documents before the same one, agents with different system prompts read the same file or tool output, multi-agent workflows use specialized LLMs on shared material, and an updated model reads documents cached by its previous version. Because KV caches depend on both the preceding text and the model weights, direct reuse can reduce answer quality. Many methods repair or compress the reused cache, but each paper uses its own tasks, models, and cost measures, and existing benchmarks mainly test long-context processing or reuse of an unchanged prefix. We introduce KVShareArena, a benchmark and open evaluation framework for comparing them under the same conditions. KVShareArena has (1) reuse tests on 2,150 questions from three QA datasets, where the preceding text, the cache-writing LLM, or both change while the answering LLM and input stay fixed; (2) five dense and mixture-of-experts LLMs (4B-30B) and six LLM pairs where one version of an LLM reads caches written by another, for 33 model-dataset settings; (3) 11 repair and compression methods from six method classes; (4) four evaluation perspectives: answer quality, prefill computation, KV-cache memory, and latency; and (5) a common interface for adding new methods and an interactive leaderboard. Experiments yield two findings. First, both the quality loss from reuse and which repairs help depend on the LLM, even between two 8B models. Second, most repairs keep their quality when another LLM version wrote the cache, but a trained repair adapter loses quality in 12 of 18 pair-dataset tests. Code and data: https://github.com/xishi404/KVShare-Arena"},"title":{"value":"KVShareArena: KV-Cache Reuse Across Contexts and Model Checkpoints"},"authors":{"value":[{"fullname":"Xi Shi","username":"~Xi_Shi2"},{"fullname":"Mengxin Zheng","username":""},{"fullname":"Qian Lou","username":""}]}},"tmdate":1791858197345,"pdate":1798675200000,"externalIds":["dblp:journals/corr/abs-2609-10266"],"tcdate":1791858194446,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Xi_Shi2"],"forum":"TDZNVAZ2qi","license":"CC BY-SA 4.0","number":174662,"cdate":1788220800000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1791858197345,"domain":"OpenReview.net/Public_Article","id":"TDZNVAZ2qi","version":2},{"content":{"summary":{"value":"The authors propose a form of cache compression based on converting the tokens in the cache into the frequency domain for compressed storage. This results in lower storage requirements, lower attention complexity in decoding, and a kind of natural context extension since positional embeddings since the compression projects the cache into a fixed cardinality which will not expand to use up all of the positional embeddings."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"For a clearer understanding of how the DCT and IDCT matrices transform the inputs, can you add the dimensions for both the DCT and IDCT matrices?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"- KV cache compression is an important topic due to the efficiency gains and power consumption concerns of modern transformers.\n\n- In addition to compression the cache, the method also seems to cause an efficiency gain in the decoding attention operation."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- Llama 3 is used as a baseline model. This is important because I believe the only reason some baselines show poor performance is because they have exceeded the number of training positional embeddings. However, Llama 3 is already 1.5 years old at this point. There are already 3 releases of Llama3 which go up to 3.3 and have 131K native positional embeddings. Can FreqKV be applied to these models and show the same good performance past 131K?\n\n- There is no comparison of latency with baselines such as SnapKV.\n\n---\n\nOverall, it would be more convincing if the authors could provide a case on what happens with extremely long contexts. Due to the compressive nature of the cache, it may only be able to hold information up to a certain point before the compression becomes noise. However, if the same trend witnessed at 8K-->16K can be witnessed on a 131K-->232K model, I would find this very compelling. This could be done with the exact same experimental setup and swapping in the Llama 3.x models."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762923027305,"tcdate":1761206725040,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission12053/Reviewer_phR3"],"signatures":["ICLR.cc/2026/Conference/Submission12053/Reviewer_phR3"],"forum":"wFSOtyvQ9d","number":1,"license":"CC BY 4.0","cdate":1761206725040,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission12053/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762923027305,"domain":"ICLR.cc/2026/Conference","replyto":"wFSOtyvQ9d","id":"76XHV2FKpR","forumContent":{"TLDR":{"value":"This paper introduces FreqKV, an efficient context extension method that iteratively compresses key-value states in the frequency domain."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Large Language Models","KV Compression","Context Extension"]},"supplementary_material":{"value":"/attachment/aeaaca5a8ccf085d42756b47515d691b14f989d6.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Existing key-value (KV) cache compression methods for large language models (LLMs) often rely on token eviction, which risks losing critical local information in both long prefilling and decoding scenarios. When extrapolating beyond the pretrained context length, their performance degrades sharply on long-context benchmarks. Motivated by the observation in the frequency domain that the context information is concentrated in the low-frequency components, we propose FreqKV, a parameter-free and architecture-agnostic approach. It iteratively compresses the increasing KV cache in the frequency domain, allowing models to process lengthy contexts efficiently. With minimal training at 8K length, FreqKV extends the context window of LLaMA-2-7B up to 256K tokens while maintaining stable perplexity. Extensive experiments on both prefilling and decoding stages demonstrate that FreqKV enables robust context window extension and consistently outperforms existing KV cache compression methods, highlighting its effectiveness for both understanding and generation in long contexts."},"_bibtex":{"value":"@inproceedings{\nkai2026freqkv,\ntitle={Freq{KV}: Key-Value Compression in Frequency Domain for Context Window Extension},\nauthor={Jushi Kai and Yixuan Wang and Boyi Zeng and Haoli Bai and Bo Jiang and Ziwei He and Zhouhan Lin},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=wFSOtyvQ9d}\n}"},"title":{"value":"FreqKV: Key-Value Compression in Frequency Domain for Context Window Extension"},"pdf":{"value":"/pdf/f8b008c90190b10f54c5e7143d744efd3d39e01c.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"kai|freqkv_keyvalue_compression_in_frequency_domain_for_context_window_extension"},"authorids":{"value":["~Jushi_Kai1","~Yixuan_Wang10","~Boyi_Zeng2","~Haoli_Bai2","~Bo_Jiang2","~Ziwei_He1","~Zhouhan_Lin1"]},"authors":{"value":["Jushi Kai","Yixuan Wang","Boyi Zeng","Haoli Bai","Bo Jiang","Ziwei He","Zhouhan Lin"]}},"version":2},{"content":{"venue":{"value":"ACL ARR 2026 October Submission"},"research_area_keywords":{"value":"LLM efficiency, pruning, inference methods"},"keywords":{"value":["Large Language Models","Long-Context Inference","KV Cache Compression","Token Eviction","Efficient Inference"]},"_bibtex":{"value":"@inproceedings{\nanonymous2026disco,\ntitle={{DISCO}: Dispersion-Guided Sparse {KV} Cache Compression for Efficient Long-Context Inference in Large Language Models},\nauthor={Anonymous},\nbooktitle={Submitted to ACL Rolling Review - October 2026},\nyear={2026},\nurl={https://openreview.net/forum?id=278XuNWPdY},\nnote={under review}\n}"},"title":{"value":"DISCO: Dispersion-Guided Sparse KV Cache Compression for Efficient Long-Context Inference in Large Language Models"},"languages_studied":{"value":["eng","zho"]},"contribution_types":{"value":"Approaches to low-compute settings (efficiency)"},"abstract":{"value":"Long-context inference in LLMs is constrained by the KV cache, whose size grows linearly with sequence length and depth. Existing compression methods rely on attention-derived importance or online allocation, which over-emphasize attention sinks, under-protect mid-context evidence, and add inference overhead. \nWe propose DISCO, a dispersion-guided sparse KV cache compression framework that uses\nmodel-specific dispersion profiles as empirical proxies for compression sensitivity. DISCO estimates these profiles offline via PCA, allocates non-uniform layer-wise budgets, and performs online dispersion-preserving token selection by combining attention relevance with centroid-based geometric contribution. Experiments on long-context understanding, retrieval, efficiency, and medical reasoning show consistent performance-efficiency gains under fixed KV budgets. Calibration analyses further show stable dispersion profiles under small calibration sets and moderate domain shifts, supporting DISCO as lightweight and task-agnostic."},"paper_type":{"value":"Long"},"pdf":{"value":"/pdf/9c7270a3a30338c9f104a5fc35acf21715ac5e24.pdf"},"research_area":{"value":"Efficient Methods for NLP"},"venueid":{"value":"aclweb.org/ACL/ARR/2026/October/Submission"}},"tmdate":1792185033691,"tcdate":1791879044177,"writers":["aclweb.org/ACL/ARR/2026/October","aclweb.org/ACL/ARR/2026/October/Submission10248/Authors"],"signatures":["aclweb.org/ACL/ARR/2026/October/Submission10248/Authors"],"forum":"278XuNWPdY","license":"CC BY 4.0","number":10248,"cdate":1791879044177,"readers":["everyone"],"invitations":["aclweb.org/ACL/ARR/2026/October/-/Submission","aclweb.org/ACL/ARR/2026/October/-/Responsible_NLP_Checklist_Status","aclweb.org/ACL/ARR/2026/October/-/Edit","aclweb.org/ACL/ARR/2026/October/-/Preprint_Post_Submission"],"mdate":1792185033691,"odate":1791894541772,"domain":"aclweb.org/ACL/ARR/2026/October","id":"278XuNWPdY","version":2},{"content":{"venue":{"value":"Submitted to ICLR 2026"},"TLDR":{"value":"We propose ForesightKV, a training-based KV cache eviction method for long-context reasoning tasks, which consists of supervised training and reinforcment learning."},"keywords":{"value":["KV cache eviction","long-context reasoning","reinforcement learning"]},"supplementary_material":{"value":"/attachment/83b3fb008b3fd1b4ea76dcf0aec203ed94697217.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Recently, large language models (LLMs) have shown remarkable reasoning abilities by producing long reasoning traces. However, as the sequence length grows, the key-value (KV) cache expands linearly, incurring significant memory and computation costs. Existing cache eviction methods mitigate this issue by discarding less important KV pairs, but often fail to capture complex KV dependencies, resulting in performance degradation. To better balance efficiency and performance, we introduce ForesightKV, a training-based KV cache eviction framework that learns to predict which KV pairs to evict during long-text generations. We first design the Golden Eviction algorithm, which identifies the optimal eviction KV pairs at each step using future attention scores. These traces and the scores at each step are then distilled via supervised training with a Pairwise Ranking Loss. Furthermore, we formulate cache eviction as a Markov Decision Process and apply the GRPO algorithm to mitigate the significant language modeling loss increase on low-entropy tokens. Experiments on AIME2024 and AIME2025 benchmarks with Qwen3-1.7B and Qwen3-4B demonstrate that ForesightKV consistently outperforms prior methods under only half the cache budget, while benefiting synergistically from both supervised and reinforcement learning approaches."},"_bibtex":{"value":"@misc{\ndong2026foresightkv,\ntitle={Foresight{KV}: Optimizing {KV} Cache Eviction for Reasoning Models by Learning Long-Term Contribution},\nauthor={Zican Dong and Peiyu Liu and Junyi Li and Zhipeng Chen and Han Peng and Shuo Wang and Xin Zhao},\nyear={2026},\nurl={https://openreview.net/forum?id=k5h29Qv7GN}\n}"},"title":{"value":"ForesightKV: Optimizing KV Cache Eviction for Reasoning Models by Learning Long-Term Contribution"},"pdf":{"value":"/pdf/6957849ec917fb6c85af208e505300924f9aa93b.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"dong|foresightkv_optimizing_kv_cache_eviction_for_reasoning_models_by_learning_longterm_contribution"},"authorids":{"value":["~Zican_Dong2","~Peiyu_Liu1","~Junyi_Li4","~Zhipeng_Chen2","~Han_Peng3","~Shuo_Wang13","~Xin_Zhao10"]},"authors":{"value":["Zican Dong","Peiyu Liu","Junyi Li","Zhipeng Chen","Han Peng","Shuo Wang","Xin Zhao"]}},"tmdate":1770804848463,"tcdate":1757644798906,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission4237/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission4237/Authors"],"forum":"k5h29Qv7GN","license":"CC BY 4.0","number":4237,"cdate":1757644798906,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/-/Submission","ICLR.cc/2026/Conference/-/Post_Submission","ICLR.cc/2026/Conference/Submission4237/-/Full_Submission","ICLR.cc/2026/Conference/Submission4237/-/Rebuttal_Revision","ICLR.cc/2026/Conference/-/Edit"],"mdate":1770804848463,"odate":1759896705795,"domain":"ICLR.cc/2026/Conference","id":"k5h29Qv7GN","version":2},{"content":{"summary":{"value":"This paper introduces dLLM-Cache, a novel training-free framework designed to accelerate\ninference in diffusion LLMs. It proposes a dual-component caching strategy. First, it\nemploys long-interval caching for the static input prompt, refreshing its features only\ninfrequently. Second, it uses an adaptive, short-interval caching mechanism for the\ndynamically evolving response. This adaptive component is guided by a lightweight "V-\nverify" mechanism. The paper demonstrates through extensive experiments on LLaDA and\nDream models that dLLM-Cache can achieve up to a 9.1x speedup, bringing dLLM inference\nlatency closer to that of autoregressive models without compromising output quality."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"See weakness."},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. The paper addresses a critical and widely recognized problem: the high inference\ncost of diffusion LLMs. The achieved speedups are substantial and represent a major\nstep towards making these models practical for real-world applications.\n\n2. The V-verify mechanism is a clever, lightweight, and empirically-grounded solution\nfor adaptively selecting which tokens to update, avoiding the overhead of more\ncomplex methods.\n3. A key strength is that dLLM-Cache does not require any model retraining. This\nmakes the method easy to adopt."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The high-level diagrams provide a helpful overview of the caching workflow.\nHowever, the specific structure of the cache, particularly whether it is a single global\nentity or maintained on a per-layer basis, is not explicitly detailed.\n2. The paper could better justify the necessity of caching four feature types (K, V,\nAttnOut, FFNOut), as this is a departure from the more standard KV-caching in\nARMs.\n3. It is unclear why some configurations, such as MMLU with LLaDA Instruct, benefit\nless from caching than the Base model. An analysis for the variance in speedup\nacross different benchmarks may be beneficial."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762924082855,"tcdate":1761870259854,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission13465/Reviewer_BFE1"],"signatures":["ICLR.cc/2026/Conference/Submission13465/Reviewer_BFE1"],"forum":"DriG3hgh42","number":1,"license":"CC BY 4.0","cdate":1761870259854,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission13465/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762924082855,"domain":"ICLR.cc/2026/Conference","replyto":"DriG3hgh42","id":"JfHVlgfwmh","forumContent":{"TLDR":{"value":"We accelerate diffusion LLMs with a training-free adaptive caching strategy, reusing prompt and response features to significantly reduce inference latency."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Diffusion Large Language Models","Model Acceleration","Adaptive Caching"]},"supplementary_material":{"value":"/attachment/7fbf8792aa1a3f3f392030e87abf6d6ef0aa7d3e.zip"},"primary_area":{"value":"generative models"},"abstract":{"value":"Autoregressive Models (ARMs) have long dominated the landscape of Large Language Models. Recently, a new paradigm has emerged in the form of diffusion-based Large Language Models (dLLMs), which generate text by iteratively denoising masked segments. This approach has shown significant advantages and potential. However, dLLMs suffer from high inference latency. Traditional ARM acceleration techniques, such as Key-Value caching, are incompatible with dLLMs due to their bidirectional attention mechanism. To address this specific challenge, our work begins with a key observation that dLLM inference involves a static prompt and a partially dynamic response, where most tokens remain stable across adjacent denoising steps. Based on this, we propose dLLM-Cache, a training-free adaptive caching framework that combines long-interval prompt caching with partial response updates guided by feature similarity. This design enables efficient reuse of intermediate computations without compromising model performance. Extensive experiments on representative dLLMs, including LLaDA 8B and Dream 7B, show that dLLM-Cache achieves up to *9.1*$\\times$ speedup over standard inference without compromising output quality. Notably, our method brings dLLM inference latency close to that of ARMs under many settings. *Codes are provided in the supplementary material and will be released publicly on GitHub.*"},"_bibtex":{"value":"@misc{\nliu2026dllmcache,\ntitle={d{LLM}-Cache: Accelerating Diffusion Large Language Models with Adaptive Caching},\nauthor={Zhiyuan Liu and Yicun Yang and Yaojie Zhang and Junjie Chen and Chang Zou and Qingyan Wei and Shaobo Wang and Yichen Zhu and Linfeng Zhang},\nyear={2026},\nurl={https://openreview.net/forum?id=DriG3hgh42}\n}"},"title":{"value":"dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Caching"},"pdf":{"value":"/pdf/bbb0d922478ec6b430e560769f62a1cb1fcbcc0e.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"liu|dllmcache_accelerating_diffusion_large_language_models_with_adaptive_caching"},"authorids":{"value":["~Zhiyuan_Liu15","~Yicun_Yang1","~Yaojie_Zhang2","~Junjie_Chen9","~Chang_Zou2","~Qingyan_Wei1","~Shaobo_Wang1","~Yichen_Zhu1","~Linfeng_Zhang2"]},"authors":{"value":["Zhiyuan Liu","Yicun Yang","Yaojie Zhang","Junjie Chen","Chang Zou","Qingyan Wei","Shaobo Wang","Yichen Zhu","Linfeng Zhang"]}},"version":2},{"content":{"summary":{"value":"The paper proposes DimPO, a reference-free list-wise preference-optimization loss that learns a single linear projection to compress query/key vectors while keeping the attention distribution essentially unchanged. Applied to the last few layers of LLaMA-3 (1\\~8 B) and Qwen-2.5/3 (4\\~7 B) models, DimPO yields 10–15 % KV-cache memory reduction with ≤5 % average drop on five short-context benchmarks and comparable or slightly larger degradation on long-context suites. It outperforms triplet, PCA, random-projection, and existing pairwise preference losses (SimPO, ORPO, CPO) by 2–3× in KL divergence between original and projected attention weights."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1. Why restrict the projection to a single shared linear matrix per layer? Did you try head-specific or unshared version that might yield higher compression with similar parameter count?\n\n2. How does DimPO behave when combined with 4-bit KV quantization—do the errors compound multiplicatively?\n\n3. Did you explore non-uniform layer-wise allocation of dimensions (e.g., d'=32 for early layers, d'=8 for final ones) to push compression further where attention is most robust?\n\n4. Are there attention-pattern failure modes (e.g., many-shot retrieval, long code files) where DimPO degrades catastrophically?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. Novel framing: first to treat attention dimensionality reduction as a preference-ranking problem rather than hashing or reconstruction, motivating a natural list-wise objective.\n\n2. DimPO loss is elegantly reference-free, avoids the quadratic explosion of pairwise comparisons, and consistently beats strong baselines.\n\n3. Thorough experimental sweep: ablates dimension, number of projected layers, and model scale; reports both attention-quality metrics (KL, MSE) and downstream task scores; includes long-context benchmarks and throughput measurements.\n\n4. Practical impact: a 10% cache cut translates directly into larger batch sizes or longer contexts for serving; the projection adds only a small linear layer with negligible latency."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Scope of evaluation: all experiments compress only the last few layers; no systematic study of how layer depth interacts with sensitivity (only a top-down heuristic).\n\n2. Task coverage: long-context evaluation is restricted to 4k and 8k RULER tasks and up to 65 k LongBench; no examination of >100 k tokens or code/documentation datasets where attention patterns differ.\n\n3. Baseline omissions: no comparison against simultaneous KV-quantization (e.g., 4-bit KIVI) or low-rank factorization of the full attention matrix leaving the combined settings unclear."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762999999065,"tcdate":1761894063846,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission21616/Reviewer_4Q9G"],"signatures":["ICLR.cc/2026/Conference/Submission21616/Reviewer_4Q9G"],"forum":"LlhiiTg9ux","number":3,"license":"CC BY 4.0","cdate":1761894063846,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission21616/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762999999065,"domain":"ICLR.cc/2026/Conference","replyto":"LlhiiTg9ux","id":"S0iar6DTL8","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"TLDR":{"value":"We introduce DimPO, a novel listwise preference optimization loss for dimensionality reduction of attention that reduces KV cache memory in LLMs by 10-15% with minimal performance loss."},"keywords":{"value":["preference optimization","dimensionality reduction","kv cache optimization"]},"supplementary_material":{"value":"/attachment/d98ee1e9924258d7d382b94b4f80fba7bc30ad85.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Large Language Models (LLMs) require increasing memory and computational resources to process long context, motivating methods that reduce the dimensionality of key and query representations while preserving the resulting attention patterns. A previous triplet-based method for LSH-style projections optimizes only local pairwise relations and therefore struggles to capture the global ranking structure inherent to attention. We reframe dimensionality reduction of attention as a listwise learning problem based on preference optimization and ranking losses. Our analysis shows that listwise approaches better preserve full attention distributions in low-dimensional spaces, reducing KL divergence to less than 25\\% of the value achieved by triplet-based and other pairwise methods. Building on this, we directly apply listwise-learned projections to the attention layers of LLaMA3-[1B, 3B, 8B], Qwen2.5-7B and Qwen3-4B instruct models. On general benchmark tasks, listwise projection reduces KV cache memory by 10–15% while maintaining 95% of model performance. Larger models using this projection on long-context tasks exhibit only a marginal performance drop and demonstrate compatibility with existing KV-cache reduction techniques such as SnapKV, improving throughput without appreciable accuracy degradation. These results establish listwise learning as a principled and effective approach for dimensionality reduction of attention, directly applicable to attention layers at inference."},"_bibtex":{"value":"@misc{\nlanz2026dimpo,\ntitle={Dim{PO}: Dimensionality Reduction for Attention using Preference Optimization},\nauthor={Vojt{\\v{e}}ch Lanz and Yufei Cui and Prasanna Parthasarathi},\nyear={2026},\nurl={https://openreview.net/forum?id=LlhiiTg9ux}\n}"},"title":{"value":"DimPO: Dimensionality Reduction for Attention using Preference Optimization"},"pdf":{"value":"/pdf/fb177a2a7456bc6052a0ea176bef1c8aec88cff7.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"lanz|dimpo_dimensionality_reduction_for_attention_using_preference_optimization"},"authorids":{"value":["~Vojtěch_Lanz1","~Yufei_Cui3","~Prasanna_Parthasarathi2"]},"authors":{"value":["Vojtěch Lanz","Yufei Cui","Prasanna Parthasarathi"]}},"version":2},{"content":{"summary":{"value":"The DuoAttention uses the retrieval heads and local streaming heads, realizing the KV cache compression and acceleration:\n1) the retrieval heads preserve the major information of the long-context, which can not be compressed;\n2) the non-retrieval heads are compressed by locality, which is accelerated with StreamingLLM;\n3) the retrieval heads are detected with importance learnable weights;\n4) Finally, the prefill is accelerated by chunked prefilling, and the decoding is accelerated with compressed KV cache."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"1. the offline method to find the retrieval heads is more important for practical application\n2. the compression ratio of KV cache should be improved"},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":3},"strengths":{"value":"1. the experiments is enough and confident\n2. the writing and clarity are clear\n3. the concept of retrieval heads is significant for static KV cache compression"},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. the detection of retrieval heads is based on the fine-tuning of importance weights, the usability should be enhanced\n2. the compression ratio is limited, if the retrieval heads preserve the full context information"}},"nonreaders":[],"tmdate":1731428317010,"tcdate":1730367491667,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission5112/Reviewer_n7ZA"],"signatures":["ICLR.cc/2025/Conference/Submission5112/Reviewer_n7ZA"],"forum":"cFu7ze7xUm","number":2,"license":"CC BY 4.0","cdate":1730367491667,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission5112/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428317010,"domain":"ICLR.cc/2025/Conference","replyto":"cFu7ze7xUm","id":"sqyAWlkIxt","forumContent":{"venue":{"value":"ICLR 2025 Poster"},"TLDR":{"value":"We significantly reduce both pre-filling and decoding memory and latency for long-context LLMs without sacrificing their long-context abilities."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Large Language Models; Long Context; Efficiency;"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Deploying long-context large language models (LLMs) is essential but poses significant computational and memory challenges.\nCaching all Key and Value (KV) states across all attention heads consumes substantial memory.\nExisting KV cache pruning methods either damage the long-context capabilities of LLMs or offer only limited efficiency improvements.\nIn this paper, we identify that only a fraction of attention heads, a.k.a, Retrieval Heads, are critical for processing long contexts and require full attention across all tokens.\nIn contrast, all other heads, which primarily focus on recent tokens and attention sinks—referred to as Streaming Heads—do not require full attention.\nBased on this insight, we introduce DuoAttention, a framework that only applies a full KV cache to retrieval heads while using a light-weight, constant-length KV cache for streaming heads, which reduces both LLM's decoding and pre-filling memory and latency without compromising its long-context abilities.\nDuoAttention uses a lightweight, optimization-based algorithm with synthetic data to identify retrieval heads accurately.\nOur method significantly reduces long-context inference memory by up to 2.55$\\times$ for MHA and 1.67$\\times$ for GQA models while speeding up decoding by up to 2.18$\\times$ and 1.50$\\times$ and accelerating pre-filling by up to 1.73$\\times$ and 1.63$\\times$ for MHA and GQA models, respectively, with minimal accuracy loss compared to full attention.\nNotably, combined with quantization, DuoAttention enables Llama-3-8B decoding with 3.33 million context length measured on a single A100 GPU. Code is provided in https://github.com/mit-han-lab/duo-attention."},"_bibtex":{"value":"@inproceedings{\nxiao2025duoattention,\ntitle={DuoAttention: Efficient Long-Context {LLM} Inference with Retrieval and Streaming Heads},\nauthor={Guangxuan Xiao and Jiaming Tang and Jingwei Zuo and junxian guo and Shang Yang and Haotian Tang and Yao Fu and Song Han},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=cFu7ze7xUm}\n}"},"title":{"value":"DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads"},"pdf":{"value":"/pdf/5723b4f3ab2bb241158f3f35ad3ac5b22b62192e.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"xiao|duoattention_efficient_longcontext_llm_inference_with_retrieval_and_streaming_heads"},"authorids":{"value":["~Guangxuan_Xiao1","~Jiaming_Tang1","~Jingwei_Zuo2","~junxian_guo1","~Shang_Yang1","~Haotian_Tang1","~Yao_Fu3","~Song_Han5"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Guangxuan Xiao","Jiaming Tang","Jingwei Zuo","junxian guo","Shang Yang","Haotian Tang","Yao Fu","Song Han"]}},"version":2},{"content":{"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["KV Cache; Long-Context Inference;Attention Preservation"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"During autoregressive decoding, the KV cache supports queries that may depend on different parts of a long context. Under a limited memory budget, selecting tokens by aggregate attention can leave some queries poorly covered even when the total retained attention is high. We study this imbalance through a distributional formulation of cache eviction, measuring query coverage by the attention probability mass assigned to retained tokens. For fixed queries and keys, restricting attention to a retained token subset is equivalent to conditioning the original attention distribution on that subset. The KL divergence from the conditioned distribution to the original distribution equals the negative logarithm of the retained probability mass, yielding an objective that maximizes the geometric mean of query coverage. Its marginal scores give greater weight to attention from queries with less retained mass. We instantiate this principle as FidKV, a lightweight online eviction policy that estimates retained attention mass from the current cache without additional training or calibration. Our analysis bounds changes in token rankings by the spread of retained mass across queries, recovering cumulative-attention rankings when coverage is uniform. Experiments on LongBench and RULER across three language-model families support this prediction. On six held-out LongBench tasks, the advantage of FidKV is most pronounced under aggressive compression: at 2% and 5% cache budgets, it outperforms the strongest evaluated baseline on most tasks, while remaining competitive at moderate budgets., respectively. Paired cache measurements further show improved coverage for the worst-served queries and reduced coverage imbalance, while deployment retains the memory and latency profile of standard KV eviction."},"_bibtex":{"value":"@inproceedings{\nanonymous2026balancing,\ntitle={Balancing Query Coverage in {KV} Cache Compression},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=hyTorr2WXN},\nnote={under review}\n}"},"title":{"value":"Balancing Query Coverage in KV Cache Compression"},"pdf":{"value":"/pdf/788e2d57cbed2aa936b1da905f5389ee37aa2806.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791232101073,"tcdate":1789701741733,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission36439/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission36439/Authors"],"forum":"hyTorr2WXN","license":"CC BY 4.0","number":36439,"cdate":1789701741733,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission36439/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791232101073,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"hyTorr2WXN","version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"http://arxiv.org/pdf/2503.14647v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"li|towards_more_economical_contextaugmented_llm_generation_by_reusing_stored_kv_cache"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Hanchen_Li:","~Yuhan_Liu3","https://dblp.org/search/pid/api?q=author:Yihua_Cheng:","~Kuntai_Du1","https://dblp.org/search/pid/api?q=author:Junchen_Jiang:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2503.14647"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2503-14647,\n publtype={informal},\n author={Hanchen Li and Yuhan Liu and Yihua Cheng and Kuntai Du and Junchen Jiang},\n title={Towards More Economical Context-Augmented LLM Generation by Reusing Stored KV Cache},\n year={2025},\n month={March},\n cdate={1740787200000},\n journal={CoRR},\n volume={abs/2503.14647},\n url={https://doi.org/10.48550/arXiv.2503.14647}\n}\n"},"abstract":{"value":"Across large language model (LLM) applications, we observe an emerging trend for reusing KV caches to save the prefill delays of processing repeated input texts in different LLM inputs. This has led to a broad design space, including colocating stored KV caches with (or close to) GPUs to various KV cache compression. However, a key question remains unanswered: can these delay reductions also be economically favorable? Specifically, we ask whether a developer can use public cloud services to store precomputed KV caches and reuse them to save delay without incurring more costs in terms of compute, storage, and network. To answer this question, we propose an validated analytical model for the cloud cost (in compute, storage, and network) of storing and reusing KV caches based on various workload parameters, such as reuse frequency, generated text lengths, model sizes, etc. Preliminary results show that KV cache reusing is able to save both delay and cloud cost across a range of workloads with long context. And we call more efforts on building more economical context augmented LLM by KV cache reusing."},"title":{"value":"Towards More Economical Context-Augmented LLM Generation by Reusing Stored KV Cache"},"authors":{"value":["Hanchen Li","Yuhan Liu","Yihua Cheng","Kuntai Du","Junchen Jiang"]}},"tmdate":1769503793773,"pdate":1735689600000,"externalIds":["dblp:journals/corr/abs-2503-14647"],"tcdate":1758650621618,"writers":["~"],"signatures":["~Kuntai_Du1"],"forum":"svPagmFBM6","license":"CC BY-SA 4.0","number":627570,"cdate":1740787200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1769503793773,"domain":"DBLP.org","id":"svPagmFBM6","version":2},{"content":{"summary":{"value":"The authors propose GUI-KV, a smart and efficient KV-cache compression framework for GUI agents that deal with tons of screenshots and long tasks. Instead of retraining the models, GUI-KV just plugs into existing agents and makes them faster and lighter by pruning redundant cache states. The key idea is that GUI screenshots are full of repetitive visuals, so the paper introduces a spatio-temporal theorem-guided method: (1) spatial saliency guidance, which scores tokens based on their L2-norm (how much “information energy” they carry) and attention weights; and (2) temporal redundancy scoring, which uses QR decomposition to detect overlapping key subspaces between frames and drop repetitive tokens. This simple yet well-reasoned design leads to large speedups and even slight accuracy gains over full-cache models."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"How sensitive is GUI-KV’s performance to the rank r used in QR decomposition—does lowering r hurt accuracy, and can adaptive r per layer improve efficiency?\n\nCould the authors formally characterize conditions (beyond uniform sparsity) under which the uniform-budget theorem holds, and how it might adapt to multimodal domains with variable sparsity (e.g., GUI + video)?"},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"The paper builds an argument from the observation (Fig. 1, lines 108–215 and 162–215) that GUI attention sparsity is uniformly above 0.99 across all layers. This leads to a clean, theory-backed conclusion that uniform budget allocation is optimal for GUI workloads, overturning prior assumptions from methods like PyramidKV and VL-Cache that rely on varying per-layer budgets.\n\nhe proposed saliency + redundancy scoring (lines 270–323) ties practical algorithmic components to theoretical motivation. It’s easy to implement, works plug-and-play without retraining."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The core theorem and sparsity analysis (lines 162–215) hinge on an assumption of uniform high attention sparsity (> 0.99). This is empirically true for GUI screenshots but might not hold for other multimodal agents (e.g., video or natural-scene reasoning). The paper doesn’t clarify how GUI-KV behaves when this uniformity breaks down.\n\nThe temporal redundancy mechanism (lines 270–323) assumes low-rank subspaces (r ≪ dₕ), but the computational overhead of repeated QR per frame could become significant in real-time agents or larger contexts. While the authors mention negligible prefill cost (lines 432–485), no timing breakdown is provided for larger r or more screenshots.\n\nAlthough the authors reason that uniform budget allocation is “optimal,” there is no formal proof or convergence guarantee. The argument is empirical rather than mathematical, so the claimed “theorem-like” property is more observational than theoretically rigorous.\n\nMinor weakness: (1) Some parameter choices (e.g., α = 0.5, τ = 0.1 in Eq. 5) are not justified. (2) The interaction between spatial and temporal modules isn’t theoretically characterized (just shown empirically in Fig. 4)."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762933878846,"tcdate":1760861339712,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission20434/Reviewer_uck8"],"signatures":["ICLR.cc/2026/Conference/Submission20434/Reviewer_uck8"],"forum":"d9IJGziaLv","number":1,"license":"CC BY 4.0","cdate":1760861339712,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission20434/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762933878846,"domain":"ICLR.cc/2026/Conference","replyto":"d9IJGziaLv","id":"5lpIFNkuHH","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Vision Language Models; KV Cache Compression; GUI Agent"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"Graphical user interface (GUI) agents face severe efficiency bottlenecks when processing long sequences of high-resolution screenshots, making inference costly and memory-bound. Existing KV cache compression methods, designed for natural images, remain suboptimal as they fail to exploit the unique spatial and temporal redundancies of GUIs. In this work, we first demonstrate that unlike natural images, GUI attention sparsity is uniformly high (>0.99) across all transformer layers, invalidating complex layer-varying budget strategies. Building on this insight, we introduce GUI-KV, a training-free compression method that allocates a uniform budget driven by two novel mechanisms: (1) spatial saliency guidance, which augments attention with residual stream L2 norms to preserve semantic visual tokens; and (2) temporal redundancy scoring, which employs subspace projection to identify and prune historical frames that are linearly redundant with the current view. Across six benchmarks, GUI-KV outperforms competitive baselines, often recovering near-full-cache accuracy at 10-20% budgets. Notably, on AgentNetBench, it reduces decoding FLOPs by 38.9% while increasing step accuracy by 4.1% over the full-cache baseline."},"_bibtex":{"value":"@misc{\nhuang2026guikv,\ntitle={{GUI}-{KV}: Efficient {GUI} Agents via {KV} Cache with Spatio-Temporal Awareness},\nauthor={Kung-Hsiang Huang and Haoyi Qiu and Yutong Dai and Caiming Xiong and Chien-Sheng Wu},\nyear={2026},\nurl={https://openreview.net/forum?id=d9IJGziaLv}\n}"},"title":{"value":"GUI-KV: Efficient GUI Agents via KV Cache with Spatio-Temporal Awareness"},"pdf":{"value":"/pdf/6c3fd69dc6cafc71cd3404963bfedce535aca8b4.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"huang|guikv_efficient_gui_agents_via_kv_cache_with_spatiotemporal_awareness"},"authorids":{"value":["~Kung-Hsiang_Huang1","~Haoyi_Qiu1","~Yutong_Dai2","~Caiming_Xiong1","~Chien-Sheng_Wu1"]},"authors":{"value":["Kung-Hsiang Huang","Haoyi Qiu","Yutong Dai","Caiming Xiong","Chien-Sheng Wu"]}},"version":2},{"content":{"summary":{"value":"This paper proposes PT-Recformer, a novel transformer-like architecture for sequential recommendation that addresses two inefficiencies in current self-attention based models: inadequate temporal modeling and KV cache redundancy. The authors introduce Rotary Temporal Encoding (RoTE) to handle temporal variations in user interaction sequences (ranging from seconds to years), and Personalized Multi-head Latent Attention (PMLA) that integrates user profile features while reducing KV cache costs through low-rank compression and sliding window attention. The method is evaluated on three public benchmarks and deployed in a commercial system, showing improvements over SOTA baselines with as little as 12% of the KV cache storage and 75% training time of vanilla self-attention models."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"Please see above. Addressing issues in W2/W3/W4 would be especially helpful."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"S1: Problem motivation: The paper provides empirical evidence (Figure 1) showing that self-attention models focus predominantly on the last few items, leading to massive KV cache redundancy.\n\nS2. Technical contributions: The paper makes the following technical contributions, including a/ RoTE with learnable frequencies, an elegant solution for handling extreme temporal ranges in recommendation data, b/ Integration of personalized features directly into attention mechanisms (rather than late fusion), and c/ Combination of low-rank compression with sliding window attention to reduce KV cache sizes\n\nS3. Comprehensive experimental validations: a/ The authors conduct extensive experiments on three public benchmarks with consistent improvements, with thorough ablation studies (eg Table 4). b/ The authors deployed the solution in real-world settings, with A/B testing showing 3.13% eCPM improvement.\n\nS4. Overall, the method seems like a strong improvement over current SotA, with as little as 12% KV cache, 75% training time, and faster inference, which is significant for practical deployment."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"W1: Limited novelty in individual components: While the combination is novel, individual components (sliding window attention, low-rank compression, RoPE) are well-established techniques. The primary contribution of this paper appears to be in their adaptation and combination for recommendation.\n\nW2. Design\n* Despite observation of \"self-attention models focusing predominantly on the last few items\", the optimal hyperparameter for ML-20M is 100, suggesting that sliding window may not be that effective.\n* While PMLA is a step forward on top of MLA, the usage of user profile tokens can introduce leakage.\n* The temporal relative attention bias mechanism in HSTU can be combined with PMLA too. What are the tradeoffs of (temporal aware) RAB vs RoPE on top of PMLA?\n\nW3. Experiment comparisons\n* Table 1 comparison is misleading; we should not compare sliding window style attention and full attention in the same table, as one can easily integrate sliding window attention into various baselines (eg SASRec, HSTU, etc).\n* Table 3: 0.37B model being almost as fast as 1B HSTU models suggest PT-Recformer implementation may not be sufficiently optimized?\n* While the solution is deployed in production, additional details for baseline/test would be helpful - eg scale of baseline model, distribution of sequence lengths, model refresh frequencies, whether dataset consists of news-like items with shorter timespans or retail with seasonal patterns, etc. \n\nW4. Writing\n- Incorrect citations. https://arxiv.org/abs/2111.11294 was the first work to study scaling laws in recommendations. Zhang et al. (2023)'s work came much later.\n- Figure 1 and Figure 2(b) are hard to read\n- Table 2 suggests that the results are accurate up to .00001 Recall@K but I doubt this is the case. Why not report variance given you've already conducted multiple runs?\n- Mathematical notation could be cleaner\n- Typos: Time interval resolution factor β is set 6.7 -- ??\n- Formatting in various Tables could be improved (eg 7169083s in Table 9 should be converted to hours)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762919003149,"tcdate":1762653264497,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission6715/Reviewer_xc4X"],"signatures":["ICLR.cc/2026/Conference/Submission6715/Reviewer_xc4X"],"forum":"byotX3p7xN","number":4,"license":"CC BY 4.0","cdate":1762653264497,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission6715/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762919003149,"domain":"ICLR.cc/2026/Conference","replyto":"byotX3p7xN","id":"dtSK3fMln4","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Sequential Recommendation","Generative Recommendation"]},"supplementary_material":{"value":"/attachment/9641df9ef82abbed25af612adc135020e14af90e.zip"},"primary_area":{"value":"other topics in machine learning (i.e., none of the above)"},"abstract":{"value":"Recent discoveries of scaling laws in autoregressive and large generative recommendation models have attracted considerable attention in sequential recommendation. Larger models entail increased inference latency and training costs, while most recommendation applications have stringent latency constraints. Through empirical evaluations of recent self-attention-based recommendation models, we identified two critical issues that impair model performance while increasing computational costs: inadequate modeling of temporal information and KV cache redundancy. In this paper, we propose a novel method, termed Personalized and Temporal-aware Transformer for generative recommendation (PT-Recformer), to effectively mitigate these issues. On the one hand, we introduce a novel Rotary Temporal Encoding method to effectively model extreme temporal variations inherent in user contexts. On the other hand, we propose a personalized multi-head latent attention mechanism to enhance expressive power by integrating personalized and temporal features and significantly reduce KV cache costs. Comprehensive experiments conducted on several benchmark datasets demonstrate that PT-Recformer consistently outperforms state-of-the-art baselines with only 12\\% of the KV cache storage required by vanilla self-attention based models. Online A/B testing on a commercial platform has validated the effectiveness of the PT-Recformer in large-scale industrial settings, resulting in a 3.13\\% increase in effective Cost Per Mille. Our codes are available at \\url{https://anonymous.4open.science/r/GRec-B386}."},"_bibtex":{"value":"@misc{\njun2026personalized,\ntitle={Personalized and Temporal-aware Attention for Efficient Generative Recommendation},\nauthor={Yuan Jun},\nyear={2026},\nurl={https://openreview.net/forum?id=byotX3p7xN}\n}"},"title":{"value":"Personalized and Temporal-aware Attention for Efficient Generative Recommendation"},"pdf":{"value":"/pdf/fa9acc82ca4870eb8afd97326f145bb1fae262a5.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"jun|personalized_and_temporalaware_attention_for_efficient_generative_recommendation"},"authorids":{"value":["~Yuan_Jun1"]},"authors":{"value":["Yuan Jun"]}},"version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2502.14051v3"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"behnam|rocketkv_accelerating_longcontext_llm_inference_via_twostage_kv_cache_compression"},"html":{"value":"https://doi.org/10.48550/arXiv.2502.14051"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2502-14051,\n publtype={informal},\n author={Payman Behnam and Yaosheng Fu and Ritchie Zhao and Po-An Tsai and Zhiding Yu and Alexey Tumanov},\n title={RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression},\n year={2025},\n month={February},\n cdate={1738368000000},\n journal={CoRR},\n volume={abs/2502.14051},\n url={https://doi.org/10.48550/arXiv.2502.14051}\n}\n"},"abstract":{"value":"Transformer-based Large Language Models rely critically on the KV cache to efficiently handle extended contexts during the decode phase. Yet, the size of the KV cache grows proportionally with the input length, burdening both memory bandwidth and capacity as decoding progresses. To address this challenge, we present RocketKV, a training-free KV cache compression strategy containing two consecutive stages. In the first stage, it performs coarse-grain permanent KV cache eviction on the input sequence tokens. In the second stage, it adopts a hybrid sparse attention method to conduct fine-grain top-k sparse attention, approximating the attention scores by leveraging both head and sequence dimensionality reductions. We show that RocketKV provides a compression ratio of up to 400$\\times$, end-to-end speedup of up to 3.7$\\times$ as well as peak memory reduction of up to 32.6% in the decode phase on an NVIDIA A100 GPU compared to the full KV cache baseline, while achieving negligible accuracy loss on a variety of long-context tasks. We also propose a variant of RocketKV for multi-turn scenarios, which consistently outperforms other existing methods and achieves accuracy nearly on par with an oracle top-k attention scheme. The source code is available here: https://github.com/NVlabs/RocketKV."},"title":{"value":"RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression"},"authors":{"value":[{"fullname":"Payman Behnam","username":""},{"fullname":"Yaosheng Fu","username":""},{"fullname":"Ritchie Zhao","username":""},{"fullname":"Po-An Tsai","username":""},{"fullname":"Zhiding Yu","username":""},{"fullname":"Alexey Tumanov","username":"~Alexey_Tumanov1"}]}},"tmdate":1779817654607,"pdate":1767139200000,"externalIds":["dblp:journals/corr/abs-2502-14051"],"tcdate":1779817651090,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Alexey_Tumanov1"],"forum":"s1K3meS2xx","license":"CC BY-SA 4.0","number":31481,"cdate":1738368000000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1779817654607,"domain":"OpenReview.net/Public_Article","id":"s1K3meS2xx","version":2},{"content":{"venue":{"value":"ICML 2025"},"pdf":{"value":"https://raw.githubusercontent.com/mlresearch/v267/main/assets/behnam25a/behnam25a.pdf"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"behnam|rocketkv_accelerating_longcontext_llm_inference_via_twostage_kv_cache_compression"},"html":{"value":"https://proceedings.mlr.press/v267/behnam25a.html"},"_bibtex":{"value":"@inproceedings{DBLP:conf/icml/BehnamFZTYT25,\n author={Payman Behnam and Yaosheng Fu and Ritchie Zhao and Po-An Tsai and Zhiding Yu and Alexey Tumanov},\n title={RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression},\n year={2025},\n cdate={1735689600000},\n url={https://proceedings.mlr.press/v267/behnam25a.html},\n booktitle={ICML},\n crossref={conf/icml/2025}\n}\n"},"abstract":{"value":"Transformer-based Large Language Models rely critically on the KV cache to efficiently handle extended contexts during the decode phase. Yet, the size of the KV cache grows proportionally with the input length, burdening both memory bandwidth and capacity as decoding progresses. To address this challenge, we present RocketKV, a training-free KV cache compression strategy containing two consecutive stages. In the first stage, it performs coarse-grain permanent KV cache eviction on the input sequence tokens. In the second stage, it adopts a hybrid sparse attention method to conduct fine-grain top-k sparse attention, approximating the attention scores by leveraging both head and sequence dimensionality reductions. We show that RocketKV provides a compression ratio of up to 400$\\times$, end-to-end speedup of up to 3.7$\\times$ as well as peak memory reduction of up to 32.6% in the decode phase on an NVIDIA A100 GPU compared to the full KV cache baseline, while achieving negligible accuracy loss on a variety of long-context tasks. We also propose a variant of RocketKV for multi-turn scenarios, which consistently outperforms other existing methods and achieves accuracy nearly on par with an oracle top-k attention scheme. The source code is available here: https://github.com/NVlabs/RocketKV."},"title":{"value":"RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression"},"authors":{"value":[{"fullname":"Payman Behnam","username":""},{"fullname":"Yaosheng Fu","username":"~Yaosheng_Fu1"},{"fullname":"Ritchie Zhao","username":""},{"fullname":"Po-An Tsai","username":""},{"fullname":"Zhiding Yu","username":""},{"fullname":"Alexey Tumanov","username":"~Alexey_Tumanov1"}]}},"tmdate":1779817651897,"pdate":1767139200000,"externalIds":["dblp:conf/icml/BehnamFZTYT25"],"tcdate":1777761251770,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Yaosheng_Fu1"],"forum":"OPJRhPe8w0","license":"CC BY-SA 4.0","number":2213,"cdate":1735689600000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit","OpenReview.net/Public_Article/-/Authorship_Claim"],"mdate":1779817651897,"domain":"OpenReview.net/Public_Article","id":"OPJRhPe8w0","version":2},{"content":{"venue":{"value":"ICML 2025 poster"},"keywords":{"value":["LLM inference","KV cache compression","Efficiency"]},"_bibtex":{"value":"@inproceedings{\nbehnam2025rocketkv,\ntitle={Rocket{KV}: Accelerating Long-Context {LLM} Inference via Two-Stage {KV} Cache Compression},\nauthor={Payman Behnam and Yaosheng Fu and Ritchie Zhao and Po-An Tsai and Zhiding Yu and Alexey Tumanov},\nbooktitle={Forty-second International Conference on Machine Learning},\nyear={2025},\nurl={https://openreview.net/forum?id=RyOpooIxDF}\n}"},"title":{"value":"RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression"},"paperhash":{"value":"behnam|rocketkv_accelerating_longcontext_llm_inference_via_twostage_kv_cache_compression"},"TLDR":{"value":"RocketKV achieves significant KV cache memory bandwidth and storage savings during decode stage of LLM inference while maintains comparable accuracy to full KV cache attention."},"primary_area":{"value":"deep_learning->generative_models_and_autoencoders"},"abstract":{"value":"Transformer-based Large Language Models rely critically on the KV cache to efficiently handle extended contexts during the decode phase. Yet, the size of the KV cache grows proportionally with the input length, burdening both memory bandwidth and capacity as decoding progresses. To address this challenge, we present RocketKV, a training-free KV cache compression strategy containing two consecutive stages. In the first stage, it performs coarse-grain permanent KV cache eviction on the input sequence tokens. In the second stage, it adopts a hybrid sparse attention method to conduct fine-grain top-k sparse attention, approximating the attention scores by leveraging both head and sequence dimensionality reductions. We show that RocketKV provides a compression ratio of up to 400×, end-to-end speedup of up to 3.7× as well as peak memory reduction of up to 32.6% in the decode phase on an NVIDIA A100 GPU compared to the full KV cache baseline, while achieving negligible accuracy loss on a variety of long-context tasks. We also propose a variant of RocketKV for multi-turn scenarios, which consistently outperforms other existing methods and achieves accuracy nearly on par with an oracle top-k attention scheme. The source code is available here: https://github.com/NVlabs/RocketKV."},"link_to_code":{"value":"https://github.com/NVlabs/RocketKV"},"pdf":{"value":"/pdf/3adf72e58bb75ac49dbc04149532c4c7c3022073.pdf"},"lay_summary":{"value":"Large language models (LLMs), like ChatGPT, are excellent at generating human-like text but struggle when handling very lengthy documents. Their main issue arises from the need to remember extensive past information, causing them to slow down as text generation continues. \nTo overcome this challenge, we developed a training-free technique called RocketKV. RocketKV compresses the memory requirement in two consecutive stages. In the first stage, it permanently removes information determined to be less important, significantly reducing memory usage right away. In the second stage, RocketKV carefully selects the most relevant pieces of the remaining memory based on the current query, dynamically deciding what to prioritize at each text generation step. This two-stage strategy ensures that the model can efficiently maintain essential information without becoming overwhelmed by irrelevant details. \nBy intelligently combining permanent pruning with dynamic selection, RocketKV dramatically reduces memory needs by up to 400 times and speeds up by 3.7 times when running on an Nvidia A100 GPU. This innovation allows LLMs to handle long conversations and extensive documents more effectively, making advanced language technologies more practical and accessible in everyday applications."},"venueid":{"value":"ICML.cc/2025/Conference"},"authorids":{"value":["~Payman_Behnam1","~Yaosheng_Fu1","~Ritchie_Zhao1","~Po-An_Tsai1","~Zhiding_Yu1","~Alexey_Tumanov1"]},"authors":{"value":["Payman Behnam","Yaosheng Fu","Ritchie Zhao","Po-An Tsai","Zhiding Yu","Alexey Tumanov"]}},"tmdate":1755024828039,"pdate":1746105446257,"tcdate":1737603059047,"writers":["ICML.cc/2025/Conference","ICML.cc/2025/Conference/Submission8773/Authors"],"signatures":["ICML.cc/2025/Conference/Submission8773/Authors"],"forum":"RyOpooIxDF","license":"CC BY 4.0","number":8773,"cdate":1737603059047,"readers":["everyone"],"invitations":["ICML.cc/2025/Conference/-/Submission","ICML.cc/2025/Conference/-/Post_Submission","ICML.cc/2025/Conference/Submission8773/-/Full_Submission","ICML.cc/2025/Conference/-/Edit","ICML.cc/2025/Conference/Submission8773/-/Camera_Ready_Revision"],"mdate":1755024828039,"odate":1750231275353,"domain":"ICML.cc/2025/Conference","id":"RyOpooIxDF","version":2},{"content":{"TLDR":{"value":"We propose a KV cache quantization technique that leverages Hadamard transformation and trainable modules to minimize error in attention."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["KV cache quantization","KV cache compession","Efficient Inference","Large Language Models"]},"supplementary_material":{"value":"/attachment/2058aeb3d210ea7ff3b6b2d675c4e0878a8448b2.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Quantizing the key-value (KV) cache is a promising strategy for improving the inference efficiency of large language models (LLMs). However, aggressive quantization to very low precision (e.g., 2 bits) introduces significant errors in the stored key and value tensors, which propagate through the dot-product attention mechanism and ultimately degrade generation quality. To address this, we propose KVLinC, a framework to mitigate attention errors introduced by KV cache quantization in the extreme low-precision regime. KVLinC combines a Hadamard rotation, which reduces quantization error in values, with lightweight linear correction adapters that explicitly compensate for errors introduced by quantized keys. Across extensive evaluations on the LLaMA, Qwen2.5, and Qwen3 model families, KVLinC consistently matches or surpasses strong baselines while achieving higher KV-cache compression. Furthermore, we implement a custom attention kernel that results in upto 2.55x faster inference compared to Flash Attention baseline, enabling efficient long-context LLM inference."},"_bibtex":{"value":"@misc{\nsaxena2026kvlinc,\ntitle={{KVL}inC: {KV} Cache Quantization with Hadamard Rotation and Linear Correction},\nauthor={Utkarsh Saxena and Kaushik Roy},\nyear={2026},\nurl={https://openreview.net/forum?id=FkaDML963W}\n}"},"title":{"value":"KVLinC: KV Cache Quantization with Hadamard Rotation and Linear Correction"},"pdf":{"value":"/pdf/9b382cd5502f3ebd6a53cd3139eaa0850221b8ed.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"saxena|kvlinc_kv_cache_quantization_with_hadamard_rotation_and_linear_correction"},"authorids":{"value":["~Utkarsh_Saxena1","~Kaushik_Roy1"]},"authors":{"value":["Utkarsh Saxena","Kaushik Roy"]}},"tmdate":1770805088902,"tcdate":1758289627337,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission18633/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission18633/Authors"],"forum":"FkaDML963W","license":"CC BY 4.0","number":18633,"cdate":1758289627337,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/-/Submission","ICLR.cc/2026/Conference/-/Post_Submission","ICLR.cc/2026/Conference/Submission18633/-/Full_Submission","ICLR.cc/2026/Conference/Submission18633/-/Rebuttal_Revision","ICLR.cc/2026/Conference/-/Edit"],"mdate":1770805088902,"odate":1759896705795,"domain":"ICLR.cc/2026/Conference","id":"FkaDML963W","version":2},{"content":{"venue":{"value":"CoRR 2026"},"pdf":{"value":"https://arxiv.org/pdf/2602.02958v3"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"xi|quant_videogen_autoregressive_long_video_generation_via_2bit_kvcache_quantization"},"html":{"value":"https://doi.org/10.48550/arXiv.2602.02958"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2602-02958,\n publtype={informal},\n author={Haocheng Xi and Shuo Yang and Yilong Zhao and Muyang Li and Han Cai and Xingyang Li and Yujun Lin and Zhuoyang Zhang and Jintao Zhang and Xiuyu Li and Zhiying Xu and Jun Wu and Chenfeng Xu and Ion Stoica and Song Han and Kurt Keutzer},\n title={Quant VideoGen: Auto-Regressive Long Video Generation via 2-Bit KV-Cache Quantization},\n year={2026},\n month={February},\n cdate={1769904000000},\n journal={CoRR},\n volume={abs/2602.02958},\n url={https://doi.org/10.48550/arXiv.2602.02958}\n}\n"},"abstract":{"value":"Despite rapid progress in autoregressive video diffusion, an emerging system algorithm bottleneck limits both deployability and generation capability: KV cache memory. In autoregressive video generation models, the KV cache grows with generation history and quickly dominates GPU memory, often exceeding 30 GB, preventing deployment on widely available hardware. More critically, constrained KV cache budgets restrict the effective working memory, directly degrading long horizon consistency in identity, layout, and motion. To address this challenge, we present Quant VideoGen (QVG), a training free KV cache quantization framework for autoregressive video diffusion models. QVG leverages video spatiotemporal redundancy through Semantic Aware Smoothing, producing low magnitude, quantization friendly residuals. It further introduces Progressive Residual Quantization, a coarse to fine multi stage scheme that reduces quantization error while enabling a smooth quality memory trade off. Across LongCat Video, HY WorldPlay, and Self Forcing benchmarks, QVG establishes a new Pareto frontier between quality and memory efficiency, reducing KV cache memory by up to 7.0 times with less than 4% end to end latency overhead while consistently outperforming existing baselines in generation quality."},"title":{"value":"Quant VideoGen: Auto-Regressive Long Video Generation via 2-Bit KV-Cache Quantization"},"authors":{"value":[{"fullname":"Haocheng Xi","username":""},{"fullname":"Shuo Yang","username":""},{"fullname":"Yilong Zhao","username":""},{"fullname":"Muyang Li","username":""},{"fullname":"Han Cai","username":""},{"fullname":"Xingyang Li","username":""},{"fullname":"Yujun Lin","username":""},{"fullname":"Zhuoyang Zhang","username":"~Zhuoyang_Zhang1"},{"fullname":"Jintao Zhang","username":""},{"fullname":"Xiuyu Li","username":""},{"fullname":"Zhiying Xu","username":""},{"fullname":"Jun Wu","username":""},{"fullname":"Chenfeng Xu","username":""},{"fullname":"Ion Stoica","username":""},{"fullname":"Song Han","username":""},{"fullname":"Kurt Keutzer","username":""}]}},"tmdate":1777481402917,"pdate":1798675200000,"externalIds":["dblp:journals/corr/abs-2602-02958"],"tcdate":1777481399131,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Zhuoyang_Zhang1"],"forum":"xSYIJ56ekC","license":"CC BY-SA 4.0","number":932,"cdate":1769904000000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1777481402917,"domain":"OpenReview.net/Public_Article","id":"xSYIJ56ekC","version":2},{"content":{"summary":{"value":"This submission investigates the problem of KV cache management in LLMs and proposes a method called KVP which adopts Reinforcement Learning to learn a strategy minimizing the future value of tokens to be evicted from the cache. While this is an interesting, novel approach, the submission does not make a compelling case that KVP improves the performance of SOTA methods in a significant way."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"What is the latency and storage space of the various tested methods?\n\nAre the perplexity gains reported in figure 2 statistically significant?\n\nWhy do you use different performance metrics for RULER (accuracy) and for OASST2 (perplexity)?\n\nHow does KVP work for another LLM, such as a Llama model?\n\nWhat is the main benefit of KVP: better accuracy, reduced latency, reduced storage space, etc.?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":3},"strengths":{"value":"The formulation of KV cache eviction as a learning problem is original. The authors prove that, under two reasonable assumptions, the subset selection problem can be reduced to a ranking problem, which can then be formulated as an RL problem.\n\nIt is interesting that the policy requires only the keys, values, and their positions as input and no attention information. It is a strength of KVP that it can be pretained and does not incur any overhead at inference time.\n\nI appreciate the ablation study demonstrating that supervised learning does not work and RL is necessary.\n\nThe paper is written clearly."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The experiments show that KVP achieves the best accuracy or perplexity on RULER and OASST2 for most cache sizes (fig 2) and competitive accuracy on the downstream tasks BOOLQ and ARC CHALLENGE (fig 3). However, there is typically a tradeoff between accuracy and latency and storage space. Therefore, the authors should also report the latency and storage space of the various tested methods.\n\nThe authors have only performed experiments with a version of Owen, and I would like to see whether their results generalize to another LLM, such as a Llama model.\n\nWhile the learning approach of KVP is interesting and well-described, the main benefit of KVP remains unclear: is it better accuracy, reduced latency, reduced storage space, etc.?"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762933677486,"tcdate":1761448551578,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission20161/Reviewer_xyfd"],"signatures":["ICLR.cc/2026/Conference/Submission20161/Reviewer_xyfd"],"forum":"t7lJ2OEGbJ","number":2,"license":"CC BY 4.0","cdate":1761448551578,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission20161/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762933677486,"domain":"ICLR.cc/2026/Conference","replyto":"t7lJ2OEGbJ","id":"GuLFhZwkKY","forumContent":{"TLDR":{"value":"We reframe KV cache management from a heuristic-based problem to a learnable one, aiming to rank cache entries by their predicted future utility"},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Large Language Model","Key-Value Cache Compression","Natural Language Processing"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"The growing size of Large Language Models (LLMs) makes efficient inference challenging, primarily due to the memory demands of the autoregressive Key-Value (KV) cache. Existing eviction or compression methods reduce cost but rely on heuristics, such as recency or past attention scores, which serve only as indirect proxies for a token’s future utility and introduce computational overhead. We reframe KV cache eviction as a reinforcement learning (RL) problem: learning to rank tokens by their predicted usefulness for future decoding. To this end, we introduce KV Policy (KVP), a framework of lightweight per-head RL agents trained on pre-computed generation traces using only key and value vectors. Each agent learns a specialized eviction policy guided by a holistic reward, derived from future utility, that evaluates the quality of the ranking across all cache budgets, requiring no modifications to the underlying LLM or additional inference. Evaluated on the long-context benchmark RULER and the multi-turn dialogue benchmark OASST2-4k, KVP significantly outperforms baselines. Furthermore, zero-shot tests on standard downstream tasks indicate that KVP generalizes well beyond its training distribution. These results demonstrate that learning to predict future token utility is a powerful and scalable paradigm for adaptive KV cache management."},"_bibtex":{"value":"@misc{\nmoschella2026learning,\ntitle={Learning to Evict from Key-Value Cache},\nauthor={Luca Moschella and Laura Manduchi and Ozan Sener},\nyear={2026},\nurl={https://openreview.net/forum?id=t7lJ2OEGbJ}\n}"},"title":{"value":"Learning to Evict from Key-Value Cache"},"pdf":{"value":"/pdf/a0aca112e5fe99b0812d83eef7db0e2487e732b1.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"moschella|learning_to_evict_from_keyvalue_cache"},"authorids":{"value":["~Luca_Moschella1","~Laura_Manduchi2","~Ozan_Sener1"]},"authors":{"value":["Luca Moschella","Laura Manduchi","Ozan Sener"]}},"version":2},{"content":{"venue":{"value":"CoRR 2026"},"pdf":{"value":"https://arxiv.org/pdf/2605.23640v1"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"wu|cacheprune_privacyaware_and_finegrained_kv_cache_sharing_for_efficient_llm_inference"},"html":{"value":"https://doi.org/10.48550/arXiv.2605.23640"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2605-23640,\n publtype={informal},\n author={Guanlong Wu and Zhaohan Li and Yao Zhang and Zheng Zhang and Jianyu Niu and Ye Wu and Yinqian Zhang},\n title={CachePrune: Privacy-Aware and Fine-Grained KV Cache Sharing for Efficient LLM Inference},\n year={2026},\n month={May},\n cdate={1777593600000},\n journal={CoRR},\n volume={abs/2605.23640},\n url={https://doi.org/10.48550/arXiv.2605.23640}\n}\n"},"abstract":{"value":"Large Language Models (LLMs) rely on Key-Value (KV) caching to accelerate inference, and many serving systems further share the KV cache across users' requests to reduce redundant computation. While widely adopted, unrestricted cross-user sharing introduces side-channel vulnerabilities, allowing an adversary to infer user inputs by probing for cache reuse. Existing defenses disable sharing entirely to prevent leakage; yet such a coarse-grained strategy sacrifices substantial reuse potential, since prompts often include large portions of privacy-irrelevant segments, such as system instructions or publicly accessible materials. Building on this, we present CachePrune, a privacy-aware KV cache sharing mechanism that enables fine-grained reuse of KV entries across requests. Realizing such fine granularity requires token-level cache management, as reusable segments vary in length and position due to sensitivity masking, making reuse more complex than the fixed-size or sentence-level chunking used in existing coarse-grained schemes. Specifically, CachePrune makes fine-grained reuse practical by addressing two key challenges: accurately and efficiently deriving reusable KV segments and efficiently retrieving them over variable-length spans. We implement CachePrune on top of vLLM and evaluate it on three datasets, showing that it eliminates direct leakage through KV cache reuse side channels while reducing TTFT by 4.5x and increasing cache hit rates by 44% compared with state-of-the-art approaches."},"title":{"value":"CachePrune: Privacy-Aware and Fine-Grained KV Cache Sharing for Efficient LLM Inference"},"authors":{"value":[{"fullname":"Guanlong Wu","username":"~Guanlong_Wu1"},{"fullname":"Zhaohan Li","username":""},{"fullname":"Yao Zhang","username":""},{"fullname":"Zheng Zhang","username":""},{"fullname":"Jianyu Niu","username":""},{"fullname":"Ye Wu","username":""},{"fullname":"Yinqian Zhang","username":"~Yinqian_Zhang3"}]}},"tmdate":1786453421307,"pdate":1798675200000,"externalIds":["dblp:journals/corr/abs-2605-23640"],"tcdate":1784640020523,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Guanlong_Wu1"],"forum":"7ErgcuzPgc","license":"CC BY-SA 4.0","number":83529,"cdate":1777593600000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit","OpenReview.net/Public_Article/-/Authorship_Claim"],"mdate":1786453421307,"domain":"OpenReview.net/Public_Article","id":"7ErgcuzPgc","version":2},{"content":{"venue":{"value":"CoRR 2026"},"pdf":{"value":"https://arxiv.org/pdf/2605.09649v1"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"bui|make_each_token_count_towards_improving_longcontext_performance_with_kv_cache_eviction"},"html":{"value":"https://doi.org/10.48550/arXiv.2605.09649"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2605-09649,\n publtype={informal},\n author={Ngoc Bui and Hieu Trung Nguyen and Arman Cohan and Rex Ying},\n title={Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction},\n year={2026},\n month={May},\n cdate={1777593600000},\n journal={CoRR},\n volume={abs/2605.09649},\n url={https://doi.org/10.48550/arXiv.2605.09649}\n}\n"},"abstract":{"value":"The key-value (KV) cache is a major bottleneck in long-context inference, where memory and computation grow with sequence length. Existing KV eviction methods reduce this cost but typically degrade performance relative to full-cache inference. Our key insight is that full-cache attention is not always optimal: in long contexts, irrelevant tokens can dilute attention away from useful evidence, so selective, learnable eviction can improve generation rather than merely approximate the full cache. We introduce a global retention-based KV eviction method that learns each token's future utility under a unified memory budget. Lightweight retention gates assign utility scores to cached KV entries, and a shared final scoring projection calibrates these scores across all layers and heads. This enables a single global eviction policy in which tokens from different layers, heads, and modalities compete directly for cache capacity. We further provide theoretical analysis showing that preferentially retaining useful tokens reduces attention dilution, and we justify geometric retention as a query-agnostic proxy for future utility. Across diverse long-context language and vision-language reasoning, and multi-turn dialogue benchmarks, our method substantially reduces KV memory while matching or surpassing full-cache inference. These results suggest that learned, globally calibrated KV eviction is not only a compression technique, but also a mechanism for improving long-context reasoning."},"title":{"value":"Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction"},"authors":{"value":[{"fullname":"Ngoc Bui","username":""},{"fullname":"Hieu Trung Nguyen","username":"~Hieu_Trung_Nguyen2"},{"fullname":"Arman Cohan","username":""},{"fullname":"Rex Ying","username":""}]}},"tmdate":1784694159742,"pdate":1798675200000,"externalIds":["dblp:journals/corr/abs-2605-09649"],"tcdate":1784694155221,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Hieu_Trung_Nguyen2"],"forum":"Yj7SV3rGxL","license":"CC BY-SA 4.0","number":87838,"cdate":1777593600000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1784694159742,"domain":"OpenReview.net/Public_Article","id":"Yj7SV3rGxL","version":2},{"content":{"summary":{"value":"ICaRus introduces “Identical Cache Reuse” to let multiple task-specialized LLMs share the same KV cache for identical prompts, avoiding redundant memory use and recomputation in multi-model inference. It splits a decoder-only Transformer into a frozen logical encoder (that builds KV caches) and a fine-tuned logical decoder (that predicts tokens), so all models reuse one shared cache. Using lightweight adapters (e.g., LoRA), ICaRus achieves similar accuracy to task-specific fine-tuning but cuts P95 latency by up to 11× and boosts throughput up to 3.8× in multi-agent workflows."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"1. How does the work deal with cases when agent finishes prefill and does some decoding? How to reuse the decode token for that part? Or how does system perform if that part is not used? Especially in long decoding scenarios like coding.\n\n2. How does the accuracy compare with not sharing at all? A simpler workflow without multi-model inference. Not sure if the dataset needs multi-model inference."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. Innovative idea that changes how multi-model share KV cache.\n2. Demonstrates good potential system improvement over previous methods by the new design.\n3. Promising path to efficient multi-model inference."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Unclear accuracy comparison to workflows without any sharing or naive baselines. How does a base model perform if I do not use the new workflow?\n2. Training seems to be more complex. How robust is this training method? It will be more convincing to do more experiments on bigger models and more datasets."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1764358354112,"tcdate":1761608071315,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission18052/Reviewer_eTzX"],"signatures":["ICLR.cc/2026/Conference/Submission18052/Reviewer_eTzX"],"forum":"qrMo6R7lOS","number":2,"license":"CC BY 4.0","cdate":1761608071315,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission18052/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1764358354112,"domain":"ICLR.cc/2026/Conference","replyto":"qrMo6R7lOS","id":"czv7OKtjLr","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"TLDR":{"value":"We propose ICaRus, the first architecture that enables multi model to fully share KV caches across all layers, providing a principled solution to inefficiencies in conventional multi model serving."},"keywords":{"value":["multi model inference","agentic ai","prefix caching","fine-tuning"]},"primary_area":{"value":"other topics in machine learning (i.e., none of the above)"},"abstract":{"value":"Multi model inference, where multiple task-specialized models collaborate to solve complex real-world problems, has recently emerged as a prominent paradigm, particularly in the development of agentic AI systems. However, in such scenarios, each model must maintain its own Key-Value (KV) cache for the identical prompt, leading to explosive memory consumption.\nThis explosive growth of KV caches forces LLM serving systems to evict previously stored caches, which in turn introduces significant recomputation overhead whenever the evicted caches are required again. \nMoreover, prefix caching is inherently infeasible across different models, forcing each model to recompute KV cache for the identical prompt, which leads to signficant overhead. To alleviate these issues, we propose Identical Cache Reuse (ICaRus), a novel architecture that allows multiple models to share identical KV caches across all layers. ICaRus is based on the key observation that a decoder-only Transformer can be conceptually decomposed into a logical encoder, which generates KV caches, and a logical decoder, which predicts output tokens from the KV caches. ICaRus fine-tunes only the logical decoder while freezing the logical encoder, enabling multiple models to share an identical KV cache. This eliminates cache memory explosion and unexpected evictions while also allowing cross-model reuse of KV caches for new input tokens, thereby removing redundant recomputation in multi model inference achieving both efficiency and scalability. Moreover, by incorporating lightweight adapters such as LoRA, ICaRus parallelizes KV cache generation and next-token prediction during decoding. ICaRus achieves comparable accuracy to task-specific fine-tuned model across a diverse set of tasks, while allowing multiple specialized models to fully share KV caches. ICaRus achieves up to $11.1\\times$ lower P95 latency and $3.8\\times$ higher throughput in multi agent scenarios with 8 different models, compared to prior multi model system."},"_bibtex":{"value":"@inproceedings{\nwoo2026icarus,\ntitle={{IC}aRus: Identical Cache Reuse for Efficient Multi-Model Inference},\nauthor={Sunghyeon Woo and Jaeeun Kil and Hoseung Kim and Minsub Kim and Joonghoon Kim and Ahreum Seo and Sungjae Lee and Minjung Jo and Jiwon Ryu and Baeseong park and Se Jung Kwon and Dongsoo Lee},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=qrMo6R7lOS}\n}"},"title":{"value":"ICaRus: Identical Cache Reuse for Efficient Multi-Model Inference"},"pdf":{"value":"/pdf/b336f947a721179fe4654de18119386e44768469.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"woo|icarus_identical_cache_reuse_for_efficient_multimodel_inference"},"authorids":{"value":["~Sunghyeon_Woo1","~Jaeeun_Kil1","~Hoseung_Kim2","~Minsub_Kim3","~Joonghoon_Kim2","~Ahreum_Seo1","~Sungjae_Lee1","~Minjung_Jo1","~Jiwon_Ryu2","~Baeseong_park1","~Se_Jung_Kwon1","~Dongsoo_Lee1"]},"authors":{"value":["Sunghyeon Woo","Jaeeun Kil","Hoseung Kim","Minsub Kim","Joonghoon Kim","Ahreum Seo","Sungjae Lee","Minjung Jo","Jiwon Ryu","Baeseong park","Se Jung Kwon","Dongsoo Lee"]}},"version":2},{"content":{"TLDR":{"value":"We ternarize the weights and the KV cache into the same two-plane ternary format, placing the weights' second plane on a sparse subset of elements."},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["post-training quantization","ternary quantization","KV cache quantization","large language models","model compression","multiplication-free inference","efficient inference"]},"supplementary_material":{"value":"/attachment/094182c809e57ef64053cf1653296dbfaa790fcb.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Reading the weights and KV cache dominates large language model inference. Ternary quantization to {-1, 0, +1} shrinks both and removes per-element multipliers: each weight product becomes a sign flip or skip, so a signed adder tree suffices. Ternary weights alone, however, leave multipliers in attention's $QK^\\top$ and $PV$, whose operands are both run-time floating-point tensors.\n\nWe present ESPTQ (element-sparse-plane ternary post-training quantization), which quantizes the weights and KV cache into one ternary storage format, removing per-element multipliers from every matrix product inside the Transformer layers except the rotation of layer inputs. By design, ESPTQ caps the residual multiplication ratio at the weight-side budget of two multiplications per 128-element reduction ($2/128 = 1.5625\\%$). For weights, after a learnable butterfly rotation within each 128-column group, a rate-distortion criterion selects, at a continuously variable second-plane assignment ratio $\\rho$, which elements receive a sparse second ternary plane. For the KV cache, static tables replace per-token offsets and the V-side scaling factor, meeting the same budget.\n\nWe compare methods on effective bits per weight, serializing each in its own storage format with losslessly compressed discrete codes, so scaling factors, rotation angles and other auxiliary metadata count. On LLaMA-2-7B, ESPTQ matches TWLA in perplexity and average zero-shot accuracy at less than half the effective bits, and at comparable bits lowers perplexity by 36% against PT2-LLM. Ternarizing the KV cache as well shrinks it to about one fifth of its FP16 size, at less cost in perplexity and zero-shot accuracy than ternarizing the weights."},"_bibtex":{"value":"@inproceedings{\nanonymous2026esptq,\ntitle={{ESPTQ}: Post-Training Ternarization of Weights and {KV} Cache for Eliminating Per-Element Multiplications},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=lJj9QgYW2w},\nnote={under review}\n}"},"title":{"value":"ESPTQ: Post-Training Ternarization of Weights and KV Cache for Eliminating Per-Element Multiplications"},"pdf":{"value":"/pdf/b9e8bcc76cb6d8b9e39e6c7ba6825d580849223a.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791231678968,"tcdate":1789664480603,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission33371/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission33371/Authors"],"forum":"lJj9QgYW2w","license":"CC BY 4.0","number":33371,"cdate":1789664480603,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission33371/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791231678968,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"lJj9QgYW2w","version":2},{"content":{"summary":{"value":"This paper proposes a training-free KV cache compression method that exploits heterogeneous roles across transformer head dimensions. The authors observe that lower dimensions capture local context while upper dimensions capture long-range dependencies, and compress the former using Fourier basis functions from the HiPPO framework. A custom Triton kernel (FlashFourierAttention) is implemented for efficient deployment. The observation and the custom kernel are nice, but the evaluation is not comprehensive enough. Both the choice of the baselines (compression vs quantization) and the range of performed experiments and models selected are not enough. Also, the paper would benefit from a better framing of the method with respect to the literature about KV Cache compression."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"- Could you frame the contribution wrt KV Cache compression, Quantization and Low Rank reduction.\n- Could you provide exact numerical results for latency and memory consumption rather than just plots?\n- Why aren't other architectures included in the evaluation ? Could you include them ? \n- Could you provide results in Table1 for different compression ratios ?"},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"- The empirical finding that different head dimensions serve distinct roles (local vs. global context) is interesting and well-validated through ablation studies\n- Leveraging the HiPPO framework provides mathematical rigor, and the adaptation from complex to real-valued representations is sensible for practical implementation.\n- The custom Triton kernel shows practical engineering effort and is very nice."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- (General) The title and abstract claim this is \"KV cache compression,\" but the actual mechanism is better described as lossy approximation or dimensionality reduction. For this reason, while comparing to Palu (dimensionality reduction) makes sense, comparing to SnapKV and PyramidKV is kind of strange as these perform KV Cache eviction in a different setting. The authors should (a) discuss the differences between KV Cache compression, quantization, reduction in a clear way and (b) include quantization in the baselines.\n- (Method) The observation about dimension heterogeneity is a bit over-claimed and not entirely new, as prior work has noted similar patterns. This is cited by the authors themselves.\n- In Table 1. The comparison shows that the methods have been tested with different compression ratios. Why not use the same compression ratio for fair comparison? \n- In Table 1 and in general across the paper, for a fair and comprehensive evaluation one should consider a range of compression rations for all methods instead of a specific one.\n- The choice of L-init=4, L-local=1024, N=512 appears arbitrary with no ablation study or principled justification. How sensitive is performance to these choices?\n- The method is only evaluated on Llama models. This strongly hinders the generalization of the method to different architectures.\n- I am not sure I understood how is the dimension selection actually performed? The authors say \"we directly compress and decompress all KV caches, prioritizing dimensions with smaller mean-squared error\" but doesn't specify the algorithm. Is this done once offline or adaptively?"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1764360359402,"tcdate":1762533735226,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission5870/Reviewer_smj6"],"signatures":["ICLR.cc/2026/Conference/Submission5870/Reviewer_smj6"],"forum":"4sx3Jzrg5w","number":4,"license":"CC BY 4.0","cdate":1762533735226,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission5870/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1764360359402,"domain":"ICLR.cc/2026/Conference","replyto":"4sx3Jzrg5w","id":"pYpn7A5PRx","forumContent":{"TLDR":{"value":"We propose FourierAttention, a training-free framework that exploits the heterogeneous roles of transformer head dimensions."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Large Language Model","Long-Context LLM","KV cache optimization"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Large Language Models struggle with memory demands from the growing Key-Value (KV) cache as context lengths increase. Existing compression methods homogenize head dimensions or rely on attention-guided token pruning, often sacrificing accuracy or introducing computational overhead. We propose FourierAttention, a training-free framework that exploits the heterogeneous roles of transformer head dimensions: lower dimensions prioritize local context, while upper ones capture long-range dependencies. By projecting the long-context-insensitive dimensions onto orthogonal Fourier bases, FourierAttention approximates their temporal evolution with fixed-length spectral coefficients. Evaluations on LLaMA models show FourierAttention achieves the best long-context accuracy on LongBench and Needle-In-A-Haystack (NIAH). Besides, a custom Triton kernel, FlashFourierAttention, is designed to optimize memory via streamlined read-write operations, enabling efficient deployment without performance compromise."},"_bibtex":{"value":"@misc{\nliu2026beyond,\ntitle={Beyond Homogeneous Attention: Memory-Efficient {LLM}s via Fourier-Approximated {KV} Cache},\nauthor={Xiaoran Liu and Siyang He and Qiqi Wang and Ruixiao Li and Yuerong Song and Zhigeng Liu and Mianqiu Huang and Zengfeng Huang and Qipeng Guo and Ziwei He and Xipeng Qiu},\nyear={2026},\nurl={https://openreview.net/forum?id=4sx3Jzrg5w}\n}"},"title":{"value":"Beyond Homogeneous Attention: Memory-Efficient LLMs via Fourier-Approximated KV Cache"},"pdf":{"value":"/pdf/4751bc43b0365bbbdfcaaebe72a58b648e3b60ed.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"liu|beyond_homogeneous_attention_memoryefficient_llms_via_fourierapproximated_kv_cache"},"authorids":{"value":["~Xiaoran_Liu1","~Siyang_He1","~Qiqi_Wang3","~Ruixiao_Li1","~Yuerong_Song1","~Zhigeng_Liu1","~Mianqiu_Huang1","~Zengfeng_Huang1","~Qipeng_Guo1","~Ziwei_He1","~Xipeng_Qiu1"]},"authors":{"value":["Xiaoran Liu","Siyang He","Qiqi Wang","Ruixiao Li","Yuerong Song","Zhigeng Liu","Mianqiu Huang","Zengfeng Huang","Qipeng Guo","Ziwei He","Xipeng Qiu"]}},"version":2},{"content":{"venue":{"value":"CoRR 2026"},"pdf":{"value":"https://arxiv.org/pdf/2603.20616v1"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"miao|beyond_token_eviction_mixeddimension_budget_allocation_for_efficient_kv_cache_compression"},"html":{"value":"https://doi.org/10.48550/arXiv.2603.20616"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2603-20616,\n publtype={informal},\n author={Ruijie Miao and Zhiming Wang and Wang Li and Shiwei Wu and Shufan Liu and Yanbing Jiang and Tong Yang},\n title={Beyond Token Eviction: Mixed-Dimension Budget Allocation for Efficient KV Cache Compression},\n year={2026},\n month={March},\n cdate={1772323200000},\n journal={CoRR},\n volume={abs/2603.20616},\n url={https://doi.org/10.48550/arXiv.2603.20616}\n}\n"},"abstract":{"value":"Key-value (KV) caching is widely used to accelerate transformer inference, but its memory cost grows linearly with input length, limiting long-context deployment. Existing token eviction methods reduce memory by discarding less important tokens, which can be viewed as a coarse form of dimensionality reduction that assigns each token either zero or full dimension. We propose MixedDimKV, a mixed-dimension KV cache compression method that allocates dimensions to tokens at a more granular level, and MixedDimKV-H, which further integrates head-level importance information. Experiments on long-context benchmarks show that MixedDimKV outperforms prior KV cache compression methods that do not rely on head-level importance profiling. When equipped with the same head-level importance information, MixedDimKV-H consistently outperforms HeadKV. Notably, our approach achieves comparable performance to full attention on LongBench with only 6.25% of the KV cache. Furthermore, in the Needle-in-a-Haystack test, our solution maintains 100% accuracy at a 50K context length while using as little as 0.26% of the cache."},"title":{"value":"Beyond Token Eviction: Mixed-Dimension Budget Allocation for Efficient KV Cache Compression"},"authors":{"value":[{"fullname":"Ruijie Miao","username":""},{"fullname":"Zhiming Wang","username":""},{"fullname":"Wang Li","username":""},{"fullname":"Shiwei Wu","username":""},{"fullname":"Shufan Liu","username":""},{"fullname":"Yanbing Jiang","username":"~Yanbing_Jiang1"},{"fullname":"Tong Yang","username":"~Tong_Yang6"}]}},"tmdate":1784539244783,"pdate":1798675200000,"externalIds":["dblp:journals/corr/abs-2603-20616"],"tcdate":1784471391425,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Yanbing_Jiang1"],"forum":"TB0GufkFbI","license":"CC BY-SA 4.0","number":64918,"cdate":1772323200000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit","OpenReview.net/Public_Article/-/Authorship_Claim"],"mdate":1784539244783,"domain":"OpenReview.net/Public_Article","id":"TB0GufkFbI","version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"http://arxiv.org/pdf/2502.03805v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"feng|identify_critical_kv_cache_in_llm_inference_from_an_output_perturbation_perspective"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Yuan_Feng:","https://dblp.org/search/pid/api?q=author:Junlin_Lv:","~Yukun_Cao1","https://dblp.org/search/pid/api?q=author:Xike_Xie:","https://dblp.org/search/pid/api?q=author:S._Kevin_Zhou:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2502.03805"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2502-03805,\n publtype={informal},\n author={Yuan Feng and Junlin Lv and Yukun Cao and Xike Xie and S. Kevin Zhou},\n title={Identify Critical KV Cache in LLM Inference from an Output Perturbation Perspective},\n year={2025},\n month={February},\n cdate={1738368000000},\n journal={CoRR},\n volume={abs/2502.03805},\n url={https://doi.org/10.48550/arXiv.2502.03805}\n}\n"},"abstract":{"value":"Large language models have revolutionized natural language processing but face significant challenges of high storage and runtime costs, due to the transformer architecture's reliance on self-attention, particularly the large Key-Value (KV) cache for long-sequence inference. Recent efforts to reduce KV cache size by pruning less critical entries based on attention weights remain empirical and lack formal grounding. This paper presents a formal study on identifying critical KV cache entries by analyzing attention output perturbation. Our analysis reveals that, beyond attention weights, the value states within KV entries and pretrained parameter matrices are also crucial. Based on this, we propose a perturbation-constrained selection algorithm that optimizes the worst-case output perturbation to identify critical entries. Evaluations on the Needle-in-a-Haystack test and Longbench benchmark show our algorithm enhances state-of-the-art cache eviction methods. Further empirical analysis confirms that our algorithm achieves lower output perturbations in over 92% attention heads in Llama model, thereby providing a significant improvement over existing methods."},"title":{"value":"Identify Critical KV Cache in LLM Inference from an Output Perturbation Perspective"},"authors":{"value":["Yuan Feng","Junlin Lv","Yukun Cao","Xike Xie","S. Kevin Zhou"]}},"tmdate":1744767623247,"pdate":1735689600000,"tcdate":1744767618777,"writers":["~"],"signatures":["~Yukun_Cao1"],"forum":"0xquNXbzx7","license":"CC BY-SA 4.0","number":398448,"cdate":1738368000000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1744767623247,"domain":"DBLP.org","id":"0xquNXbzx7","version":2},{"content":{"summary":{"value":"INFOBLEND enables position independent kv cache reuse for multimodal LLMs. While previous work like, CacheBlend recomputes tokens with the largest KV deviation, INFOBLEND also considers attention importance to selectively recompute only critical tokens, mainly at the start of image sequences. INFOBLEND uses a single-step selective attention process instead of CacheBlend’s two-step recomputation, supports parallel KV loading for multimodal data, and achieves higher throughput and lower latency."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"1. How to set hyperparameters in infoblend? aka how to tune the delay-accuracy tradeoff curve?\n2. Questions in the weakness section."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"1. Well motivated problem about multimodal KV cache reuse. \n2. Good results on 4 model-dataset pairs, comparison with previous work.\n3. Support for multimodal by extending into disk storage and related system optimizations around them.\n4. Novel combination of attention sink and kv deviation."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Comparison between this and CacheBlend not clear enough. \n2. Not sure what is new about multimodal. The delta mentioned with previous work is that there are attention sink considerations in infoblend. However, why is this multimodal specific?\n3. Different rates of Infoblend seem to be not on a tradeoff frontier curve but rather a cluster of points. There should be difference in time vs accuracy. Why isn't there difference?\n4. Can we report on bigger models or more sota methods like qwen-vl? Lack of discussion for these newer models"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762943378334,"tcdate":1761606809986,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission25243/Reviewer_a8xw"],"signatures":["ICLR.cc/2026/Conference/Submission25243/Reviewer_a8xw"],"forum":"bld5GVRad0","number":2,"license":"CC BY 4.0","cdate":1761606809986,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission25243/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762943378334,"domain":"ICLR.cc/2026/Conference","replyto":"bld5GVRad0","id":"xExXkyekRc","forumContent":{"TLDR":{"value":"The KV cache can be reused without positional restriction, through partial recomputation."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Multimodal Large Language Model","AI System","Position-Independent Caching"]},"supplementary_material":{"value":"/attachment/7c1cdb642495362f339e608259469e73183144d6.zip"},"primary_area":{"value":"infrastructure, software libraries, hardware, systems, etc."},"abstract":{"value":"The context caching technique is employed to accelerate the Multimodal Large Language Model (MLLM) inference by prevailing serving platforms currently. However, this approach merely reuses the Key-Value (KV) cache of the initial sequence of prompt, resulting in full KV cache recomputation even if the prefix differs slightly. This becomes particularly inefficient in the context of interleaved text and images, as well as multimodal retrieval-augmented generation. This paper proposes position-independent caching as a more effective approach for multimodal information management. We have designed and implemented a caching system, named InfoBlend, to address both system-level and algorithm-level challenges. InfoBlend stores the KV cache on local disks when receiving multimodal data, and calculates and loads the KV cache in parallel during inference. To mitigate accuracy degradation, we have incorporated the integrated reuse and recompute mechanism within the system. The experimental results demonstrate that InfoBlend can achieve up to 54\\% reduction in response time and 2$\\times$ improvement in throughput compared to existing context caching systems, while maintaining negligible or no accuracy loss."},"_bibtex":{"value":"@misc{\nzhao2026infoblend,\ntitle={InfoBlend: Storing and Reusing {KV} Caches of Multimodal Information without Positional Restriction},\nauthor={Shiju Zhao and Junhao Hu and Rongxiao Huang and Jiaqi Zheng and Guihai Chen},\nyear={2026},\nurl={https://openreview.net/forum?id=bld5GVRad0}\n}"},"title":{"value":"InfoBlend: Storing and Reusing KV Caches of Multimodal Information without Positional Restriction"},"pdf":{"value":"/pdf/b0d06f97f9c6ad80b2f9594560c7ee8d676679a3.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"zhao|infoblend_storing_and_reusing_kv_caches_of_multimodal_information_without_positional_restriction"},"authorids":{"value":["~Shiju_Zhao1","~Junhao_Hu3","~Rongxiao_Huang1","~Jiaqi_Zheng1","~Guihai_Chen1"]},"authors":{"value":["Shiju Zhao","Junhao Hu","Rongxiao Huang","Jiaqi Zheng","Guihai Chen"]}},"version":2},{"content":{"venue":{"value":"CoRR 2026"},"pdf":{"value":"https://arxiv.org/pdf/2604.10044v1"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"xu|loopguard_breaking_selfreinforcing_attention_loops_via_dynamic_kv_cache_intervention"},"html":{"value":"https://doi.org/10.48550/arXiv.2604.10044"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2604-10044,\n publtype={informal},\n author={Dongjie Xu and Hao Wu and Weijie Shi and Yue Cui and Yuanjun Liu and Jiawei Li and Haolun Ma and An Liu and Jia Zhu and Jiajie Xu},\n title={LoopGuard: Breaking Self-Reinforcing Attention Loops via Dynamic KV Cache Intervention},\n year={2026},\n month={April},\n cdate={1775001600000},\n journal={CoRR},\n volume={abs/2604.10044},\n url={https://doi.org/10.48550/arXiv.2604.10044}\n}\n"},"abstract":{"value":"Through systematic experiments on long-context generation, we observe a damaging failure mode in which decoding can collapse into persistent repetition loops. We find that this degeneration is driven by collapsed attention patterns, where a subset of heads locks onto a narrow suffix of the history, and is further stabilized by inference-time KV cache reuse. Crucially, since many existing KV cache policies rely on attention-based importance, this collapse can produce spuriously high scores for repetitive tokens, causing cache management to inadvertently amplify repetition. To study this phenomenon in a controlled and reproducible manner, we introduce LoopBench, a benchmark with explicit loop-inducing conditions and loop-oriented metrics that quantify repetition severity and generation instability beyond downstream task scores. Building on these insights, we propose LoopGuard, a lightweight, plug-in KV cache guard that detects loop onset online and disrupts the feedback cycle by pruning repetitive tail spans under a fixed cache budget. Experiments on LoopBench show that LoopGuard reduces loop incidence by over 90 percentage points, while restoring output diversity and reducing token waste."},"title":{"value":"LoopGuard: Breaking Self-Reinforcing Attention Loops via Dynamic KV Cache Intervention"},"authors":{"value":[{"fullname":"Dongjie Xu","username":""},{"fullname":"Hao Wu","username":""},{"fullname":"Weijie Shi","username":"~Weijie_Shi1"},{"fullname":"Yue Cui","username":""},{"fullname":"Yuanjun Liu","username":""},{"fullname":"Jiawei Li","username":""},{"fullname":"Haolun Ma","username":""},{"fullname":"An Liu","username":""},{"fullname":"Jia Zhu","username":""},{"fullname":"Jiajie Xu","username":""}]}},"tmdate":1791785770872,"pdate":1798675200000,"externalIds":["dblp:journals/corr/abs-2604-10044"],"tcdate":1791785765776,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Weijie_Shi1"],"forum":"f3bGHGJyvZ","license":"CC BY-SA 4.0","number":172332,"cdate":1775001600000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1791785770872,"domain":"OpenReview.net/Public_Article","id":"f3bGHGJyvZ","version":2},{"content":{"summary":{"value":"This paper considers the problem of reducing the \"KV cache\" (i.e. the cache needed to compute each entry of the Attention layer). Specifically, the paper observes experimentally that the keys in the existing systems that use Attention tend to be clustered. Then it proposes to utilize this insight into creating an approximation for the Attention layer by at a _very_ high level by only only remembering the \"centers\" of the said clusters. Doing this reduces both the space needed to store the KV cache as well as the time needed to compute the (clustered approximation) to the Attention layer.\n\nThe paper presents algorithms/data structures to both maintain the cluster and to compute the approximation to the Attention later. Assuming $(m,\\delta)$-_clusterability_ (i.e. there are $m$ clusters and the max radius of each cluster is $\\delta$), the paper proves that the proposed algorithms use sub-linear time and space (as opposed to the linear time and space for the vanilla Attention implementation) (under certain assumptions on the values of $m$ and $\\delta$).\n\nThe paper also presents experimental results that shows that the proposed method performs better than two existing KV cache compression systems-- Sink and H20."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"Please address the two high level weaknesses pointed out above. Specifically,\n\n1. Please comment on why the presented improvements in the RAM model would translate into wall clock improvement in modern GPU/TPUs.\n2. Please compare (both the theoretical and experimental) existing results (as pointed out above and are not compared in the paper to the proposed system) with the proposed (theoretical and experimental) results.\n\nPost-rebuttal comments\n----------------------------\n\nThe authors addressed most of my concerns so am upping my score to a 5."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"* The paper presents a nice clean theoretical modeling on how Attention keys cluster (and presents some experimental results to show that this is relevant in practice).\n\n* The paper presents nice clean theoretical bounds on the performance of the proposed algorithms (that show that they improve upon the bounds of the plain vanilla Attention implementation).\n\n* The experimental results show improvements over two existing systems-- Sink and H20."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"There are two major weaknesses of the paper for me-- first, the paper analyzes the improvements in time and space in the RAM/CPU model but deep learning (DL) systems and specifically Transformer models today are run on GPUs (as is the case with the experiments in the paper) and it is not clear that the theoretical improvements in the paper would translate to the GPU model. The second (and related) weakness is that the paper does not do comparison with other architectures that aim to reduce the KV cache but do have implementations that are \"GPU aware.\"\n\nTheoretical Analysis is on the RAM model\n----------------------------------------\n\nThe theoretical analysis presented in the paper is for the RAM model and presents a random sampling based algorithm to show improvement in the RAM/streaming model. However, modern day system pretty much all run on GPUs and TPUs where random sampling does not tend to work well. This is primarily because in modern hardware all operations happen at the level of blocks-- in other words, reading one entry from a block is the same as reading the entire block. So e.g. if the random sampling does not take this block structure into account (which the algorithms in the paper from what I can tell do _not_ do), then it is not clear if the theoretical gains will manifest in actual systems since it might still be reading all the blocks even if the actual number of entries read by random sampling is much lower. Specifically, it is not clear to me that the proposed implementation would be competitive against an implementation of Attention like FlashAttention (https://github.com/Dao-AILab/flash-attention), which is an _exact_ Attention implementation that is hardware aware. Further, modern GPUs/TPUs allow for block-block matrix-multiplication in $O(1)$ time and an algorithm that is created specifically to exploit such natively supported operations might have a higher number of FLOPs than the proposed algorithm but would still beat the proposed algorithm in wall clock speed.\n\nIn summary, the modern hardware systems have native support for specialized operations, which the proposed algorithm does not seem to exploit. This does not make me confident that the proposed algorithm would work better than these hardware aware implementations.\n\nInsufficient comparison with prior related work\n-----------------------------------------------\n\nThe paper's weakness with adequate comparison with prior work can in turn be divided into three parts:\n\n1. Of course it is entirely possible that the proposed algorithms actually run faster hardware aware algorithms (like FlashAttention) but no such comparison is provided in the experimental section of the paper. It is possible that the \"Exact\" Attention implementation uses a FlashAttention implementation but it is not clear from the paper whether this is the case. It would be good to either get confirmation that the \"Exact\" Attention uses a FlashAttention implementation (if so, which version) or if not, then it would be good to see comparison performance numbers with FlashAttention.\n2. Moving beyond hardware aware exact FlashAttention implementations, there are approximate Attention implementations that have good implementations on modern hardware. Two recent examples-- The Hedgehog & the Porcupine (https://arxiv.org/abs/2402.04347) and Based (https://arxiv.org/abs/2402.18668) utilize the fact that one can approximate the exp function with low degree polynomials and then noting that apply a low degree polynomial to each entry of a low rank matrix gives rise to low rank matrices allows one to use these Taylor series approximations to define a \"kernel\" and then to use linear Attention instead of softmax Attention to reduce the KV cache size. These recent papers have implementations suited to modern hardware and have accuracy that matches those of Transformers (but are much more efficient).\n * I would like to point out that the above idea of using Taylor series + linear attention/low rank matrices have been exploited before. E.g. see the paper of Alman and Song (https://arxiv.org/abs/2302.13214) and the earlier work of Chen et al. (https://arxiv.org/abs/2110.15343). The latter paper also proposed to compress the KV cache under the assumption of keys being clustered (though the clustering model in this paper is cleaner and more general). But in general, for the theoretical results, it would be good to compare the theoretical results in this paper and the results e.g. in the Alman and Song paper (for the case when they get $n^{1+o(1)}$ implementation of Attention.\n * In addition to the above body of work on approximating Attention there have been alternative Attention free models that have garnered a lot of Attention. Mamba (https://arxiv.org/abs/2312.00752, also see Mamba 2: https://arxiv.org/abs/2405.21060) is probably the model that has garnered most attention. These models have comparable accuracy to Transformer but are much faster and use much smaller cache sizes.\n * Overall the paper should present a comparison with the above body of work (for approximate Attention models this should be done in both Tables 2 and 3, while for Attention free models comparison should be made in Table 3).\n3. Finally, one big downside of the results presented in this paper is that the _accuracy_ over the exact Attention implementation takes a big hit. The proposed new algorithms do get an improvement of up to $50$ percent in cache size but also take a similar hit in accuracy. By comparison, the related work mentioned in the above bullet get a much larger improvement in cache size while _essentially not losing in accuracy/perplexity at all_ when compared to exact Transformers. However, these works are not evaluated in Tables 2 and 3 in the paper."}},"nonreaders":[],"tmdate":1731888671509,"tcdate":1730601309422,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission13479/Reviewer_aidT"],"signatures":["ICLR.cc/2025/Conference/Submission13479/Reviewer_aidT"],"forum":"M922KJFO7O","number":3,"license":"CC BY 4.0","cdate":1730601309422,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission13479/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731888671509,"domain":"ICLR.cc/2025/Conference","replyto":"M922KJFO7O","id":"a1JZY6D13I","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["KV cache","large language models","clustering"]},"supplementary_material":{"value":"/attachment/3ac354289ef3639bd96898d40b95bcff5c18dd7e.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Despite the significant success of large language models (LLMs), their extensive memory requirements pose challenges for deploying them in long-context token generation. The substantial memory footprint of LLM decoders arises from the necessity to store all previous tokens in the attention module, a requirement imposed by key-value (KV) caching. In this work, our focus is on developing an efficient compression technique for the KV cache. Empirical evidence indicates a significant clustering tendency within key embeddings in the attention module. Building on this key insight, we have devised a novel caching method with sublinear complexity, employing online clustering on key tokens and online \n sampling on values. The result is a provably accurate and efficient attention decoding algorithm, termed ClusterGen. Not only does this algorithm ensure a sublinear memory footprint and sublinear time complexity, but we also establish a tight error bound for our approach. Empirical evaluations on long-context question-answering tasks demonstrate that ClusterGen significantly outperforms existing and state-of-the-art KV cache compression methods in terms of performance and efficiency."},"_bibtex":{"value":"@misc{\nzandieh2025clustergen,\ntitle={ClusterGen: Token Generation in Sublinear Time and Memory with Clustering {KV} Cache},\nauthor={Amir Zandieh and Insu Han and Vahab Mirrokni and Amin Karbasi},\nyear={2025},\nurl={https://openreview.net/forum?id=M922KJFO7O}\n}"},"title":{"value":"ClusterGen: Token Generation in Sublinear Time and Memory with Clustering KV Cache"},"pdf":{"value":"/pdf/3e04eac1d99f4ccb46e0ab59726b90441e799ce1.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"zandieh|clustergen_token_generation_in_sublinear_time_and_memory_with_clustering_kv_cache"},"authorids":{"value":["~Amir_Zandieh2","~Insu_Han1","~Vahab_Mirrokni2","~Amin_Karbasi3"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Amir Zandieh","Insu Han","Vahab Mirrokni","Amin Karbasi"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"https://arxiv.org/pdf/2410.03065v2"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"jin|compute_or_load_kv_cache_why_not_both"},"html":{"value":"https://doi.org/10.48550/arXiv.2410.03065"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2410-03065,\n publtype={informal},\n author={Shuowei Jin and Xueshen Liu and Qingzhao Zhang and Z. Morley Mao},\n title={Compute Or Load KV Cache? Why Not Both?},\n year={2024},\n cdate={1704067200000},\n journal={CoRR},\n volume={abs/2410.03065},\n url={https://doi.org/10.48550/arXiv.2410.03065}\n}\n"},"abstract":{"value":"Large Language Models (LLMs) are increasingly deployed in large-scale online services, enabling sophisticated applications. However, the computational overhead of generating key-value (KV) caches in the prefill stage presents a major bottleneck, particularly for long-context inputs. Prefix caching mitigates this issue by storing KV caches for reuse, reducing redundant computation. Despite its advantages, prefix caching suffers from high latency due to the limited I/O bandwidth of storage devices, constraining inference efficiency. To address this challenge, we introduce Cake, a novel KV cache loading system that optimally utilizes both computational and I/O resources in parallel. Cake employs a bidirectional scheduling strategy that dynamically balances KV cache computation and loading, ensuring efficient resource utilization. Additionally, Cake incorporates an adaptive scheduling mechanism that seamlessly integrates with non-prefix caching requests, improving system throughput and adapting to fluctuating resource availabilty. Through extensive evaluations across various hardware configurations, datasets, and storage conditions, Cake achieves on average 2.6x reduction in Time to First Token (TTFT) compared to compute-only and I/O-only methods. Our findings highlight Cake as an effective and practical solution for optimizing long-context LLM inference, bridging the gap between computation and I/O efficiency in large-scale AI deployments."},"title":{"value":"Compute Or Load KV Cache? Why Not Both?"},"authors":{"value":[{"fullname":"Shuowei Jin","username":""},{"fullname":"Xueshen Liu","username":""},{"fullname":"Qingzhao Zhang","username":"~Qingzhao_Zhang1"},{"fullname":"Z. Morley Mao","username":""}]}},"tmdate":1784546234815,"pdate":1735603200000,"externalIds":["dblp:journals/corr/abs-2410-03065"],"tcdate":1784546231772,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Qingzhao_Zhang1"],"forum":"LIAbmwCgnv","license":"CC BY-SA 4.0","number":72048,"cdate":1704067200000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1784546234815,"domain":"OpenReview.net/Public_Article","id":"LIAbmwCgnv","version":2},{"content":{"venue":{"value":"ICML 2025"},"pdf":{"value":"https://raw.githubusercontent.com/mlresearch/v267/main/assets/jin25d/jin25d.pdf"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"jin|compute_or_load_kv_cache_why_not_both"},"html":{"value":"https://proceedings.mlr.press/v267/jin25d.html"},"_bibtex":{"value":"@inproceedings{DBLP:conf/icml/JinLZ025,\n author={Shuowei Jin and Xueshen Liu and Qingzhao Zhang and Zhuoqing Mao},\n title={Compute or Load KV Cache? Why Not Both?},\n year={2025},\n cdate={1735689600000},\n url={https://proceedings.mlr.press/v267/jin25d.html},\n booktitle={ICML},\n crossref={conf/icml/2025}\n}\n"},"abstract":{"value":"Large Language Models (LLMs) are increasingly deployed in large-scale online services, enabling sophisticated applications. However, the computational overhead of generating key-value (KV) caches in the prefill stage presents a major bottleneck, particularly for long-context inputs. Prefix caching mitigates this issue by storing KV caches for reuse, reducing redundant computation. Despite its advantages, prefix caching suffers from high latency due to the limited I/O bandwidth of storage devices, constraining inference efficiency. To address this challenge, we introduce Cake, a novel KV cache loading system that optimally utilizes both computational and I/O resources in parallel. Cake employs a bidirectional scheduling strategy that dynamically balances KV cache computation and loading, ensuring efficient resource utilization. Additionally, Cake incorporates an adaptive scheduling mechanism that seamlessly integrates with non-prefix caching requests, improving system throughput and adapting to fluctuating resource availabilty. Through extensive evaluations across various hardware configurations, datasets, and storage conditions, Cake achieves on average 2.6$\\times$ reduction in Time to First Token (TTFT) compared to compute-only and I/O-only methods. Our findings highlight Cake as an effective and practical solution for optimizing long-context LLM inference, bridging the gap between computation and I/O efficiency in large-scale AI deployments."},"title":{"value":"Compute or Load KV Cache? Why Not Both?"},"authors":{"value":[{"fullname":"Shuowei Jin","username":""},{"fullname":"Xueshen Liu","username":""},{"fullname":"Qingzhao Zhang","username":"~Qingzhao_Zhang1"},{"fullname":"Zhuoqing Mao","username":""}]}},"tmdate":1784546234301,"pdate":1767139200000,"externalIds":["dblp:conf/icml/JinLZ025"],"tcdate":1784546231822,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Qingzhao_Zhang1"],"forum":"1g9lgaCzS2","license":"CC BY-SA 4.0","number":72050,"cdate":1735689600000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1784546234301,"domain":"OpenReview.net/Public_Article","id":"1g9lgaCzS2","version":2},{"content":{"venue":{"value":"ICML 2025 poster"},"TLDR":{"value":"Cake is a KV cache loading system that optimally balances computation and I/O resources through a bidirectional scheduling strategy, reducing Time to First Token (TTFT) by 2.6× on average and improving long-context LLM inference efficiency."},"keywords":{"value":["Prefix caching System Optimization","KV Cache","Low-Latency LLM Inference"]},"primary_area":{"value":"general_machine_learning->hardware_and_software"},"abstract":{"value":"Large Language Models (LLMs) are increasingly deployed in large-scale online services, enabling sophisticated applications. However, the computational overhead of generating key-value (KV) caches in the prefill stage presents a major bottleneck, particularly for long-context inputs. Prefix caching mitigates this issue by storing KV caches for reuse, reducing redundant computation. Despite its advantages, prefix caching suffers from high latency due to the limited I/O bandwidth of storage devices, constraining inference efficiency. To address this challenge, we introduce Cake, a novel KV cache loading system that optimally utilizes both computational and I/O resources in parallel. Cake employs a bidirectional scheduling strategy that dynamically balances KV cache computation and loading, ensuring efficient resource utilization. Additionally, Cake incorporates an adaptive scheduling mechanism that seamlessly integrates with non-prefix caching requests, improving system throughput and adapting to fluctuating resource availabilty. Through extensive evaluations across various hardware configurations, datasets, and storage conditions, Cake achieves on average 2.6× reduction in Time to First Token (TTFT) compared to compute-only and I/O-only methods. Our findings highlight Cake as an effective and practical solution for optimizing long-context LLM inference, bridging the gap between computation and I/O efficiency in large-scale AI deployments."},"_bibtex":{"value":"@inproceedings{\njin2025compute,\ntitle={Compute or Load {KV} Cache? Why Not Both?},\nauthor={Shuowei Jin and Xueshen Liu and Qingzhao Zhang and Zhuoqing Mao},\nbooktitle={Forty-second International Conference on Machine Learning},\nyear={2025},\nurl={https://openreview.net/forum?id=WOyOtaO6lQ}\n}"},"title":{"value":"Compute or Load KV Cache? Why Not Both?"},"pdf":{"value":"/pdf/367a534dac0cfcac16a8506794609c1bc6a4a15c.pdf"},"lay_summary":{"value":"Large Language Models (LLMs) have become vital tools in various online applications, helping users with tasks like answering questions, summarizing documents, and chatting naturally. However, using these models with lengthy text inputs (like an entire book or a long conversation) can be slow, as they need to perform extensive calculations before providing their first response.\n\nPrevious approaches typically relied solely on loading pre-calculated data from storage, but these methods alone have limitations and often fail to sufficiently reduce the latency experienced by users, especially when storage bandwidth is limited. In this work, we introduce a new approach called \"Cake,\" which smartly combines recomputing calculations and loading pre-stored data simultaneously. Cake efficiently manages resources by recomputing certain data while loading other parts from storage, significantly reducing the waiting time even under constrained storage bandwidth.\n\nBy dynamically adapting to available computing power and storage bandwidth, Cake effectively handles real-world conditions, enhancing the responsiveness of LLMs. Our extensive experiments demonstrate that Cake consistently accelerates the initial response time, making large language models faster and more practical for everyday applications."},"venueid":{"value":"ICML.cc/2025/Conference"},"paperhash":{"value":"jin|compute_or_load_kv_cache_why_not_both"},"authorids":{"value":["~Shuowei_Jin1","~Xueshen_Liu1","~Qingzhao_Zhang1","~Zhuoqing_Mao1"]},"authors":{"value":["Shuowei Jin","Xueshen Liu","Qingzhao Zhang","Zhuoqing Mao"]}},"tmdate":1753292800723,"pdate":1746105109730,"tcdate":1736712133166,"writers":["ICML.cc/2025/Conference","ICML.cc/2025/Conference/Submission1121/Authors"],"signatures":["ICML.cc/2025/Conference/Submission1121/Authors"],"forum":"WOyOtaO6lQ","license":"CC BY 4.0","number":1121,"cdate":1736712133166,"readers":["everyone"],"invitations":["ICML.cc/2025/Conference/-/Submission","ICML.cc/2025/Conference/-/Post_Submission","ICML.cc/2025/Conference/Submission1121/-/Full_Submission","ICML.cc/2025/Conference/-/Edit","ICML.cc/2025/Conference/Submission1121/-/Camera_Ready_Revision"],"mdate":1753292800723,"odate":1750230921197,"domain":"ICML.cc/2025/Conference","id":"WOyOtaO6lQ","version":2},{"content":{"summary":{"value":"The paper introduces RCStat, a method for measuring attribution to prior context by using unnormalized attention weights. It proposes both an exact form and a more efficient approximation of this measure. Then, the paper demonstrates the utility of RCStat by using it to develop both a KV cache eviction method and text attribution method."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"Q1. Do you have any intuition for why VeriGran performance drops so much more dramatically than QuoteSum performance with increasing number of heads for Qwen in Fig 8b? \n\nQ2. My main critique is that the paper, in trying to do a lot of different things, is not quite doing all of them to a sufficient level. In your conceptualization of this work, do you see the theoretical framing of RCStat itself as the main contribution (with RCStat-based KV cache eviction + RCStat-based attribution being two example applications), or would you place all three on equal footing? \n\nQ3. Like most attention-based methods, this requires explicitly instantiating the attention matrix in memory, instead of calling an efficient kernelized implementation, right? Can you discuss the VRAM requirements imposed by this? \n\nQ4. various specific questions/feedback on the sections, as detailed in the weaknesses above. \n\nSmall typos:\n- line 463: generalizability misspelled\n- line 100: this citation for the phrase mechanistic interpretability seems strange to me -- not clear why this must be cited or that this is the right thing to cite? I'm not as familiar with this literature though so if there is a real reason for this please disregard.\n- line 112: \"up to 84%\" is what that work observed, but this phrasing makes it seem like this is a hard-and-fast maximum, which of course is not true\n- line 419: \"the an\"; more generally I don't really understand what this sentence is trying to say"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"S1. The idea of RCStat is intriguing-- I like the argument that looking at weights post-softmax loses valuable information. To the best of my knowledge, this is a novel way of measuring influence from prior context.\n\nS2. The formulation of RCStat with separation of context into two contrastable segments is also interesting, and the way that this allows for span-level identification of importance is a nice property.\n\nS3. Attribution is a really hard problem, and one that this paper makes an nice contribution towards. I think the analysis of attribution performance growing worse with adding too many irrelevant attention heads is an interesting one."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"W1. The paper presents a lot of insights, which is nice, but because it takes on three different focuses (deriving RCStat, using RCStat to design a competitive-with-SOTA KV cache eviction strategy, and analyzing RCStat's usefulness for attribution), it sometimes struggles to provide sufficient evidence for each. I organize the rest of the critique into these three subparts. \n\nW2. The initial explanation of RCStat was a bit hard to follow. I found the shapes of density functions drawn in 2(d) to be a bit distracting, and something that it would have been helpful to have clarified earlier-- in particular, why do these density functions overlap only slightly? \n\nW3. To make an argument about applying RCStat as a new KV cache eviction method, you really must address latency in the main text; I think the results in the appendix are somewhat promising, but I don't agree with the statement that this is only \"modestly slower\" than methods that are 4-5x faster per layer in the compression step (I understand the argument that, compared to prefill+decode, the overall time difference is negligible, but if that is the case you want to make, you should additionally report some metrics of overall time!). ROUGE-1 is not a reasonable quality metric to use; even an older neural metric would make a stronger case. And it seems this is missing several KV cache methods that are prominent in the community -- what about H20 or Pyramid KV? Why is this particular set of baselines the most reasonable choice? \n\nW4. The attribution section is interesting, but I think could benefit from some presentation improvements. Can you report standard deviations and/or significance test the differences between scores in Table 1? L3.1-8B's HS baseline is in a different section from the other L3.1-3B results, but for the Llama 8B and Qwen numbers this is in the same grouping. The pre/post comparison for Llama 8B might be better positioned in a separate table. Are the \"least RC\" numbers listed for the other models the pre-softmax ones? I was also wishing for a bit more exploration of which heads were good predictors of attribution-- the relevance of depth is mentioned a few times (and discussed in other work), but it would be interesting to hear more about how this works for RCStat's measure of head relevance in particular."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762924229287,"tcdate":1762146469167,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission13660/Reviewer_kCPH"],"signatures":["ICLR.cc/2026/Conference/Submission13660/Reviewer_kCPH"],"forum":"3zdoLd2Q93","number":3,"license":"CC BY 4.0","cdate":1762146469167,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission13660/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762924229287,"domain":"ICLR.cc/2026/Conference","replyto":"3zdoLd2Q93","id":"QgreKlTgVd","forumContent":{"TLDR":{"value":"RCStat leverages raw, pre-Softmax attention logits to quantify token importance at multiple granularities, enabling adaptive KV-cache eviction and high-fidelity attributions, all without any model retraining."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Large Language Models","Pre-Softmax Analysis","Attention Logits","Relative Contextualization","KV-Compression","Attribution"]},"supplementary_material":{"value":"/attachment/6eee29ca38110c853a2ddf3d268fdb3db484eafc.zip"},"primary_area":{"value":"generative models"},"abstract":{"value":"Estimating the importance of input tokens and their activations in auto-regressive models is a fundamental requirement in many applications, such as key-value (KV) cache compression and attribution. Prior work computes token importance using attention weights, which are obtained by normalizing the raw attention logits (query-key inner products) with a softmax operation. However, the softmax normalization suppresses the rich information within the attention logits. We introduce RCStat, a statistical framework that harnesses the raw attention logits via Relative Contextualization (RC) -- a random variable measuring contextual influence from one subset of tokens to another. We derive computationally efficient bounds on the expected RC and demonstrate its utility in two applications: (i) KV compression, where RC‐based adaptive thresholding evicts substantial portions of the KV cache with minimal quality loss in token generation; and (ii) Attribution, where attention heads with high expected RC yield accurate span‐level attribution. Across QA, summarization, and attribution benchmarks, RCStat achieves state-of-the-art performance, improving generation quality by 15–40\\% and attribution accuracy by 2–16\\%, all without any model retraining."},"_bibtex":{"value":"@misc{\nmahapatra2026rcstat,\ntitle={{RCS}tat: A Statistical Framework of Relative Contextualization in Transformers},\nauthor={Debabrata Mahapatra and Shubham Agarwal and Apoorv Saxena and Subrata Mitra},\nyear={2026},\nurl={https://openreview.net/forum?id=3zdoLd2Q93}\n}"},"title":{"value":"RCStat: A Statistical Framework of Relative Contextualization in Transformers"},"pdf":{"value":"/pdf/b9876d6b2b6c2a51d9fd990e69571905b47f6678.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"mahapatra|rcstat_a_statistical_framework_of_relative_contextualization_in_transformers"},"authorids":{"value":["~Debabrata_Mahapatra1","~Shubham_Agarwal4","~Apoorv_Saxena1","~Subrata_Mitra1"]},"authors":{"value":["Debabrata Mahapatra","Shubham Agarwal","Apoorv Saxena","Subrata Mitra"]}},"version":2},{"content":{"TLDR":{"value":"We proposed SyncKV, which compresses the KV cache by dynamically recalling important tokens, thereby reducing GPU memory usage and accelerating long context inference."},"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["NLP","Large Language Models"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"KV cache accelerates the inference of Large Language Models (LLMs) by caching the key and value states of previous tokens, but its linearly increasing memory footprint poses a huge bottleneck for long-context tasks. To mitigate this, many previous studies evict unimportant tokens based on attention scores from the prefill stage or cumulative attention. However, by permanently evicting tokens, such static compression algorithms fail to preserve globally important tokens, as they overlook the \"attention drift\" phenomenon inherent in inference. Our analysis highlights this drift, showing that after generating just 50 tokens, the set of important tokens retains only about a 30\\% overlap with the one identified during the prefill stage. To address this, our core innovative insight is twofold: (1) the set of important tokens exhibits high temporal locality across adjacent generation steps, and (2) this set is highly similar among attention heads within the same layer. Based on these insights, we propose SyncKV, a training-free dynamic KV cache compression method. SyncKV takes advantage of these properties through a novel syncopated strategy in which a few \"representative heads\" periodically identify important tokens, triggering an asynchronous upload of the relevant KV cache from the CPU. We designed a parallelization strategy that overlaps the I/O overhead with the subsequent forward computing stage, thereby effectively hiding the delay of data transmission and achieving an acceleration effect. Experiments show that SyncKV has achieved state-of-the-art performance in multiple long-context benchmarks, reducing the GPU memory usage of the KV cache by up to 80\\%. Our code will be open-source."},"_bibtex":{"value":"@misc{\nshi2025synckv,\ntitle={Sync{KV}: A Syncopated Scheduling Approach to {KV} Cache Compression for Efficient Long-Context {LLM} Inference},\nauthor={Zhiyuan Shi and Qibo Qiu and Xuefeng and Zhonglin Jiang and Li Yu and Jian Jiang and Xiaofei He and Wenxiao Wang},\nyear={2025},\nurl={https://openreview.net/forum?id=542c8KxeQt}\n}"},"title":{"value":"SyncKV: A Syncopated Scheduling Approach to KV Cache Compression for Efficient Long-Context LLM Inference"},"pdf":{"value":"/pdf/8024a2877273d4209dd925ca67e25152deb02107.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"shi|synckv_a_syncopated_scheduling_approach_to_kv_cache_compression_for_efficient_longcontext_llm_inference"},"authorids":{"value":["~Zhiyuan_Shi7","~Qibo_Qiu1","~Xuefeng1","~Zhonglin_Jiang1","~Li_Yu11","~Jian_Jiang7","~Xiaofei_He2","~Wenxiao_Wang2"]},"authors":{"value":["Zhiyuan Shi","Qibo Qiu","Xuefeng","Zhonglin Jiang","Li Yu","Jian Jiang","Xiaofei He","Wenxiao Wang"]}},"tmdate":1764579136973,"tcdate":1758008966957,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7133/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission7133/Authors"],"forum":"542c8KxeQt","license":"CC BY 4.0","number":7133,"cdate":1758008966957,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/-/Submission","ICLR.cc/2026/Conference/-/Post_Submission","ICLR.cc/2026/Conference/Submission7133/-/Full_Submission","ICLR.cc/2026/Conference/-/Withdrawn_Submission"],"mdate":1764579136973,"odate":1759896705795,"domain":"ICLR.cc/2026/Conference","id":"542c8KxeQt","version":2},{"content":{"venue":{"value":"ICLR 2027 Conference Submission"},"TLDR":{"value":"QREAM proposes a training-free KV-cache compression method based on attention matching, with a simple and deployment-friendly formulation that achieves competitive compression performance."},"keywords":{"value":["KV cache compression","attention matching","long-context","inference"]},"supplementary_material":{"value":"/attachment/ca455888dcd761c2d670d8336d53c6794a32aaef.zip"},"primary_area":{"value":"optimization"},"abstract":{"value":"Large key-value (KV) caches are a major bottleneck for long-context large language model (LLM) inference, motivating methods that reduce their size while preserving information needed for future queries. Attention matching (AM) has recently emerged as a practical and theoretically grounded framework for controlling attention behavior, with successful applications to prefix tuning and KV-cache compression. However, existing AM-based methods for KV-cache compression separately match the numerator and normalization factor of attention, requiring auxiliary per-key biases that are incompatible with standard KV-cache representations. In this work, we propose QREAM, a bias-free formulation of attention matching that directly matches the normalized attention output rather than its numerator and normalization factor separately. Once the compressed keys are selected, this formulation reduces attention matching to an efficient linear solve over the compressed values, requiring neither training nor modifications to the standard KV-cache representation. To select informative keys, QREAM further introduces a pivoted-QR-based procedure that favors linearly independent attention contributions, providing an efficient alternative to matching-pursuit methods. We evaluate QREAM across multiple Qwen model scales, long-context QA benchmarks, and compression ratios from 5× to 100×. QREAM is competitive with existing attention-matching methods while substantially outperforming token-scoring baselines under aggressive compression, despite its simpler and more deployment-friendly formulation."},"_bibtex":{"value":"@inproceedings{\nanonymous2026qream,\ntitle={{QREAM}: {KV} Cache Compression via Pivoted {QR} and Explicit Attention Matching},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=QJoILTys3B},\nnote={under review}\n}"},"title":{"value":"QREAM: KV Cache Compression via Pivoted QR and Explicit Attention Matching"},"pdf":{"value":"/pdf/cc9b996d23a3c737959abd18525842d1c0bf51d1.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791231861604,"tcdate":1789681972989,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission34732/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission34732/Authors"],"forum":"QJoILTys3B","license":"CC BY 4.0","number":34732,"cdate":1789681972989,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission34732/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791231861604,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"QJoILTys3B","version":2},{"content":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["LLM","KV cache compression","low-resource","token eviction"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Large Language Models (LLMs) require substantial computational resources during generation. While the Key-Value (KV) cache significantly accelerates this process by storing attention intermediates, its memory footprint grows linearly with sequence length, batch size, and model size, creating a bottleneck in long-context scenarios. Various KV cache compression techniques, including token eviction, quantization, and low-rank projection, have been proposed to mitigate this bottleneck, often complementing each other.\nThis paper focuses on enhancing token eviction strategies.\nToken eviction leverages the observation that the attention patterns are often sparse, allowing for the removal of less critical KV entries to save memory. However, this reduction usually comes at the cost of notable accuracy degradation, particularly under high compression ratios. \nTo address this issue, we propose CaliDrop, a novel strategy that enhances token eviction through calibration. Our preliminary experiments show that queries at nearby positions exhibit high similarity. Building on this observation, CaliDrop performs speculative calibration on the discarded tokens to mitigate the accuracy loss caused by token eviction.\nExtensive experiments demonstrate that CaliDrop significantly improves the accuracy of existing token eviction methods."},"_bibtex":{"value":"@misc{\nsu2026calidrop,\ntitle={CaliDrop: {KV} Cache Compression with Query-based Calibration},\nauthor={Yi Su and Quantong Qiu and Yuechi Zhou and Qingrong Xia and Ping Li and Xinyu Duan and Zhefeng Wang and Juntao Li and Min Zhang},\nyear={2026},\nurl={https://openreview.net/forum?id=2WzCzpkeTc}\n}"},"title":{"value":"CaliDrop: KV Cache Compression with Query-based Calibration"},"pdf":{"value":"/pdf/2d2ca0c88ea8ae9820d1913c0766766ed3a42c18.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"su|calidrop_kv_cache_compression_with_querybased_calibration"},"authorids":{"value":["~Yi_Su3","~Quantong_Qiu1","~Yuechi_Zhou1","~Qingrong_Xia1","~Ping_Li16","~Xinyu_Duan1","~Zhefeng_Wang1","~Juntao_Li2","~Min_Zhang9"]},"authors":{"value":["Yi Su","Quantong Qiu","Yuechi Zhou","Qingrong Xia","Ping Li","Xinyu Duan","Zhefeng Wang","Juntao Li","Min Zhang"]}},"tmdate":1770804960521,"tcdate":1758198498568,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission11414/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission11414/Authors"],"forum":"2WzCzpkeTc","license":"CC BY 4.0","number":11414,"cdate":1758198498568,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/-/Submission","ICLR.cc/2026/Conference/-/Post_Submission","ICLR.cc/2026/Conference/Submission11414/-/Full_Submission","ICLR.cc/2026/Conference/-/Edit"],"mdate":1770804960521,"odate":1759896705795,"domain":"ICLR.cc/2026/Conference","id":"2WzCzpkeTc","version":2},{"content":{"venue":{"value":"ICLR 2027 Conference Submission"},"TLDR":{"value":"Training a reasoning model to compress each finished block of its own chain of thought into a few learned KV entries matches full-KV accuracy at a quarter of the cache and finishes problems 3.2× faster in vLLM."},"keywords":{"value":["Key-Value Compression","Large Language Model","Reasoning"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Long chain-of-thought decoding is memory-bound: the KV cache of the generated tokens grows linearly with the test-time\ncompute being spent. Evicting cache entries does not fix this, because a model never trained on a cache with entries removed\ngenerates longer, often up to the length limit. We instead train the model to compress its own output. \\method{} writes each\ncompleted block of generated text into the KV entries of a few learned pause tokens and frees the block's cache, trained end to\nend with the next-token loss. Three novel choices make this work: the summary rows take positions inside the block they\nreplace, the row budget is spread across layers by each layer's measured value, and the blocks are large (512 tokens). On 446 competition problems with 32K-token generations,\nkeeping a quarter of the generated cache matches an identically trained full-KV model in accuracy and output length; the\nstrongest published evictor at a matched budget writes far longer, and the closest trained compressor, retrained on the same\ndata, loses 10 points. Served in vLLM, the cache is $3.5\\times$ smaller per sequence and a saturated H100 finishes a problem $3.2\\times$\nfaster than with full KV."},"_bibtex":{"value":"@inproceedings{\nanonymous2026abridgekv,\ntitle={Abridge{KV}: Compressing Long Reasoning into the {KV} Cache of Learned Pause Tokens},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=1BGsVqZmAl},\nnote={under review}\n}"},"title":{"value":"AbridgeKV: Compressing Long Reasoning into the KV Cache of Learned Pause Tokens"},"pdf":{"value":"/pdf/766c10ba1dffdf35148aec01f3cc869813e62c3a.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791231673977,"tcdate":1789664022731,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission33320/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission33320/Authors"],"forum":"1BGsVqZmAl","license":"CC BY 4.0","number":33320,"cdate":1789664022731,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission33320/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791231673977,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"1BGsVqZmAl","version":2},{"content":{"venue":{"value":"ICLR 2027 Conference Desk Rejected Submission"},"keywords":{"value":["Modular Models","Shadow Residual","SR","Low-Rank Adaptation","LoRA","Activated LoRA","aLoRA","Parameter-Efficient Fine-Tuning","PEFT","Adapter-Based Learning","Adapter Composition","Capability Modularity","Foundation Models","Large Language Models","LLMs","Transformer Decoders","KV Cache","KV Cache Reuse","KV Cache Contamination","Attention Mechanisms","Attention State Decoupling","Residual Streams","Parallel Residual Architecture","Efficient Inference","Scalable Model Serving","Adapter Switching","Multi-Capability Serving","Shared Base Models","Model Composition","Memory-Efficient Inference","Context Reuse","Cache-Efficient Serving","Modular AI","Composable AI Systems"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Modular models aim to decouple capability development from foundation model training, allowing capabilities to be independently developed, shared, and composed through lightweight adapters while leveraging a common foundation model. This paradigm enables many capabilities to be served from a single model rather than maintaining separate models for each task. Realizing this vision requires an inference architecture that supports efficient capability composition and adapter switching while introducing minimal overhead relative to the shared base model. Low-Rank Adaptation (LoRA) provides a natural building block for modular models, enabling independently trained adapters to be selected and composed over a shared backbone. However, LoRA adapters modify the key and value projections in the self-attention mechanism. As a result, the KV cache stores adapter-specific representations, which can prevent subsequent adapters from being safely activated on the same context, a phenomenon we refer to as KV-cache contamination. We show that KV-cache contamination consistently degrades performance across adaptation targets, contamination levels, and adapter ranks, reaching up to 28.5 percentage points in the most severe settings, increasing as more context is cached under prior adapters. To mitigate this issue, we introduce Shadow Residual (SR), a transformer decoder architecture that decouples adapter-specific computation from the cached attention state. SR maintains capability-specific information in a parallel residual stream while the KV cache is written exclusively by the base model. This enables arbitrary adapter transitions within a single generation without rebuilding cached states or incurring additional KV-memory overhead. Across diverse tasks, domains, context lengths, and model scales (3B-30B parameters), SR recovers 85.2\\% of the performance gain achieved by LoRA while eliminating KV-cache contamination. Beyond quality, SR enables efficient multi-adapter execution, reducing time-to-completion by up to $10.34\\times$ on ReAct-style agent workloads. These results establish Shadow Residual as an an effective architectural primitive and efficient capability composition."},"_bibtex":{"value":"@misc{\nanonymous2026shadow,\ntitle={Shadow Residual: Dual-Stream Low-Rank Adaptation for {KV}-Cache Reuse},\nauthor={Anonymous},\nyear={2026},\nurl={https://openreview.net/forum?id=eGfO3bTlNY}\n}"},"title":{"value":"Shadow Residual: Dual-Stream Low-Rank Adaptation for KV-Cache Reuse"},"pdf":{"value":"/pdf/f95346955a3a7ba5a3873a878aaa86822c895618.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Desk_Rejected_Submission"}},"tmdate":1791218486451,"tcdate":1789642148104,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission30614/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission30614/Authors"],"forum":"eGfO3bTlNY","license":"CC BY 4.0","number":30614,"cdate":1789642148104,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission30614/-/Full_Submission","ICLR.cc/2027/Conference/-/Desk_Rejected_Submission","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791218486451,"domain":"ICLR.cc/2027/Conference","id":"eGfO3bTlNY","version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2510.01290v2"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"ramachandran|thinkv_thoughtadaptive_kv_cache_compression_for_efficient_reasoning_models"},"html":{"value":"https://doi.org/10.48550/arXiv.2510.01290"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2510-01290,\n publtype={informal},\n author={Akshat Ramachandran and Marina Neseem and Charbel Sakr and Rangharajan Venkatesan and Brucek Khailany and Tushar Krishna},\n title={ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Models},\n year={2025},\n month={October},\n cdate={1759276800000},\n journal={CoRR},\n volume={abs/2510.01290},\n url={https://doi.org/10.48550/arXiv.2510.01290}\n}\n"},"abstract":{"value":"The long-output context generation of large reasoning models enables extended chain of thought (CoT) but also drives rapid growth of the key-value (KV) cache, quickly overwhelming GPU memory. To address this challenge, we propose ThinKV, a thought-adaptive KV cache compression framework. ThinKV is based on the observation that attention sparsity reveals distinct thought types with varying importance within the CoT. It applies a hybrid quantization-eviction strategy, assigning token precision by thought importance and progressively evicting tokens from less critical thoughts as reasoning trajectories evolve. Furthermore, to implement ThinKV, we design a kernel that extends PagedAttention to enable efficient reuse of evicted tokens' memory slots, eliminating compaction overheads. Extensive experiments on DeepSeek-R1-Distill, GPT-OSS, and NVIDIA AceReason across mathematics and coding benchmarks show that ThinKV achieves near-lossless accuracy with less than 5% of the original KV cache, while improving performance with up to 5.8x higher inference throughput over state-of-the-art baselines."},"title":{"value":"ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Models"},"authors":{"value":[{"fullname":"Akshat Ramachandran","username":""},{"fullname":"Marina Neseem","username":""},{"fullname":"Charbel Sakr","username":""},{"fullname":"Rangharajan Venkatesan","username":""},{"fullname":"Brucek Khailany","username":""},{"fullname":"Tushar Krishna","username":"~Tushar_Krishna1"}]}},"tmdate":1785994485086,"pdate":1767139200000,"externalIds":["dblp:journals/corr/abs-2510-01290"],"tcdate":1785994472188,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Tushar_Krishna1"],"forum":"PZ5JBBDcaM","license":"CC BY-SA 4.0","number":124032,"cdate":1759276800000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1785994485086,"domain":"OpenReview.net/Public_Article","id":"PZ5JBBDcaM","version":2},{"content":{"venue":{"value":"ICLR 2026 Oral"},"keywords":{"value":["Large Reasoning Models","KV Cache Compression","Quantization","Eviction","Sparsity","Thought-Aware Compression"]},"primary_area":{"value":"generative models"},"abstract":{"value":"The long-output context generation of large reasoning models enables extended chain of thought (CoT) but also drives rapid growth of the key–value (KV) cache, quickly overwhelming GPU memory. To address this challenge, we propose ThinKV, a thought-adaptive KV cache compression framework. ThinKV is based on the observation that attention sparsity reveals distinct thought types with varying importance within the CoT. It applies a hybrid quantization–eviction strategy, assigning token precision by thought importance and progressively evicting tokens from less critical thoughts as reasoning trajectories evolve. Furthermore, to implement ThinKV, we design a kernel that extends PagedAttention to enable efficient reuse of evicted tokens' memory slots, eliminating compaction overheads. Extensive experiments on DeepSeek-R1-Distill, GPT-OSS, and NVIDIA AceReason across mathematics and coding benchmarks show that ThinKV achieves near-lossless accuracy with less than 5% of the original KV cache, while improving performance with up to 5.8x higher inference throughput over SoTA baselines."},"_bibtex":{"value":"@inproceedings{\nramachandran2026thinkv,\ntitle={Thin{KV}: Thought-Adaptive {KV} Cache Compression for Efficient Reasoning Models},\nauthor={Akshat Ramachandran and Marina Neseem and Charbel Sakr and Rangharajan Venkatesan and Brucek Khailany and Tushar Krishna},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=M3CeHnZKNC}\n}"},"title":{"value":"ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Models"},"pdf":{"value":"/pdf/9bd2af3f279ffb70af78d6556ff3adb8be880e97.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"ramachandran|thinkv_thoughtadaptive_kv_cache_compression_for_efficient_reasoning_models"},"authorids":{"value":["~Akshat_Ramachandran1","~Marina_Neseem2","~Charbel_Sakr1","~Rangharajan_Venkatesan1","~Brucek_Khailany1","~Tushar_Krishna1"]},"authors":{"value":["Akshat Ramachandran","Marina Neseem","Charbel Sakr","Rangharajan Venkatesan","Brucek Khailany","Tushar Krishna"]}},"tmdate":1775876924152,"pdate":1769435730725,"tcdate":1757440525559,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission3474/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission3474/Authors"],"forum":"M3CeHnZKNC","license":"CC BY 4.0","number":3474,"cdate":1757440525559,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/-/Submission","ICLR.cc/2026/Conference/-/Post_Submission","ICLR.cc/2026/Conference/Submission3474/-/Full_Submission","ICLR.cc/2026/Conference/Submission3474/-/Rebuttal_Revision","ICLR.cc/2026/Conference/-/Edit","ICLR.cc/2026/Conference/Submission3474/-/Camera_Ready_Revision"],"mdate":1775876924152,"odate":1759896705795,"domain":"ICLR.cc/2026/Conference","id":"M3CeHnZKNC","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2402.18096v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"yang|no_token_left_behind_reliable_kv_cache_compression_via_importanceaware_mixed_precision_quantization"},"authorids":{"value":["~June_Yong_Yang1","~Byeongwook_Kim1","https://dblp.org/search/pid/api?q=author:Jeongin_Bae:","https://dblp.org/search/pid/api?q=author:Beomseok_Kwon:","~Gunho_Park1","https://dblp.org/search/pid/api?q=author:Eunho_Yang:","~Se_Jung_Kwon1","https://dblp.org/search/pid/api?q=author:Dongsoo_Lee:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2402.18096"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2402-18096,\n publtype={informal},\n author={June Yong Yang and Byeongwook Kim and Jeongin Bae and Beomseok Kwon and Gunho Park and Eunho Yang and Se Jung Kwon and Dongsoo Lee},\n title={No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization},\n year={2024},\n cdate={1704067200000},\n journal={CoRR},\n volume={abs/2402.18096},\n url={https://doi.org/10.48550/arXiv.2402.18096}\n}\n"},"abstract":{"value":"Key-Value (KV) Caching has become an essential technique for accelerating the inference speed and throughput of generative Large Language Models~(LLMs). However, the memory footprint of the KV cache poses a critical bottleneck in LLM deployment as the cache size grows with batch size and sequence length, often surpassing even the size of the model itself. Although recent methods were proposed to select and evict unimportant KV pairs from the cache to reduce memory consumption, the potential ramifications of eviction on the generative process are yet to be thoroughly examined. In this paper, we examine the detrimental impact of cache eviction and observe that unforeseen risks arise as the information contained in the KV pairs is exhaustively discarded, resulting in safety breaches, hallucinations, and context loss. Surprisingly, we find that preserving even a small amount of information contained in the evicted KV pairs via reduced precision quantization substantially recovers the incurred degradation. On the other hand, we observe that the important KV pairs must be kept at a relatively higher precision to safeguard the generation quality. Motivated by these observations, we propose \\textit{Mixed-precision KV cache}~(MiKV), a reliable cache compression method that simultaneously preserves the context details by retaining the evicted KV pairs in low-precision and ensure generation quality by keeping the important KV pairs in high-precision. Experiments on diverse benchmarks and LLM backbones show that our proposed method offers a state-of-the-art trade-off between compression ratio and performance, compared to other baselines."},"title":{"value":"No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization"},"authors":{"value":["June Yong Yang","Byeongwook Kim","Jeongin Bae","Beomseok Kwon","Gunho Park","Eunho Yang","Se Jung Kwon","Dongsoo Lee"]}},"tmdate":1747359135472,"pdate":1704067200000,"tcdate":1727756208774,"writers":["~"],"signatures":["~Byeongwook_Kim1"],"forum":"35mZgOWGNO","license":"CC BY-SA 4.0","number":127741,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1747359135472,"domain":"DBLP.org","id":"35mZgOWGNO","version":2},{"content":{"venue":{"value":"ICLR 2027 Conference Submission"},"TLDR":{"value":"HeteroFold enables prefill-free KV cache transfer across model families while maintaining receiver behavior."},"keywords":{"value":["KV transfer","Cross-Family","Multi-Agent Systems","Efficient Machine Learning"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Recent multi-agent LLM systems increasingly combine heterogeneous models for specialized agent roles. However, text-based communication requires each receiver to prefill shared context already processed by the sender. Reusing the sender's key-value (KV) cache avoids this redundancy, but prefill-free transfer across model families must handle differences in tokenization, model depth, and KV representations. To address these issues, we propose \\textit{HeteroFold}, a prefill-free cross-family KV cache transfer method that keeps both the sender and receiver frozen. HeteroFold aligns model structures, maps the sender cache into the receiver space, and calibrates it to preserve receiver behavior. Across six transfer directions, HeteroFold achieves the best cache-transfer performance on all four long-context benchmarks and most short-context settings. It also matches text-based communication on the multi-agent benchmark. At 32K context length, Llama-3.1-8B$\\rightarrow$Ministral-3-14B transfer is $10.7\\times$ faster than Native Prefill and $1.18$--$1.47\\times$ faster than the state-of-the-art prefill-free baselines, Dense Latent and KV Ridge. These results show that HeteroFold enables efficient cross-family KV reuse without receiver prefill."},"_bibtex":{"value":"@inproceedings{\nanonymous2026prefillfree,\ntitle={Prefill-Free Cross-Family {KV} Cache Transfer for Heterogeneous Multi-Agent {LLM}s},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=5gFgaB9Rym},\nnote={under review}\n}"},"title":{"value":"Prefill-Free Cross-Family KV Cache Transfer for Heterogeneous Multi-Agent LLMs"},"pdf":{"value":"/pdf/0491f8bfa247edb9a7e26ba1f6a0c4ce62db59ba.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791229935093,"tcdate":1789502175787,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission22260/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission22260/Authors"],"forum":"5gFgaB9Rym","license":"CC BY 4.0","number":22260,"cdate":1789502175787,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission22260/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791229935093,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"5gFgaB9Rym","version":2},{"content":{"venue":{"value":"EMNLP (Findings) 2024"},"pdf":{"value":"https://aclanthology.org/2024.findings-emnlp.235.pdf"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"wan|lookm_lookonce_optimization_in_kv_cache_for_efficient_multimodal_longcontext_inference"},"html":{"value":"https://doi.org/10.18653/v1/2024.findings-emnlp.235"},"_bibtex":{"value":"@inproceedings{DBLP:conf/emnlp/WanWLHZJW024,\n author={Zhongwei Wan and Ziang Wu and Che Liu and Jinfa Huang and Zhihong Zhu and Peng Jin and Longyue Wang and Li Yuan},\n title={LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference},\n year={2024},\n cdate={1704067200000},\n pages={4065-4078},\n url={https://doi.org/10.18653/v1/2024.findings-emnlp.235},\n booktitle={EMNLP (Findings)},\n crossref={conf/emnlp/2024f}\n}\n"},"abstract":{"value":"Long-context Multimodal Large Language Models (MLLMs) demand substantial computational resources for inference as the growth of their multimodal Key-Value (KV) cache, in response to increasing input lengths, challenges memory and time efficiency. Unlike single-modality LLMs that manage only textual contexts, the KV cache of long-context MLLMs includes representations from multiple images with temporal and spatial relationships and related textual contexts. The predominance of image tokens means traditional optimizations for LLMs’ KV caches are unsuitable for multimodal long-context settings, and no prior works have addressed this challenge.In this work, we introduce **LOOK-M**, a pioneering, fine-tuning-free approach that efficiently reduces the multimodal KV cache size while maintaining performance comparable to a full cache. We observe that during prompt prefill, the model prioritizes more textual attention over image features, and based on the multimodal interaction observation, a new proposed text-prior method is explored to compress the KV cache. Furthermore, to mitigate the degradation of image contextual information, we propose several compensatory strategies using KV pairs merging. **LOOK-M** demonstrates that with a significant reduction in KV Cache memory usage, such as reducing it by **80%** in some cases, it not only achieves approximately **1.3x** faster decoding but also maintains or even **enhances** performance across a variety of long context multimodal tasks."},"title":{"value":"LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference"},"authors":{"value":[{"fullname":"Zhongwei Wan","username":""},{"fullname":"Ziang Wu","username":""},{"fullname":"Che Liu","username":""},{"fullname":"Jinfa Huang","username":""},{"fullname":"Zhihong Zhu","username":"~Zhihong_Zhu1"},{"fullname":"Peng Jin","username":""},{"fullname":"Longyue Wang","username":""},{"fullname":"Li Yuan","username":""}]}},"tmdate":1784621726106,"pdate":1735603200000,"externalIds":["dblp:conf/emnlp/WanWLHZJW024"],"tcdate":1784621714956,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Zhihong_Zhu1"],"forum":"mHDHuWvqNr","license":"CC BY-SA 4.0","number":80787,"cdate":1704067200000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1784621726106,"domain":"OpenReview.net/Public_Article","id":"mHDHuWvqNr","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2406.18139v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"wan|lookm_lookonce_optimization_in_kv_cache_for_efficient_multimodal_longcontext_inference"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Zhongwei_Wan:","https://dblp.org/search/pid/api?q=author:Ziang_Wu:","~Che_Liu3","~Jinfa_Huang2","~Zhihong_Zhu1","https://dblp.org/search/pid/api?q=author:Peng_Jin_0001:","https://dblp.org/search/pid/api?q=author:Longyue_Wang:","~Li_Yuan2"]},"html":{"value":"https://doi.org/10.48550/arXiv.2406.18139"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2406-18139,\n publtype={informal},\n author={Zhongwei Wan and Ziang Wu and Che Liu and Jinfa Huang and Zhihong Zhu and Peng Jin and Longyue Wang and Li Yuan},\n title={LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference},\n year={2024},\n cdate={1704067200000},\n journal={CoRR},\n volume={abs/2406.18139},\n url={https://doi.org/10.48550/arXiv.2406.18139}\n}\n"},"abstract":{"value":"Long-context Multimodal Large Language Models (MLLMs) demand substantial computational resources for inference as the growth of their multimodal Key-Value (KV) cache, in response to increasing input lengths, challenges memory and time efficiency. Unlike single-modality LLMs that manage only textual contexts, the KV cache of long-context MLLMs includes representations from multiple images with temporal and spatial relationships and related textual contexts. The predominance of image tokens means traditional optimizations for LLMs' KV caches are unsuitable for multimodal long-context settings, and no prior works have addressed this challenge. In this work, we introduce LOOK-M, a pioneering, fine-tuning-free approach that efficiently reduces the multimodal KV cache size while maintaining performance comparable to a full cache. We observe that during prompt prefill, the model prioritizes more textual attention over image features, and based on the multimodal interaction observation, a new proposed text-prior method is explored to compress the KV cache. Furthermore, to mitigate the degradation of image contextual information, we propose several compensatory strategies using KV pairs merging. LOOK-M demonstrates that with a significant reduction in KV Cache memory usage, such as reducing it by 80% in some cases, it not only achieves up to 1.5x faster decoding but also maintains or even enhances performance across a variety of long context multimodal tasks."},"title":{"value":"LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference"},"authors":{"value":["Zhongwei Wan","Ziang Wu","Che Liu","Jinfa Huang","Zhihong Zhu","Peng Jin","Longyue Wang","Li Yuan"]}},"tmdate":1784621716326,"pdate":1704067200000,"tcdate":1727701379093,"writers":["~"],"signatures":["~Jinfa_Huang2"],"forum":"gq0TKSzSfV","license":"CC BY-SA 4.0","number":113074,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1784621716326,"domain":"DBLP.org","id":"gq0TKSzSfV","version":2},{"content":{"summary":{"value":"This paper proposes CommonKV, a training-free KV cache compression method. It addresses the dissimilarity of KV caches across layers by using offline Singular Value Decomposition (SVD) to share parameters between adjacent layers. This creates a highly consistent \"latent KV cache\" from similar hidden states, which can be merged with significantly less performance loss than merging the original KV caches directly."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"The idea of low rank decomposition and weight sharing sounds very related to \"LORC: Low-Rank Compression for LLMs KV Cache\nwith a Progressive Compression Strategy\", where attention weights and caches in the same layer are shared."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"The method's core strength is its novel idea of creating a more consistent latent cache via parameter sharing, which directly addresses the root cause of poor performance in direct KV cache merging. Its training-free nature makes it highly practical and easy to apply to existing models. CommonKV demonstrates superior empirical performance over baselines at high compression ratios with minimal inference latency overhead. Furthermore, its ability to be combined with quantization and eviction methods to achieve up to 98% compression is a significant advantage ."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The paper would be stronger with a more detailed sensitivity analysis of the SVD rank hyperparameter. Additionally, the handling of GQA models feels like a workaround, and a deeper analysis of the architectural interaction would be beneficial."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762926246304,"tcdate":1761971145585,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission16049/Reviewer_F8ju"],"signatures":["ICLR.cc/2026/Conference/Submission16049/Reviewer_F8ju"],"forum":"ELGBM1aEjB","number":3,"license":"CC BY 4.0","cdate":1761971145585,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission16049/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762926246304,"domain":"ICLR.cc/2026/Conference","replyto":"ELGBM1aEjB","id":"11CkmZXOix","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Efficient LLM","KV Cache Compression"]},"supplementary_material":{"value":"/attachment/11e26cc4f2ac92912e5fabbd45bfe9604418c39d.zip"},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"Large Language Models (LLMs) confront significant memory challenges due to the escalating KV cache with increasing sequence length. As a crucial technique, existing cross-layer KV cache sharing methods either necessitate modified model architectures with subsequent pre-training or incur significant performance degradation at high compression rates. To mitigate these challenges, we propose CommonKV, a training-free method for cross-layer KV cache compression through adjacent parameters sharing. Inspired by the high similarity observed in cross-layer hidden states, we utilize Singular Value Decomposition (SVD) to achieve weight sharing across adjacent parameters, resulting in a more easily mergeable latent KV cache. Furthermore, we also introduce an adaptive budget allocation strategy. It dynamically assigns compression budgets based on cosine similarity, ensuring that dissimilar caches are not over-compressed. Experiments across multiple backbone models and benchmarks including LongBench and Ruler demonstrate that the proposed method consistently outperforms other low-rank and cross-layer approaches at various compression ratios. Moreover, we find that the benefits of CommonKV are orthogonal to other quantization and eviction methods. By integrating these approaches, we can ultimately achieve a 98% compression ratio without significant performance loss."},"_bibtex":{"value":"@misc{\nwang2025commonkv,\ntitle={Common{KV}: Compressing {KV} Cache with Cross-layer Parameter Sharing},\nauthor={Yixuan Wang and Haoyu Qiao and Lujun Li and Sirui Han and Qingfu Zhu and Wanxiang Che},\nyear={2025},\nurl={https://openreview.net/forum?id=ELGBM1aEjB}\n}"},"title":{"value":"CommonKV: Compressing KV Cache with Cross-layer Parameter Sharing"},"pdf":{"value":"/pdf/5ee5af737ca8ec638238b55f1be3609d6c09f2b7.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"wang|commonkv_compressing_kv_cache_with_crosslayer_parameter_sharing"},"authorids":{"value":["~Yixuan_Wang6","~Haoyu_Qiao1","~Lujun_Li1","~Sirui_Han1","~Qingfu_Zhu1","~Wanxiang_Che1"]},"authors":{"value":["Yixuan Wang","Haoyu Qiao","Lujun Li","Sirui Han","Qingfu Zhu","Wanxiang Che"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2411.17089v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"jiang|efficient_llm_inference_with_ioaware_partial_kv_cache_recomputation"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Chaoyi_Jiang:","https://dblp.org/search/pid/api?q=author:Lei_Gao:","~Hossein_Entezari_Zarch1","~Murali_Annavaram1"]},"html":{"value":"https://doi.org/10.48550/arXiv.2411.17089"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2411-17089,\n publtype={informal},\n author={Chaoyi Jiang and Lei Gao and Hossein Entezari Zarch and Murali Annavaram},\n title={Efficient LLM Inference with I/O-Aware Partial KV Cache Recomputation},\n year={2024},\n cdate={1704067200000},\n journal={CoRR},\n volume={abs/2411.17089},\n url={https://doi.org/10.48550/arXiv.2411.17089}\n}\n"},"abstract":{"value":"Inference for Large Language Models (LLMs) is computationally demanding. To reduce the cost of auto-regressive decoding, Key-Value (KV) caching is used to store intermediate activations, enabling GPUs to perform only the incremental computation required for each new token. This approach significantly lowers the computational overhead for token generation. However, the memory required for KV caching grows rapidly, often exceeding the capacity of GPU memory. A cost-effective alternative is to offload KV cache to CPU memory, which alleviates GPU memory pressure but shifts the bottleneck to the limited bandwidth of the PCIe connection between the CPU and GPU. Existing methods attempt to address these issues by overlapping GPU computation with I/O or employing CPU-GPU heterogeneous execution, but they are hindered by excessive data movement and dependence on CPU capabilities. In this paper, we introduce an efficient CPU-GPU I/O-aware LLM inference method that avoids transferring the entire KV cache from CPU to GPU by recomputing partial KV cache from activations while concurrently transferring the remaining KV cache via PCIe bus. This approach overlaps GPU recomputation with data transfer to minimize idle GPU time and maximize inference performance. Our method is fully automated by integrating a profiler module that utilizes input characteristics and system hardware information, a scheduler module to optimize the distribution of computation and communication workloads, and a runtime module to efficiently execute the derived execution plan. Experimental results show that our method achieves up to 35.8% lower latency and 46.2% higher throughput during decoding compared to state-of-the-art approaches."},"title":{"value":"Efficient LLM Inference with I/O-Aware Partial KV Cache Recomputation"},"authors":{"value":["Chaoyi Jiang","Lei Gao","Hossein Entezari Zarch","Murali Annavaram"]}},"tmdate":1747420097071,"pdate":1704067200000,"tcdate":1739945787616,"writers":["~"],"signatures":["~Murali_Annavaram1"],"forum":"5ljSPZkvFS","license":"CC BY-SA 4.0","number":328043,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1747420097071,"domain":"DBLP.org","id":"5ljSPZkvFS","version":2},{"content":{"data_release":{"value":"We authorize the release of our submission and author names to the public in the event of acceptance."},"TLDR":{"value":"A method that reliably compresses KV cache to ~2-bit precision by keeping most channels at 2-bit TurboQuant while quantizing the 5–10% of outlier channels to 4-bit"},"venue":{"value":"SLM-Agents Poster"},"email_sharing":{"value":"We authorize the sharing of all author emails with the Program Chairs."},"keywords":{"value":["KV cache","TurboQuant","quantization","outlier-aware","mixed-precision","Small Language Models (SLMs)"]},"submission_requirements":{"value":["format","anonymity","authorship","originality","ethics_and_llm"]},"abstract":{"value":"Small Language Models (SLMs) are increasingly considered for efficient deployment, including agentic applications that require long-context processing. However, the growing Key-Value (KV) cache remains a major memory bottleneck. While vector quantization methods such as TurboQuant (TQ) achieve near-optimal distortion rates, reducing KV cache to uniform 2-bit precision can substantially degrade accuracy due to outlier activations. We propose OAMP-TQ, an outlier-aware mixed-precision TurboQuant method for aggressive KV cache compression, evaluated on LLaMA, Mistral, and Ministral 7–8B instruction-tuned models. We introduce two strategies: (1) One-Level (1L) OAMP-TQ, which quantizes inlier channels to 2-bit while preserving 5–10\\% of outlier channels in FP16; and Two-Level (2L) OAMP-TQ, a fully mixed-integer scheme applying 2-bit TurboQuant to inliers and 4-, 6-, or 8-bit TurboQuant to outliers. We evaluate both against unquantized KV cache and 4-bit TurboQuant on Needle-in-a-Haystack (NIAH) and LongBench (LB). 1L OAMP-TQ performs comparably to 4-bit TQ on LB but provides only 15\\% memory reduction and reduces NIAH recall from 98.8\\% to 88.6\\%. In contrast, 2L OAMP-TQ with 2-bit inliers and 4-bit outliers reduces KV cache memory by 45\\% over the 4-bit baseline with comparable quantization time while maintaining strong NIAH accuracy. These results show that outlier-aware mixed-precision quantization enables aggressive KV cache compression while preserving long-context accuracy, supporting memory-efficient SLM deployment in long-context and agentic settings."},"_bibtex":{"value":"@inproceedings{\nalomairy2026oamptq,\ntitle={{OAMP}-{TQ}: Ultra-Low-Bit TurboQuant {KV} Cache Compression for Small Language Models},\nauthor={Rabab Alomairy and Deema Alresheed and Lina Ziad Alyahya and Shahad Saad Alfawzan and Shaden Mohammed AlSahli and Waad Alahmed},\nbooktitle={SLM-Agents: 1st NeurIPS Workshop on SLMs for Agentic Systems},\nyear={2026},\nurl={https://openreview.net/forum?id=UR1jhmTNWF}\n}"},"title":{"value":"OAMP-TQ: Ultra-Low-Bit TurboQuant KV Cache Compression for Small Language Models"},"paper_type":{"value":"Long paper — up to 6 content pages"},"pdf":{"value":"/pdf/c06f909c52f93548cc49a1f0353ad18b26c05d79.pdf"},"venueid":{"value":"NeurIPS.cc/2026/Workshop/SLM-Agents"},"paperhash":{"value":"alomairy|oamptq_ultralowbit_turboquant_kv_cache_compression_for_small_language_models"},"primary_subject_area":{"value":"Small and efficient language models"},"publication_policy":{"value":"We understand that the workshop is non-archival and authorize public release of the accepted submission, its metadata, and author names."},"authors":{"value":[{"username":"~Rabab_Alomairy1","fullname":"Rabab Alomairy","institutions":[{"name":"Massachusetts Institute of Technology","domain":"mit.edu","country":"US"}]},{"username":"~Deema_Alresheed1","fullname":"Deema Alresheed","institutions":[{"name":"Saudi Technology and Security Comprehensive Control Company","domain":"tahakom.com","country":"SA"}]},{"username":"~Lina_Ziad_Alyahya2","fullname":"Lina Ziad Alyahya","institutions":[{"name":"Saudi Technology and Security Comprehensive Control Company","domain":"tahakom.com","country":"SA"}]},{"username":"~Shahad_Saad_Alfawzan1","fullname":"Shahad Saad Alfawzan","institutions":[{"name":"Saudi Technology and Security Comprehensive Control Company","domain":"tahakom.com","country":"SA"}]},{"username":"~Shaden_Mohammed_AlSahli1","fullname":"Shaden Mohammed AlSahli","institutions":[{"name":"Saudi Technology and Security Comprehensive Control Company","domain":"tahakom.com","country":"SA"}]},{"username":"~Waad_Alahmed1","fullname":"Waad Alahmed","institutions":[{"name":"Saudi Technology and Security Comprehensive Control Company","domain":"tahakom.com","country":"SA"}]}]}},"tmdate":1791135662430,"pdate":1791132907674,"tcdate":1787935721439,"writers":["NeurIPS.cc/2026/Workshop/SLM-Agents","NeurIPS.cc/2026/Workshop/SLM-Agents/Submission35/Authors"],"signatures":["NeurIPS.cc/2026/Workshop/SLM-Agents/Submission35/Authors"],"forum":"UR1jhmTNWF","license":"CC BY 4.0","number":35,"cdate":1787935721439,"readers":["everyone"],"invitations":["NeurIPS.cc/2026/Workshop/SLM-Agents/-/Submission","NeurIPS.cc/2026/Workshop/SLM-Agents/-/Submission_Change_Before_Bidding","NeurIPS.cc/2026/Workshop/SLM-Agents/-/Submission_Change_Before_Reviewing","NeurIPS.cc/2026/Workshop/SLM-Agents/-/Submission_Release","NeurIPS.cc/2026/Workshop/SLM-Agents/Submission35/-/Camera_Ready_Revision","NeurIPS.cc/2026/Workshop/SLM-Agents/-/Edit"],"mdate":1791135662430,"odate":1791132907674,"domain":"NeurIPS.cc/2026/Workshop/SLM-Agents","id":"UR1jhmTNWF","version":2},{"content":{"venue":{"value":"ICML 2026 regular"},"keywords":{"value":["ML Efficiency","Efficient Decoding","Lossless Compression","KV Cache"]},"_bibtex":{"value":"@inproceedings{\nle2026fafo,\ntitle={{FAFO}: Lossy {KV} Cache Compression for Lossless Inference Acceleration via Draftless Fumble Decoding},\nauthor={Hoang Anh Duy Le and Shaochen Zhong and Yifan Lu and Yingtong Dou and Jiayi Yuan and Yu-Neng Chuang and Xiran Fan and Guanchu Wang and Yuzhong Chen and Xia Hu},\nbooktitle={Forty-third International Conference on Machine Learning},\nyear={2026},\nurl={https://openreview.net/forum?id=kz7n1D8Y1o}\n}"},"title":{"value":"FAFO: Lossy KV Cache Compression for Lossless Inference Acceleration via Draftless Fumble Decoding"},"paperhash":{"value":"le|fafo_lossy_kv_cache_compression_for_lossless_inference_acceleration_via_draftless_fumble_decoding"},"originally_submitted_PDF":{"value":"/pdf/732d3e915752a9aa40c56b8d842604956e5431a1.pdf"},"primary_area":{"value":"deep_learning->everything_else"},"abstract":{"value":"*Lossy KV cache compression* is a well-explored subfield of machine learning efficiency, with improved latency being one of its major gains. However, lossy compression techniques can fumble from time to time, exhibiting various, and often catastrophic, failure patterns that are not only difficult to resolve but sometimes even hard to identify, making direct deployment of models with compressed KV cache a risky endeavor. In this work, we explore a way to preserve *lossless* generation quality while still benefiting from the acceleration provided by KV cache compression. Specifically, we draw inspiration from the *n-gram candidate pool decoding* paradigm where we purposely allow the model to Fumble Around with compressed KV cache to generate multiple lossy \"n-gram guesses\", while in parallel Find Out via lossless verification in the same forward pass. From a conceptual standpoint, our proposed framework is compatible with all typical static or dynamic KV cache compression methods from the token dropping realm, thus opening up a new avenue for the stagnant n-gram decoding paradigm. Practically, we show that this framework presents many useful traits that similar draftless baselines (e.g., Self-Speculative Decoding) cannot achieve, such as requiring only one set of KV cache and being far less sensitive to model, task, and input-length scenarios. Our comprehensive empirical results show FAFO provides 1.20-2.71× latency speedup over the original model, while consistently outperforming other lossless + draftless solutions."},"pdf":{"value":"/pdf/4019a9379c34c979be6579e8dda5f33ba40d1d7c.pdf"},"lay_summary":{"value":"Modern chatbots like ChatGPT remember every word of your conversation in a fast-access scratchpad that grows as you keep talking, eventually slowing the chatbot down. Researchers have tried shrinking this scratchpad by discarding parts of it, but doing so can subtly degrade the chatbot's answers in ways that are hard to detect. We propose Fumble Around and Find Out (FAFO): let the chatbot guess a few upcoming words using a shrunken, lower-quality scratchpad, then double-check each guess against the full, original scratchpad in a single pass — accepting correct guesses and discarding wrong ones. The result is up to 2.7× faster responses with no change to what the chatbot would have said otherwise. Unlike previous speed-up tricks, FAFO holds up reliably across many tasks, from writing assistance to long-document reasoning, making fast and trustworthy chatbot responses easier to deploy in practice."},"venueid":{"value":"ICML.cc/2026/Conference"},"authorids":{"value":["~Hoang_Anh_Duy_Le1","~Shaochen_Zhong1","~Yifan_Lu8","~Yingtong_Dou1","~Jiayi_Yuan1","~Yu-Neng_Chuang1","~Xiran_Fan1","~Guanchu_Wang1","~Yuzhong_Chen2","~Xia_Hu4"]},"authors":{"value":["Hoang Anh Duy Le","Shaochen Zhong","Yifan Lu","Yingtong Dou","Jiayi Yuan","Yu-Neng Chuang","Xiran Fan","Guanchu Wang","Yuzhong Chen","Xia Hu"]}},"tmdate":1790067383673,"pdate":1777576577105,"tcdate":1769131160304,"writers":["ICML.cc/2026/Conference","ICML.cc/2026/Conference/Submission19106/Authors"],"signatures":["ICML.cc/2026/Conference/Submission19106/Authors"],"forum":"kz7n1D8Y1o","license":"CC BY 4.0","number":19106,"cdate":1769131160304,"readers":["everyone"],"invitations":["ICML.cc/2026/Conference/-/Submission","ICML.cc/2026/Conference/-/Post_Submission","ICML.cc/2026/Conference/Submission19106/-/Full_Submission","ICML.cc/2026/Conference/-/Edit","ICML.cc/2026/Conference/Submission19106/-/Camera_Ready_Revision"],"mdate":1790067383673,"odate":1782341973570,"domain":"ICML.cc/2026/Conference","id":"kz7n1D8Y1o","version":2},{"content":{"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["Vision-Language-Action Models","KV Cache Reuse","Efficient Inference","Hardware-Aware Acceleration","Robotic Manipulation"]},"primary_area":{"value":"infrastructure, software libraries, hardware, systems, etc."},"abstract":{"value":"Vision-language-action (VLA) models require low-latency inference for responsive robotic control. Previous works have shown that cross-frame reuse of visual key-value (KV) representations can reduce redundant computation. However, reductions in FLOPs do not necessarily translate into proportional latency reductions. Profiling the prefill-stage of VLA language-backbone on GPUs reveals a pronounced latency-plateau effect in dominant linear operators: reducing the number of recomputed tokens within a plateau yields marginal speedup, whereas crossing a plateau boundary yields substantial latency reduction. Motivated by this observation, we propose LPA-CACHE, a training-free method for accelerating VLA inference on GPUs through latency-plateau-aware KV cache reuse. A one-time profiling procedure identifies latency-plateau boundaries on the target GPU platform. These boundaries are combined with preselected reuse layers to construct a fixed-shaped reuse schedule. At inference time, visual tokens estimated to be task-relevant from previous-step attentions are protected from reuse, while the remaining visual tokens are ranked by inter-frame relative changes and progressively reused according to layerwise reuse budgets. Guided by observed cross-layer consistency in attention-based task relevance, LPA-CACHE extracts attention weights from only a small subset of layers, allowing FlashAttention-2 to be used in the remaining layers for further speedup. Extensive experiments across several representative VLA models on two robotic simulation platforms (LIBERO and CALVIN) demonstrate that LPA-CACHE achieves up to 1.38$\\times$ language-backbone prefill speedup and up to 1.15$\\times$ observation-to-action inference speedup with negligible loss in task success rate."},"_bibtex":{"value":"@inproceedings{\nanonymous2026lpacache,\ntitle={{LPA}-{CACHE}: Latency-Plateau Aware {KV} Cache Reuse for Efficient Vision-Language-Action Model Inference on {GPU}s},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=mnLJ2CG0ub},\nnote={under review}\n}"},"title":{"value":"LPA-CACHE: Latency-Plateau Aware KV Cache Reuse for Efficient Vision-Language-Action Model Inference on GPUs"},"pdf":{"value":"/pdf/e9dc7a28914580886e6ed7a1059a449f5c5497a4.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791230267878,"tcdate":1789548546523,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission24302/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission24302/Authors"],"forum":"mnLJ2CG0ub","license":"CC BY 4.0","number":24302,"cdate":1789548546523,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission24302/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791230267878,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"mnLJ2CG0ub","version":2},{"content":{"summary":{"value":"This paper introduces KAVA, the first framework that distills knowledge directly from the compressed KV-cache of the teacher into a latent-reasoning student via self-distillation. This process is conducted in a step-wise fashion, removing the eviting KVs that do not are less important."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"- Why is there not a complete comparison between all the methods and models? Is this from a resource limitation? I would suggest moving those comparisons to another table.\n- How does the quantity or quality of the data affect the distillation? How can this method move beyond the limitations of current distillation? \n- How does the distillation process compare when distilled from its own CoT or a stronger model?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"- interesting idea to distill from the KV-cache\n- Overall, good comparisons with other baselines"},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- The biggest weakness of this paper is that the performance of the teacher caps any distillation process. \n- Lack of diversity in downstream tasks\n- Writing\n - Move Figure 1 earlier in the paper (i.e pg 1-2)\n - The main benefit of this method seems to be the reduction of the inference cost. However, this result/point is presented over 2 pages instead of just having a single graph/table showing the Pareto frontier curve"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762923400475,"tcdate":1761197637183,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission12538/Reviewer_MUtU"],"signatures":["ICLR.cc/2026/Conference/Submission12538/Reviewer_MUtU"],"forum":"ePrhcLbtGv","number":2,"license":"CC BY 4.0","cdate":1761197637183,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission12538/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762923400475,"domain":"ICLR.cc/2026/Conference","replyto":"ePrhcLbtGv","id":"PhZ7SbdhZ8","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"TLDR":{"value":"We introduce a latent reasoning method guided by distillation from compressed kv-cache."},"keywords":{"value":["llm","reasoning","latent reasoning","efficiency"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Large Language Models (LLMs) excel at multi-step reasoning problems with explicit chain-of-thought (CoT), but verbose traces incur significant computational costs and memory overhead, and often carry redundant, stylistic artifacts. Latent reasoning has emerged as an efficient alternative that internalizes the thought process, but it suffers from a critical lack of supervision, limiting its effectiveness on complex, natural-language reasoning traces.\nIn this work we propose KaVa, the first framework that bridges this gap by distilling knowledge directly from a compressed KV-cache of the teacher into a latent-reasoning student via self-distillation, leveraging the representational flexibility of continuous latent tokens to align stepwise KV trajectories. We show that the abstract, unstructured knowledge within compressed KV-cache, which lacks direct token correspondence, can serve as a rich supervisory signal for a latent reasoning student. \nEmpirically, the approach consistently outperforms strong latent baselines, exhibits markedly smaller degradation from equation-only to natural-language traces, and scales to larger backbones while preserving efficiency. These results establish compressed KV-cache distillation as a scalable supervision signal for latent reasoning, combining the accuracy of CoT-trained teachers with the efficiency and deployability of latent inference."},"_bibtex":{"value":"@inproceedings{\nkuzina2026kava,\ntitle={KaVa: Latent Reasoning via Compressed {KV}-Cache Distillation},\nauthor={Anna Kuzina and Maciej Pi{\\'o}ro and Babak Ehteshami Bejnordi},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=ePrhcLbtGv}\n}"},"title":{"value":"KaVa: Latent Reasoning via Compressed KV-Cache Distillation"},"pdf":{"value":"/pdf/95777660f1bc7a3933327d8196c548e2164e809d.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"kuzina|kava_latent_reasoning_via_compressed_kvcache_distillation"},"authorids":{"value":["~Anna_Kuzina1","~Maciej_Pióro1","~Babak_Ehteshami_Bejnordi1"]},"authors":{"value":["Anna Kuzina","Maciej Pióro","Babak Ehteshami Bejnordi"]}},"version":2},{"content":{"summary":{"value":"This paper introduces IceCache, a novel KV-cache management strategy designed to mitigate memory bottlenecks for LLMs processing long sequences. The core idea is to move beyond storing the KV-cache in its original token order. Instead, IceCache clusters tokens based on the semantic similarity of their key embeddings into a hierarchical structure called a DCI-tree. During inference, it employs a fast Approximate Nearest Neighbor (ANN) search algorithm (M-DCI) on the CPU to perform a query-aware, head-specific selection of the most relevant memory pages. These selected pages are then transferred from CPU to GPU for attention computation. The system is optimized with a pipelined workflow that overlaps the CPU-based page selection with GPU computations to hide latency. Experimental results across several benchmarks and models show that IceCache can maintain high accuracy (over 99% in some cases) with a significantly reduced token budget compared to the full KV-cache and outperforms other baseline methods."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"1. Could you provide performance curves showing how the inference speed (e.g., tokens per second) scales with increasing context length?\n\n2. The ANN search is currently performed on the CPU to overlap with GPU computations. Have you considered the feasibility of further accelerating the M-DCI query step by implementing it on the GPU?\n\n3. The proposed method seems to be tightly integrated with a CPU offloading strategy. Is it possible to adapt IceCache for a non-offload scenario (i.e., where the full KV-cache fits in GPU memory, but sparsity is desired to accelerate computation)? Or does the method fundamentally rely on the CPU having access to the cache for the DCI-tree query to work?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"- The authors compare IceCache against a strong and sufficient set of six recent state-of-the-art KV-cache optimization methods.\n\n- The paper is well-organized and clearly written. The motivation is well-established, the proposed method is described logically with helpful diagrams (e.g., Figure 1 and 2).\n\n- The application of a hierarchical ANN algorithm (M-DCI) to cluster and retrieve from the high-dimensional key-embedding space of the KV-cache is a novel and interesting approach."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- The paper does not provide an analysis of how different parameter choices of M-DCI affect the final model accuracy and inference latency. Furthermore, the specific parameter values used in the experiments are not clearly described, which could hinder reproducibility.\n\n- The latency analysis in Section 5.5 focuses on \"Time to the second token\" (TT2T) and \"Time per output token\" (TPOT). While informative, it lacks a direct end-to-end runtime comparison against full attention.\n\n- A central claim is that grouping semantically similar tokens into the same memory pages improves efficiency. However, there is no ablation study to isolate and quantify this specific contribution. Since the index selection happens on the CPU, a key question arises: does this memory layout truly enhance efficiency, or is the performance bottleneck dominated by the CPU search and the CPU-GPU data transfer overhead?"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762915543094,"tcdate":1761034205497,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission541/Reviewer_815Q"],"signatures":["ICLR.cc/2026/Conference/Submission541/Reviewer_815Q"],"forum":"yHxSKM9kdr","number":1,"license":"CC BY 4.0","cdate":1761034205497,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission541/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762915543094,"domain":"ICLR.cc/2026/Conference","replyto":"yHxSKM9kdr","id":"lT7jYsZdiw","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["LLM Inference; KV-cahce Optimization; Sparse Attention"]},"primary_area":{"value":"generative models"},"abstract":{"value":"Key-Value (KV) cache plays a crucial role in accelerating inference in large language models (LLMs) by storing intermediate attention states and avoiding redundant computation during autoregressive generation. However, its memory footprint scales linearly with sequence length, often leading to severe memory bottlenecks on resource-constrained hardware. Prior work has explored offloading KV-cache to the CPU while retaining only a subset on the GPU, but these approaches often rely on imprecise token selection and suffer performance degradation in long-generation tasks such as chain-of-thought reasoning. In this paper, we propose a novel KV-cache management strategy, IceCache, which integrates semantic token clustering with PagedAttention. By organizing semantically related tokens into contiguous memory regions managed by a hierarchical, dynamically updatable data structure, our method enables more efficient token selection and better utilization of memory bandwidth during CPU–GPU transfers. Experimental results on LongBench show that, with a 256-token budget, IceCache maintains 99\\% of the original accuracy achieved by the full KV-cache model. Moreover, compared to other offloading-based methods, IceCache attains competitive or even superior latency and accuracy while using only 25\\% of the KV-cache token budget, demonstrating its effectiveness in long-sequence scenarios. The code is available on our project website at https://yuzhenmao.github.io/IceCache/."},"_bibtex":{"value":"@inproceedings{\nmao2026icecache,\ntitle={IceCache: Memory-Efficient {KV}-cache Management for Long-Sequence {LLM}s},\nauthor={Yuzhen Mao and Qitong Wang and Martin Ester and Ke Li},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=yHxSKM9kdr}\n}"},"title":{"value":"IceCache: Memory-Efficient KV-cache Management for Long-Sequence LLMs"},"pdf":{"value":"/pdf/6b6c4116918b8bfb7e9d52c80b1c8c9f731ecce2.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"mao|icecache_memoryefficient_kvcache_management_for_longsequence_llms"},"authorids":{"value":["~Yuzhen_Mao2","~Qitong_Wang5","~Martin_Ester1","~Ke_Li1"]},"authors":{"value":["Yuzhen Mao","Qitong Wang","Martin Ester","Ke Li"]}},"version":2},{"content":{"summary":{"value":"The authors conducted systematic experiments to analyze the similarity of KV cache selection among queries, heads, and layers. Capitalizing on these findings, they proposed a method that shares KV selection across the dimensions of query, head, and layer, thereby reducing the overhead of KV selection."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"Please refer to weakness points 1-4"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"The sharing-based method proposed in this paper seems novel."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1.\tThe experiments only selected 6 out of the 16 commonly used English datasets from Longbench for evaluation, which may not be sufficient to draw solid conclusions. For example, the performance of H2O and Quest on these 6 datasets in Table 3 appears similar, whereas in the Quest paper, H2O showed significantly lower accuracy. This discrepancy may suggest that specific methods perform better on these six datasets. To clarify, the authors are encouraged to provide results across all datasets.\n\n2.\tIn terms of speedup, the authors analyzed the theoretical complexity of KV cache selection in Table 2. However, this analysis alone is insufficient to demonstrate that the proposed method significantly reduces computational overhead, as the selection cost may only account for a small portion of the entired decoding process. To strengthen this claim, the authors could provide direct evidence of decoding speed in experiments. Additionally, an experimental study on the trade-off between accuracy and speed would be valuable.\n\n3.\tThe authors provided only a speed comparison with FlashAttention. To better demonstrate the effectiveness of their method, comparisons with other query-aware dynamic token sparsity methods are necessary. \n\n4.\tThe authors mention using an approximate computation approach from Double Sparse ('Following the approach in Yang et al. (2024)') to calculate approximate attention weights in their implementation. As a result, it is unclear how much of the observed speed gain is attributable to HShare specifically versus the query-aware dynamic token sparsity method itself."}},"nonreaders":[],"tmdate":1733217542140,"tcdate":1730626424266,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission2383/Reviewer_ZRXS"],"signatures":["ICLR.cc/2025/Conference/Submission2383/Reviewer_ZRXS"],"forum":"Tb5PY5vwp6","number":4,"license":"CC BY 4.0","cdate":1730626424266,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission2383/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1733217542140,"domain":"ICLR.cc/2025/Conference","replyto":"Tb5PY5vwp6","id":"OiksINEMkR","forumContent":{"venue":{"value":"ICLR 2025 Poster"},"TLDR":{"value":"We propose a hierarchical critical KV cache indices sharing framework across layer, head, and query levels."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Large Language Model","Decode","Key-Value Sharing","Critical Token","Hierarchical"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The frequent retrieval of Key-Value (KV) cache data has emerged as a significant factor contributing to the inefficiency of the inference process in large language models. Previous research has demonstrated that a small subset of critical KV cache tokens largely influences attention outcomes, leading to methods that either employ fixed sparsity patterns or dynamically select critical tokens based on the query. While dynamic sparse patterns have proven to be more effective, they introduce significant computational overhead, as critical tokens must be reselected for each self-attention computation. In this paper, we reveal substantial similarities in KV cache token criticality across neighboring queries, layers, and heads. Motivated by this insight, we propose HShare, a hierarchical KV sharing framework. HShare facilitates the sharing of critical KV cache token indices across layers, heads, and queries, which significantly reduces the computational overhead associated with query-aware dynamic token sparsity. In addition, we introduce a greedy algorithm that dynamically determines the optimal layer-level and head-level sharing configuration for the decoding phase. We evaluate the effectiveness and efficiency of HShare across various tasks using three models: LLaMA2-7b, LLaMA3-70b, and Mistral-7b. Experimental results demonstrate that HShare achieves competitive accuracy with different sharing ratios, while delivering up to an $8.6\\times$ speedup in self-attention operations and a $2.7\\times$ improvement in end-to-end throughput compared with FlashAttention2 and GPT-fast respectively. The source code is publicly available at ~\\url{https://github.com/wuhuaijin/HShare}."},"_bibtex":{"value":"@inproceedings{\nwu2025hshare,\ntitle={{HS}hare: Fast {LLM} Decoding by Hierarchical Key-Value Sharing},\nauthor={Huaijin Wu and Lianqiang Li and Hantao Huang and Tu Yi and Jihang Zhang and Minghui Yu and Junchi Yan},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=Tb5PY5vwp6}\n}"},"title":{"value":"HShare: Fast LLM Decoding by Hierarchical Key-Value Sharing"},"pdf":{"value":"/pdf/fda8db959bbdef676e6fc30331fb96055a1f725f.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"wu|hshare_fast_llm_decoding_by_hierarchical_keyvalue_sharing"},"authorids":{"value":["~Huaijin_Wu1","~Lianqiang_Li1","~Hantao_Huang1","~Tu_Yi1","~Jihang_Zhang1","~Minghui_Yu3","~Junchi_Yan2"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Huaijin Wu","Lianqiang Li","Hantao Huang","Tu Yi","Jihang Zhang","Minghui Yu","Junchi Yan"]}},"version":2},{"content":{"research_area_keywords":{"value":"pruning; quantization"},"languages_studied":{"value":"English"},"venue":{"value":"ACL ARR 2024 December Submission"},"_bibtex":{"value":"@inproceedings{\nanonymous2025more,\ntitle={More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in {KV} Cache Compression},\nauthor={Anonymous},\nbooktitle={Submitted to ACL Rolling Review - December 2024},\nyear={2025},\nurl={https://openreview.net/forum?id=eq3ukehXg5},\nnote={under review}\n}"},"title":{"value":"More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression"},"contribution_types":{"value":["Approaches to low-resource settings","Approaches low compute settings-efficiency","Data analysis"]},"abstract":{"value":"As large language models (LLMs) process increasing context windows, the memory usage of KV cache has become a critical bottleneck during inference. \nThe mainstream KV compression methods, including KV pruning and KV quantization, primarily focus on either token or precision dimension and seldom explore the efficiency of their combination. \nIn this paper, we comprehensively investigate the token-precision trade-off in KV cache compression. \nExperiments demonstrate that storing more tokens \nin the KV cache with lower precision, \ni.e., quantized pruning, can significantly enhance the long-context performance of LLMs. \nFurthermore, in-depth analysis regarding token-precision trade-off from a series of key aspects exhibit that, quantized pruning achieves substantial improvements in retrieval-related tasks and consistently performs well across varying input lengths. Moreover, quantized pruning demonstrates notable stability across different KV pruning methods, quantization strategies, and model scales. These findings provide valuable insights into the token-precision trade-off in KV cache compression. We plan to release our code in the near future."},"paper_type":{"value":"Long"},"pdf":{"value":"/pdf/231729887738b5253cf4d02b50068b4163ff6463.pdf"},"research_area":{"value":"Efficient/Low-Resource Methods for NLP"},"venueid":{"value":"aclweb.org/ACL/ARR/2024/December/Submission"}},"tmdate":1782962159870,"tcdate":1734346105481,"writers":["aclweb.org/ACL/ARR/2024/December","aclweb.org/ACL/ARR/2024/December/Submission2027/Authors"],"signatures":["aclweb.org/ACL/ARR/2024/December/Submission2027/Authors"],"forum":"eq3ukehXg5","license":"CC BY 4.0","number":2027,"cdate":1734346105481,"readers":["everyone"],"invitations":["aclweb.org/ACL/ARR/2024/December/-/Submission","aclweb.org/ACL/ARR/2024/December/-/Edit","aclweb.org/ACL/ARR/2024/December/-/Post_Submission","aclweb.org/ACL/ARR/2024/December/-/Preprint_Release_Submission","aclweb.org/ACL/ARR/2024/December/-/Desk_Reject_Verification_Flag","aclweb.org/ACL/ARR/2024/December/-/Preprint_Post_Submission"],"mdate":1782962159870,"odate":1738780930893,"domain":"aclweb.org/ACL/ARR/2024/December","id":"eq3ukehXg5","version":2},{"content":{"summary":{"value":"This paper introduces PRKV, a framework for page-level KV retrieval that combines static reuse and dynamic selection through a “hybrid KV selection” strategy. The authors also propose system optimizations, namely contiguous memory indexing using the HND layout and batched KV transfers to improve retrieval efficiency. PRKV reports up to 6.75× speedup and near-full-cache accuracy across multiple benchmarks."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"1. Please justify how the HND layout efficiently supports GPU attention kernels. Most state-of-the-art implementations (e.g., FlashAttention, FlashInfer) assume NHD layout for coalesced tensor access. Has PRKV been benchmarked with real kernel implementations to confirm compatibility?\n2. The proposed “static token selection” is periodically updated. Does this imply that the entire KV cache must still be retained in CPU memory to enable re-selection? If so, how does this affect the memory footprint compared to fully static methods such as SnapKV?"},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":1},"strengths":{"value":"1. Clarity and completeness:\nThe paper is reasonably well written and provides a comprehensive experimental evaluation across several standard long-context benchmarks.\n2. Incremental algorithmic variation:\nThe proposed periodic static token update introduces a minor variation to existing retrieval schemes. While not particularly innovative, it reflects an effort to stabilize long-sequence performance by refreshing the static token set."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Limited algorithmic novelty:\nThe proposed method largely concatenates two existing paradigms of KV-cache compression, static dropping and dynamic retrieval, without introducing a fundamentally new principle. As a result, the algorithmic contribution is incremental and lacks clear conceptual advancement.\n2. Questionable system design and justification:\nThe claimed system optimizations are not well substantiated. Although the HND layout may simplify head-wise indexing, it is unclear how this layout integrates with modern GPU attention kernels, which typically assume NHD for coalesced access and contiguous memory traversal (as used in FlashAttention and vLLM). Similarly, the batched KV transfer optimization resembles standard engineering practice rather than a substantive research contribution."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1764360265408,"tcdate":1761930230397,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission3974/Reviewer_k1cZ"],"signatures":["ICLR.cc/2026/Conference/Submission3974/Reviewer_k1cZ"],"forum":"7FM0GBFhe5","number":2,"license":"CC BY 4.0","cdate":1761930230397,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission3974/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1764360265408,"domain":"ICLR.cc/2026/Conference","replyto":"7FM0GBFhe5","id":"iZluMKod7P","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"TLDR":{"value":"Hybrid KV selection for achieving high accuracy"},"keywords":{"value":["Long-Context LLM Inference","KV Cache Optimization"]},"supplementary_material":{"value":"/attachment/4a20777460d75c8c874f1c4e30433e886a9f0250.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"As the key-value(KV) cache size scales with context length, accessing large KV\ncache each step and substantial GPU memory demand challenge us to deploy\nLLMs with long contexts.Various sparse attention methods have been proposed\nand offloading-based KV retrieval preserves entire KV cache in CPU memory\nand dynamically retrieves most relevant KV pairs for each decoding step, which\nperforms higher quality and effectively reduces GPU memory consumption than\nother line works. However, exiting KV retrieval performs page-level to reduce\nestimation overhead, which introduces inaccurate KV selection and significant re-\ntrieval overhead. We propose PRKV, a framework that both-optimizes algorithm\nand system for page-level KV retrieval with KV offloading. On the algorithm side,\nPRKV introduces hybrid KV selection that combines both static and dynamic KV\nselection strategies. On the system side, PRKV employs contiguous memory in-\ndexing and batched transfer optimizations to improve retrieval efficiency. Exper-\niments demonstrate that PRKV improve accuracy across various scenarios and\nmodels, delivering up to 6.75× speedup compared to SOTA KV retrieval methods."},"_bibtex":{"value":"@misc{\nwu2026prkvpage,\ntitle={{PRKV}:Page Restruct {KV} Cache for High Accuracy and Efficiency {LLM} Generation},\nauthor={Fang Wu and Congming Gao and Weixi Zhu and Jiwu Shu},\nyear={2026},\nurl={https://openreview.net/forum?id=7FM0GBFhe5}\n}"},"title":{"value":"PRKV:Page Restruct KV Cache for High Accuracy and Efficiency LLM Generation"},"pdf":{"value":"/pdf/2e35227c8bc4c0ff32cebbc4cc89ada6aff48924.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"wu|prkvpage_restruct_kv_cache_for_high_accuracy_and_efficiency_llm_generation"},"authorids":{"value":["~Fang_Wu3","~Congming_Gao1","~Weixi_Zhu2","~Jiwu_Shu1"]},"authors":{"value":["Fang Wu","Congming Gao","Weixi Zhu","Jiwu Shu"]}},"version":2},{"content":{"summary":{"value":"This paper presents RACC, a hybrid KV cache framework that addresses the accuracy-latency trade-off in long-context inference by combining GPU-side compression with CPU-side retrieval. Its core innovation is an asynchronous retrieval mechanism that uses previously generated tokens as queries, leveraging their attention similarity to current tokens to run retrieval in parallel with GPU decoding and avoid I/O bottlenecks. RACC achieves near-lossless accuracy using only 15% of the original KV cache, demonstrating superior performance over existing compression-only and retrieval-only methods."},"soundness":{"value":4},"confidence":{"value":3},"questions":{"value":"1. Query Lag Sensitivity: What is the exact lag used in the experiments between the query token $q$ and the token $t$ currently being generated (e.g., is it $q_{t-1}$ or $q_{t-n}$)? How sensitive is RACC's performance (both accuracy and latency) to this \"query staleness\" or lag?\n\n2. Dynamic Index Bottleneck: In LILO scenarios , the CPU index must grow continuously as tokens are evicted. At what point (e.g., after how many generated tokens) does the CPU-side MIPS search and/or incremental index updating become the new performance bottleneck, potentially negating the latency gains from asynchronous execution?\n\n3. Grouping Compression Parameters: The paper introduces a \"grouping compression strategy\" (compressing every n new tokens) to reduce computational overhead. What value of n was used in the experiments? How was the trade-off between reducing compression frequency (a larger n) and ensuring the GPU cache remains sufficiently relevant (a smaller n) determined?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. The core innovation lies in using query vectors from previously generated tokens to conduct asynchronous CPU-side retrieval. This design adeptly circumvents the severe latency bottleneck caused by synchronous I/O, which plagues traditional retrieval-based methods.\n\n2. RACC successfully breaks the longstanding compromise between inference efficiency and generation accuracy. It retains the high efficiency and low memory footprint of compression on the GPU while using asynchronous retrieval to \"recall\" crucial evicted information, thus maintaining high fidelity.\n\n3. The experimental results are compelling. RACC delivers near-lossless performance with only 15% of the KV cache. Under the same compression ratio, it significantly outperforms SOTA baselines like SqueezeAttn, Scope, and CakeKV in generation accuracy on benchmarks like LongGenBench and LongProc."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. High Dependence on the Core Observation: The method's efficacy hinges entirely on the observation that \"adjacent tokens share similar attention patterns\". For tasks requiring rapid, sharp context shifts or the precise retrieval of a single, non-redundant fact from the distant past, using a stale query ($q_{t-n}$) may fail to retrieve the exact KV pairs needed by the current token ($q_t$)\n\n2. Increased System Complexity: The framework introduces non-trivial system complexity. It requires maintaining a persistent vector database on the CPU and managing sophisticated, error-free asynchronous communication and data transfer between the CPU and GPU.\n\n3. Unaddressed Dynamic Indexing Costs: For SILO and LILO scenarios, the paper states the CPU-side vector index is \"incrementally updated\". However, the paper does not analyze the computational overhead of these dynamic index updates or the potential for the MIPS search itself to become a new bottleneck as the index grows to manage millions of evicted tokens."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762925810198,"tcdate":1761944601502,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission15530/Reviewer_GWHc"],"signatures":["ICLR.cc/2026/Conference/Submission15530/Reviewer_GWHc"],"forum":"F7kDkYjBVa","number":4,"license":"CC BY 4.0","cdate":1761944601502,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission15530/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762925810198,"domain":"ICLR.cc/2026/Conference","replyto":"F7kDkYjBVa","id":"dUUUpfUrqH","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Large Language Models","Efficient Inference","KV Cache Compression","Vector Retrieval"]},"primary_area":{"value":"generative models"},"abstract":{"value":"Large Language Models (LLMs) have achieved remarkable progress in long-context generation. As the context length increases, the Key--Value (KV) cache requires the GPU memory with a linear growth rate. KV cache compression is treated as a promising method to reduce the memory usage by permanently discarding a large portion of unimportant KV pairs, but at the expense of inference accuracy. On the other hand, retrieval-based methods employ the CPU memory to store the full KV cache and compute the attention via expensive CPU-GPU I/O, which keeps the accuracy but suffers from huge inference latency. To address these issues, we propose a new inference framework called RACC, which combines both compression based methods and retrieval based methods. To be specific, we employ the KV cache compression method to maintain a high-quality KV cache in the GPU memory, while sotring all the KV pairs evicted by the compression method. In addition, efficient and accurate retrieval conducted on the CPU side finds out important tokens for the one being generated, which is then concatenated with those KV cached in the GPU memory for accurate generation. Extensive experiments demonstrate that RACC achieves near-lossless inference while using only 15\\% of the original KV cache. Moreover, its combination with prefill-only compression methods improves generation accuracy by 3--10\\%. Our code is publicly available at \\url{https://anonymous.4open.science/r/CDKEY/}."},"_bibtex":{"value":"@misc{\nyang2026racc,\ntitle={{RACC}: Retrieval-Augmented {KV} Cache Compression in Long-Context Generation},\nauthor={Zhenfei Yang and Yingfan Liu and Tong Wu and Jiasheng Zhang and Jiangtao Cui},\nyear={2026},\nurl={https://openreview.net/forum?id=F7kDkYjBVa}\n}"},"title":{"value":"RACC: Retrieval-Augmented KV Cache Compression in Long-Context Generation"},"pdf":{"value":"/pdf/2f49131473f0e32a632b70b57e15541bf11414d0.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"yang|racc_retrievalaugmented_kv_cache_compression_in_longcontext_generation"},"authorids":{"value":["~Zhenfei_Yang3","~Yingfan_Liu2","~Tong_Wu40","~Jiasheng_Zhang2","~Jiangtao_Cui1"]},"authors":{"value":["Zhenfei Yang","Yingfan Liu","Tong Wu","Jiasheng Zhang","Jiangtao Cui"]}},"version":2},{"content":{"venue":{"value":"AAAI 2026"},"pdf":{"value":"https://ojs.aaai.org/index.php/AAAI/article/download/37467/41429"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"jiang|acckv_towards_efficient_audiovideo_llms_inference_via_adaptivefocusing_and_crosscalibration_kv_cache_optimization"},"html":{"value":"https://doi.org/10.1609/aaai.v40i7.37467"},"_bibtex":{"value":"@inproceedings{DBLP:conf/aaai/JiangCLYZWLZ26,\n author={Zhonghua Jiang and Kui Chen and Kunxi Li and Keting Yin and Yiyun Zhou and Zhaode Wang and Chengfei Lv and Shengyu Zhang},\n title={AccKV: Towards Efficient Audio-Video LLMs Inference via Adaptive-Focusing and Cross-Calibration KV Cache Optimization},\n year={2026},\n cdate={1767225600000},\n pages={5494-5502},\n url={https://doi.org/10.1609/aaai.v40i7.37467},\n booktitle={AAAI},\n crossref={conf/aaai/2026}\n}\n"},"abstract":{"value":"Recent advancements in Audio-Video Large Language Models (AV-LLMs) have enhanced their capabilities in tasks like audio-visual question answering and multimodal dialog systems. Video and audio introduce an extended temporal dimension, resulting in a larger key-value (KV) cache compared to static image embedding. A naive optimization strategy is to selectively focus on and retain KV caches of audio or video based on task. However, in the experiment, we observed that the attention of AV-LLMs to various modalities in the high layers is not strictly dependent on the task. In higher layers, the attention of AV-LLMs shifts more towards the video modality. In addition, we also found that directly integrating temporal KV of audio and spatial-temporal KV of video may lead to information confusion and significant performance degradation of AV-LLMs. If audio and video are processed indiscriminately, it may also lead to excessive compression or reservation of a certain modality, thereby disrupting the alignment between modalities. To address these challenges, we propose AccKV, an Adaptive-Focusing and Cross-Calibration KV cache optimization framework designed specifically for efficient AV-LLMs inference. Our method is based on layer adaptive focusing technology, selectively focusing on key modalities according to the characteristics of different layers, and enhances the recognition of heavy hitter tokens through attention redistribution. In addition, we propose a Cross-Calibration technique that first integrates inefficient KV caches within the audio and video modalities, and then aligns low-priority modality with high-priority modality to selectively evict KV cache of low-priority modality. The experimental results show that AccKV can significantly improve the computational efficiency of AV-LLMs while maintaining accuracy."},"title":{"value":"AccKV: Towards Efficient Audio-Video LLMs Inference via Adaptive-Focusing and Cross-Calibration KV Cache Optimization"},"authors":{"value":[{"fullname":"Zhonghua Jiang","username":""},{"fullname":"Kui Chen","username":""},{"fullname":"Kunxi Li","username":""},{"fullname":"Keting Yin","username":""},{"fullname":"Yiyun Zhou","username":"~Yiyun_Zhou1"},{"fullname":"Zhaode Wang","username":""},{"fullname":"Chengfei Lv","username":""},{"fullname":"Shengyu Zhang","username":""}]}},"tmdate":1780119638411,"pdate":1798675200000,"externalIds":["dblp:conf/aaai/JiangCLYZWLZ26"],"tcdate":1780119634745,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Yiyun_Zhou1"],"forum":"y50nYbHBMk","license":"CC BY-SA 4.0","number":38539,"cdate":1767225600000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1780119638411,"domain":"OpenReview.net/Public_Article","id":"y50nYbHBMk","version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2511.11106v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"jiang|acckv_towards_efficient_audiovideo_llms_inference_via_adaptivefocusing_and_crosscalibration_kv_cache_optimization"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Zhonghua_Jiang_0007:","https://dblp.org/search/pid/api?q=author:Kui_Chen:","https://dblp.org/search/pid/api?q=author:Kunxi_Li:","https://dblp.org/search/pid/api?q=author:Keting_Yin:","~Yiyun_Zhou1","https://dblp.org/search/pid/api?q=author:Zhaode_Wang:","https://dblp.org/search/pid/api?q=author:Chengfei_Lv:","~Shengyu_Zhang2"]},"html":{"value":"https://doi.org/10.48550/arXiv.2511.11106"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2511-11106,\n publtype={informal},\n author={Zhonghua Jiang and Kui Chen and Kunxi Li and Keting Yin and Yiyun Zhou and Zhaode Wang and Chengfei Lv and Shengyu Zhang},\n title={AccKV: Towards Efficient Audio-Video LLMs Inference via Adaptive-Focusing and Cross-Calibration KV Cache Optimization},\n year={2025},\n month={November},\n cdate={1761955200000},\n journal={CoRR},\n volume={abs/2511.11106},\n url={https://doi.org/10.48550/arXiv.2511.11106}\n}\n"},"abstract":{"value":"Recent advancements in Audio-Video Large Language Models (AV-LLMs) have enhanced their capabilities in tasks like audio-visual question answering and multimodal dialog systems. Video and audio introduce an extended temporal dimension, resulting in a larger key-value (KV) cache compared to static image embedding. A naive optimization strategy is to selectively focus on and retain KV caches of audio or video based on task. However, in the experiment, we observed that the attention of AV-LLMs to various modalities in the high layers is not strictly dependent on the task. In higher layers, the attention of AV-LLMs shifts more towards the video modality. In addition, we also found that directly integrating temporal KV of audio and spatial-temporal KV of video may lead to information confusion and significant performance degradation of AV-LLMs. If audio and video are processed indiscriminately, it may also lead to excessive compression or reservation of a certain modality, thereby disrupting the alignment between modalities. To address these challenges, we propose AccKV, an Adaptive-Focusing and Cross-Calibration KV cache optimization framework designed specifically for efficient AV-LLMs inference. Our method is based on layer adaptive focusing technology, selectively focusing on key modalities according to the characteristics of different layers, and enhances the recognition of heavy hitter tokens through attention redistribution. In addition, we propose a Cross-Calibration technique that first integrates inefficient KV caches within the audio and video modalities, and then aligns low-priority modalities with high-priority modalities to selectively evict KV cache of low-priority modalities. The experimental results show that AccKV can significantly improve the computational efficiency of AV-LLMs while maintaining accuracy."},"title":{"value":"AccKV: Towards Efficient Audio-Video LLMs Inference via Adaptive-Focusing and Cross-Calibration KV Cache Optimization"},"authors":{"value":["Zhonghua Jiang","Kui Chen","Kunxi Li","Keting Yin","Yiyun Zhou","Zhaode Wang","Chengfei Lv","Shengyu Zhang"]}},"tmdate":1769350547298,"pdate":1735689600000,"externalIds":["dblp:journals/corr/abs-2511-11106"],"tcdate":1768177917260,"writers":["~"],"signatures":["~Shengyu_Zhang2"],"forum":"g7J1q70Zlz","license":"CC BY-SA 4.0","number":743355,"cdate":1761955200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1769350547298,"domain":"DBLP.org","id":"g7J1q70Zlz","version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"http://arxiv.org/pdf/2509.00105v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"feng|adaptcache_kv_cache_native_storage_hierarchy_for_lowdelay_and_highquality_language_model_serving"},"authorids":{"value":["~Shaoting_Feng1","https://dblp.org/search/pid/api?q=author:Hanchen_Li:","https://dblp.org/search/pid/api?q=author:Kuntai_Du:","https://dblp.org/search/pid/api?q=author:Zhuohan_Gu:","~Yuhan_Liu3","https://dblp.org/search/pid/api?q=author:Jiayi_Yao:","~Siddhant_Ray1","https://dblp.org/search/pid/api?q=author:Samuel_Shen:","https://dblp.org/search/pid/api?q=author:Yihua_Cheng:","~Ganesh_Ananthanarayanan1","https://dblp.org/search/pid/api?q=author:Junchen_Jiang:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2509.00105"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2509-00105,\n publtype={informal},\n author={Shaoting Feng and Hanchen Li and Kuntai Du and Zhuohan Gu and Yuhan Liu and Jiayi Yao and Siddhant Ray and Samuel Shen and Yihua Cheng and Ganesh Ananthanarayanan and Junchen Jiang},\n title={AdaptCache: KV Cache Native Storage Hierarchy for Low-Delay and High-Quality Language Model Serving},\n year={2025},\n month={September},\n cdate={1756684800000},\n journal={CoRR},\n volume={abs/2509.00105},\n url={https://doi.org/10.48550/arXiv.2509.00105}\n}\n"},"abstract":{"value":"Large language model (LLM) applications often reuse previously processed context, such as chat history and documents, which introduces significant redundant computation. Existing LLM serving systems address such redundant computation by storing the KV caches of processed context and loading the corresponding KV cache when a new request reuses the context. Further, as these LLM applications scale, the total size of KV caches becomes excessively large and requires both DRAM and SSD for full storage. However, prior work that stores KV caches in DRAM and SSD suffers from high loading delays, as most KV cache hits come from SSD, which is slow to load. To increase the KV cache hit rate on DRAM, we identify lossy KV cache compression as a promising approach. We design a lossy compression system that decides the compression algorithm, compression rate and device placement for each KV cache entry to maximise DRAM hits and minimise loading delay without significantly degrading generation quality. Compared to various static compression baselines across three tasks, our system AdaptCache achieves 1.43--2.4 x delay savings at the same quality and 6--55% quality improvements at the same delay."},"title":{"value":"AdaptCache: KV Cache Native Storage Hierarchy for Low-Delay and High-Quality Language Model Serving"},"authors":{"value":["Shaoting Feng","Hanchen Li","Kuntai Du","Zhuohan Gu","Yuhan Liu","Jiayi Yao","Siddhant Ray","Samuel Shen","Yihua Cheng","Ganesh Ananthanarayanan","Junchen Jiang"]}},"tmdate":1792423294527,"pdate":1735689600000,"externalIds":["dblp:journals/corr/abs-2509-00105"],"tcdate":1762207354066,"writers":["~"],"signatures":["~Siddhant_Ray1"],"forum":"VMcV9yX4o8","license":"CC BY-SA 4.0","number":650452,"cdate":1756684800000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1792423294527,"domain":"DBLP.org","id":"VMcV9yX4o8","version":2},{"content":{"venue":{"value":"Submitted to ICLR 2026"},"TLDR":{"value":"ReCalKV applies post-training low-rank compression with separate strategies for Keys and Values, enabling accurate long-context inference under high KV cache compression."},"keywords":{"value":["LLM","KV Cache Compression","low-rank approximation"]},"supplementary_material":{"value":"/attachment/225c1daf8b6eb86ea666027843fb7c2b30303824.pdf"},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"Large language models (LLMs) have demonstrated remarkable performance, but their long-context reasoning remains constrained by the excessive memory required for the Key-Value (KV) cache. This makes KV cache compression a critical step toward efficient long-context inference. Recent methods have explored low-rank techniques to reduce the hidden size of the KV cache. However, they neglect the distinct roles and varying importance of Keys and Values, leading to significant performance drops under high compression. To address this, we propose ReCalKV, a post-training low-rank KV cache compression approach with tailored strategies for Keys and Values. For Keys, we propose Head-wise Similarity–aware Reordering (HSR), which clusters structurally similar heads into groups, enabling more accurate low-rank approximation via grouped SVD. For Values, we propose Offline Value Calibration (OVC), which efficiently calibrates the value projection matrix using calibration data without training, ensuring an accurate representation of contextual information. Extensive experiments show that ReCalKV consistently outperforms existing low-rank compression methods, achieving high compression ratios with minimal performance loss. We will release all the code and models."},"_bibtex":{"value":"@misc{\nyan2026recalkv,\ntitle={ReCal{KV}: Low-Rank {KV} Cache Compression via Head Reordering and Offline Calibration},\nauthor={Xianglong Yan and Zhiteng Li and Tianao Zhang and Haotong Qin and Linghe Kong and Yulun Zhang and Xiaokang Yang},\nyear={2026},\nurl={https://openreview.net/forum?id=LQ1qYDOk6o}\n}"},"title":{"value":"ReCalKV: Low-Rank KV Cache Compression via Head Reordering and Offline Calibration"},"pdf":{"value":"/pdf/2e27d73288a7639f2e49b85d2eb4750329a56494.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"yan|recalkv_lowrank_kv_cache_compression_via_head_reordering_and_offline_calibration"},"authorids":{"value":["~Xianglong_Yan2","~Zhiteng_Li2","~Tianao_Zhang1","~Haotong_Qin1","~Linghe_Kong1","~Yulun_Zhang1","~Xiaokang_Yang1"]},"authors":{"value":["Xianglong Yan","Zhiteng Li","Tianao Zhang","Haotong Qin","Linghe Kong","Yulun Zhang","Xiaokang Yang"]}},"tmdate":1770805165033,"tcdate":1758334443961,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission22685/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission22685/Authors"],"forum":"LQ1qYDOk6o","license":"CC BY 4.0","number":22685,"cdate":1758334443961,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/-/Submission","ICLR.cc/2026/Conference/-/Post_Submission","ICLR.cc/2026/Conference/Submission22685/-/Full_Submission","ICLR.cc/2026/Conference/Submission22685/-/Rebuttal_Revision","ICLR.cc/2026/Conference/-/Edit"],"mdate":1770805165033,"odate":1759896705795,"domain":"ICLR.cc/2026/Conference","id":"LQ1qYDOk6o","version":2},{"content":{"TLDR":{"value":"We propose a past- and future-informed KV cache policy that learns token-level importance to enable memory-efficient and fast autoregressive video diffusion without sacrificing generation quality."},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["Autoregressive Video Generation","Video Diffusion Models","KV Cache"]},"primary_area":{"value":"generative models"},"abstract":{"value":"Video generation is pivotal to digital media creation, and recent advances in autoregressive video generation have markedly enhanced the efficiency of real-time video synthesis. However, existing approaches generally rely on heuristic KV Cache policies, which ignore differences in token importance in long-term video generation. This leads to the loss of critical spatiotemporal information and the accumulation of redundant, invalid cache, thereby degrading video generation quality and efficiency. To address this limitation, we first observe that token contributions to video generation are highly time-heterogeneous and accordingly propose a novel Past- and Future-Informed KV Cache Policy (PaFu-KV). Specifically, PaFu-KV introduces a lightweight Salience Estimation Head distilled from a bidirectional teacher to estimate salience scores, allowing the KV cache to retain informative tokens while discarding less relevant ones. This policy yields a better quality-efficiency trade-off by shrinking KV cache capacity and reducing memory footprint at inference time. Extensive experiments on benchmarks demonstrate that our method preserves high-fidelity video generation quality while enables accelerated inference, thereby enabling more efficient long-horizon video generation. Our code will be released upon paper acceptance."},"_bibtex":{"value":"@inproceedings{\nanonymous2026past,\ntitle={Past- and Future-Informed {KV} Cache Policy with Salience Estimation in Autoregressive Video Diffusion},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=O1Zxj17Phu},\nnote={under review}\n}"},"title":{"value":"Past- and Future-Informed KV Cache Policy with Salience Estimation in Autoregressive Video Diffusion"},"pdf":{"value":"/pdf/5d5472bf185775d01a3a12a5677d49fa5b80906f.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791227956713,"tcdate":1789362793881,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission16982/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission16982/Authors"],"forum":"O1Zxj17Phu","license":"CC BY 4.0","number":16982,"cdate":1789362793881,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission16982/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791227956713,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"O1Zxj17Phu","version":2},{"content":{"summary":{"value":"This paper proposes G-KV, a decoding-time KV cache eviction method that introduces a global scoring mechanism combining historical and local attention scores to better preserve long-term token importance. It further enhances performance via post-training techniques—reinforcement learning with sparse attention masks and knowledge distillation. The method is evaluated on challenging mathematical reasoning benchmarks (AMC-23, AIME-24) using strong reasoning LLMs, showing significant gains over prior SOTA, especially under tight KV cache budgets (e.g., +19.15% pass@1 at 256 tokens).\n\nThe work is well-motivated, empirically solid, and addresses a critical bottleneck in long-context reasoning LLMs. The global scoring idea is simple yet effective, and the integration with existing methods (e.g., R-KV) demonstrates strong generalizability. The training-aware extensions (RL-Sparse, Distill) are thoughtfully designed to close the train-inference gap.\nHowever, while the empirical results are compelling, the technical novelty is incremental, and several methodological and evaluation concerns limit the strength of the contribution for a top-tier conferences."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"- How does G-KV perform on non-reasoning tasks (e.g., narrative continuation, summarization) or with standard LLMs (e.g., Llama-3-8B) without reasoning distillation?\n\n- Besides math related task, what about those extremely long context reasoning task in coding domain like SWE bench? Besides, could you provide a computing and memory cost for different context length such as 4K, 8K, 16K, 32K, 64K, etc?\n\n- Since G-KV evicts early tokens, how does the model handle positional information for retained early tokens under RoPE?"},"rating":{"value":4},"details_of_ethics_concerns":{"value":"NA"},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"**Strong Empirical Results:**\n - Clear and consistent improvements over strong baselines across multiple models (Qwen-7B, Llama-8B), datasets (AMC, AIME), and KV budgets.\n - Gains are especially pronounced under low budgets (256–512 tokens), which are practically relevant for deployment.\n\n**Insightful Motivation via Empirical Observation:**\n - Figure 1 provides compelling evidence that token importance is non-stationary across decoding windows—justifying the need for global scoring.\n - The analysis of token retention distribution (Figure 5) convincingly shows that G-KV preserves more diverse context than local methods.\n\n**Practical and Modular Design:**\n - The global score is a drop-in replacement for local scores in existing eviction frameworks (H2O, SnapKV, R-KV).\n - Minimal computational overhead (confirmed by throughput results in Table 4).\n\n**Comprehensive Evaluation:**\n - Includes efficiency metrics (throughput, memory, decoding time), ablation on α and λ, cross-model validation, and qualitative case studies (Appendix J).\n - Post-training methods (RL-Sparse, Distill) are well-motivated and show meaningful gains.\n\n**Reproducibility:**\n - Code, distilled data, and environment configs are provided."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"**Limited Technical Novelty:**\n - The global score (Eq. 3) is essentially an exponentially weighted moving average (EWMA) of normalized attention scores—a well-known technique in online learning and signal processing. While effective, it lacks deep algorithmic innovation.\n - The core idea resembles “heavy-hitter” tracking with decay, similar in spirit to H2O but with historical memory.\n\n**Evaluation Scope is Narrow:**\n - Experiments are restricted to mathematical reasoning on two datasets. No evaluation on general QA, coding, or open-ended generation.\n - All models are distilled reasoning models from DeepSeek-R1. Performance on standard LLMs (e.g., Llama-3, Qwen2) or non-reasoning tasks is unverified.\n - No comparison to non-eviction compression methods (e.g., quantization, low-rank) that may offer better trade-offs.\n\n**Hyperparameter Sensitivity:**\n - Performance heavily depends on α (decay) and λ (redundancy weight). While tuned, the paper doesn’t provide robustness analysis or automatic tuning strategies.\n - The optimal α ≈ 0.8–1.0 suggests historical scores dominate—raising questions about the marginal utility of local scores.\n\n**Ambiguity in Training Protocol:**\n - RL-Sparse uses a sparse attention mask during training, but it’s unclear how this interacts with RoPE (rotary embeddings), which assumes full positional context. This could introduce positional bias.\n - Distillation uses teacher outputs from full KV, but student trains with sparse attention—a distributional mismatch not fully addressed.\n\n**Claimed SOTA May Be Overstated:**\n - The 19.15% improvement is vs. R-KV under 256 tokens—but R-KV itself may not be the strongest baseline (e.g., CAKE, LightThinker are mentioned but not compared in main results).\n - No comparison to recent methods like StreamingLLM or Infini-Attention that handle long contexts differently."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762915636302,"tcdate":1762259394332,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission891/Reviewer_ziNN"],"signatures":["ICLR.cc/2026/Conference/Submission891/Reviewer_ziNN"],"forum":"4Uxtzc5Jns","number":5,"license":"CC BY 4.0","cdate":1762259394332,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission891/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762915636302,"domain":"ICLR.cc/2026/Conference","replyto":"4Uxtzc5Jns","id":"lHe9BZ8DqE","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["KV Cache Compression; Large Language Models; Reinforcement Learning"]},"supplementary_material":{"value":"/attachment/48454fd95b3a373738365dc4a3ea63760f72877b.zip"},"primary_area":{"value":"generative models"},"abstract":{"value":"Recent reasoning large language models (LLMs) excel in complex tasks but encounter significant computational and memory challenges due to long sequence lengths. KV cache compression has emerged as an effective approach to greatly enhance the efficiency of reasoning. However, existing methods often focus on prompt compression or token eviction with local attention score, overlooking the long-term importance of tokens. We propose G-KV, a KV cache eviction method that employs a global scoring mechanism, combining local and historical attention scores to more accurately assess token importance. Additionally, we introduce post-training techniques, including reinforcement learning and distillation, to optimize models for compressed KV cache settings. The code of this paper is available on:https://anonymous.4open.science/r/G-KV-B3C0 ."},"_bibtex":{"value":"@misc{\nliao2026gkv,\ntitle={G-{KV}: Decoding-Time {KV} Cache Eviction with Global Attention},\nauthor={Mengqi Liao and Lu Wang and Chaoyun Zhang and Zekai Shen and Xiaowei Mao and Si Qin and Qingwei Lin and Saravan Rajmohan and Dongmei Zhang and Huaiyu Wan},\nyear={2026},\nurl={https://openreview.net/forum?id=4Uxtzc5Jns}\n}"},"title":{"value":"G-KV: Decoding-Time KV Cache Eviction with Global Attention"},"pdf":{"value":"/pdf/3f790d85944dd4a7c352122c5e3f40f325ce18ff.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"liao|gkv_decodingtime_kv_cache_eviction_with_global_attention"},"authorids":{"value":["~Mengqi_Liao1","~Lu_Wang11","~Chaoyun_Zhang1","~Zekai_Shen1","~Xiaowei_Mao1","~Si_Qin2","~Qingwei_Lin1","~Saravan_Rajmohan3","~Dongmei_Zhang2","~Huaiyu_Wan1"]},"authors":{"value":["Mengqi Liao","Lu Wang","Chaoyun Zhang","Zekai Shen","Xiaowei Mao","Si Qin","Qingwei Lin","Saravan Rajmohan","Dongmei Zhang","Huaiyu Wan"]}},"version":2},{"content":{"TLDR":{"value":"JoLT compresses the KV cache of LLMs with a partial Tucker decomposition plus a rotated low-bit residual, allocating ranks and bits jointly under one byte budget for near-lossless 2-3x compression across five models without retraining."},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["KV-cache compression","large language models","Tucker decomposition","low-rank approximation","quantization","Lagrangian relaxation","rate-distortion","long-context inference","randomized SVD","fused attention kernel"]},"supplementary_material":{"value":"/attachment/cfccd359aabd9455cdaaa716f8613a1ca99b9c9f.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"The key-value (KV) cache is the dominant memory bottleneck in long-context language model inference.\nExisting compression methods apply low-rank factorization or quantization independently, without jointly allocating rank and precision under a shared storage budget.\nWe introduce JoLT, a training-free compressor that treats grouped prefill caches as fourth-order tensors and applies partial Tucker decomposition along the token and feature modes, the two axes that carry low-rank structure, while leaving the head and layer modes intact.\nA rotated low-bit quantizer captures the truncation residual, and a single Lagrangian dual allocates per-group Tucker ranks and residual bit-widths under a global byte constraint.\nFlashJoLT replaces the exact token-mode SVD with a randomized approximation that matches JoLT within the free zone at a fraction of the compression cost, and a fused Triton decode kernel evaluates attention directly over the stored factors without materializing dense KV tensors.\nAcross five models from four architecture families, covering multi-head attention, grouped-query attention, and mixture-of-experts architecture, JoLT achieves 2 - 3x compression with less than 0.2% perplexity degradation, without retraining.\nOn RULER at 64K context with LLaMA-3.1-8B, retrieval accuracy remains near-lossless through 3x and declines by only 0.90 and 2.40pp at 4x and 5x, respectively.\nJoLT demonstrates that tensor-aware low-rank decomposition and quantized residuals, unified under a single storage budget, achieve near-lossless KV-cache compression across diverse model architectures without retraining."},"_bibtex":{"value":"@inproceedings{\nanonymous2026jolt,\ntitle={Jo{LT}: Near-Lossless {KV} Cache Compression via Joint Rank-Bit Allocation},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=EpP44Vczot},\nnote={under review}\n}"},"title":{"value":"JoLT: Near-Lossless KV Cache Compression via Joint Rank-Bit Allocation"},"pdf":{"value":"/pdf/c9ebaefa519d48bc591f30291933adf3a4e2ac6e.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791230463741,"tcdate":1789566399062,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission25538/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission25538/Authors"],"forum":"EpP44Vczot","license":"CC BY 4.0","number":25538,"cdate":1789566399062,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Edit","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission25538/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing"],"mdate":1791230463741,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"EpP44Vczot","version":2},{"content":{"summary":{"value":"The paper proposes QubitCache, a method that retains 15% of tokens, comprising 10% of the most recent tokens, 5% of tokens with high attention scores, and 4 anchor tokens. For the remaining tokens, the model reconstructs the pruned value vectors by interpolating between neighboring value vectors and reconstructs attention scores using quantum amplitude encoding. The authors compare QubitCache with ScissorHand, H2O, StreamingLLM, and GEAR baselines on both short- and long-context understanding tasks."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"See the above weakness section."},"rating":{"value":2},"details_of_ethics_concerns":{"value":"I do not have ethics concern regarding this paper."},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"The paper introduces a novel perspective on KV cache compression by leveraging quantum-based encoding and reconstructing the value vectors through interpolation."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- Concerns about experiments\n - The paper is missing several state-of-the-art baselines, such as SnapKV [1] and PyramidKV [2].\n - In Table 1, the compression ratio is not consistent across baselines and results are only reported for a single compression ratio for each method. It would be more informative to include a plot showing accuracy versus compression ratio (with the x-axis representing the compression ratio and the y-axis representing accuracy).\n - Additionally, based on the memory compression ratios reported in Table 3, QubitCache retains at least 15% of the cache, implying a maximum possible compression ratio of approximately 100 / 15 = 6.67x. However, the table reports a ratio of 7.0x, which appears to be inconsistent.\n- Concerns about the method\n - The proposed method reconstructs value vectors at inference time by interpolating neighboring value vectors and extracting attention patterns. However, the paper does not provide any discussion or measurement of the inference-time latency or acceleration ratio, which is essential for understanding the practical efficiency of the approach.\n - Equation (6) seems to assume that value vectors are linearly aligned, which is unlikely to hold in practice. It would be valuable to verify this assumption through statistical analysis, as it may not be generally true.\n - Furthermore, it would be interesting to explore simpler alternatives that do not rely on quantum-based amplitude encoding for reconstructing attention pattern. For instance, one could reconstruct value vectors using an averaged representation (e.g., $\\bar{V} = \\sum_{j \\in I_C} \\tilde{V}_j$) with a weighting factor ($1 - \\lambda$), or apply this in a fine-grained, group-wise manner (e.g., over every (g) tokens). Such comparisons would help clarify whether the quantum-inspired encoding provides a meaningful advantage over simpler interpolation strategies.\n- Minor\n - The first sentence of the abstract is incorrect. The KV cache memory grows linearly with the sequence length, whereas the total FLOPs computed phase grow quadratically.\n\n[1] Li, Yuhong, et al. \"Snapkv: Llm knows what you are looking for before generation.\" NeurIPS 2024.\n\n[2] Cai, Zefan, et al. \"Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling.\", ACL 2024 Findings."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762921987026,"tcdate":1761997172244,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission10775/Reviewer_wZek"],"signatures":["ICLR.cc/2026/Conference/Submission10775/Reviewer_wZek"],"forum":"WhO6Km5Rku","number":3,"license":"CC BY 4.0","cdate":1761997172244,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission10775/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762921987026,"domain":"ICLR.cc/2026/Conference","replyto":"WhO6Km5Rku","id":"vNa9Ng2v5p","forumContent":{"TLDR":{"value":"QubitCache achieves 10× KV-cache compression by preserving attention relationships through quantum-inspired amplitude encoding instead of binary token selection."},"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Quantum Machine Learning","Quantum Circuit","Large Language Model","KV Compression","Optimization"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Large language model inference suffers from quadratic KV cache memory growth that fundamentally limits long context applications. Existing compression methods achieve memory reduction through token eviction but irreversibly discard relational information essential for complex reasoning. We present QubitCache, the first framework recognizing that attention patterns between tokens constitute the primary information carrier in transformers, not tokens themselves. This insight motivates a paradigm shift from discrete token selection to continuous relational preservation through quantum-inspired encoding. QubitCache introduces a hybrid architecture where critical tokens remain in classical storage while attention patterns undergo amplitude encoding into quantum states, achieving logarithmic compression beyond classical information-theoretic limits. Unlike binary dcisions, our framework generates probabilistic attention distributions through quantum state measurements, maintaining contextual coherence via soft attention constraints. We prove QubitCache preserves rank $r$ attention structure with bounded reconstruction error, ensuring graceful degradation rather than catastrophic failure. Empirical evaluation demonstrates $7\\times$ memory reduction while maintaining 92-97\\% of baseline performance across five models and six benchmarks. Remarkably, QubitCache achieves this with only 15\\% token retention compared to 50\\% in existing SOTA methods, yet attains 15-25\\% higher F1 scores on multi-hop reasoning tasks."},"_bibtex":{"value":"@misc{\nkang2025qubitcache,\ntitle={QubitCache: Quantum-Inspired Probabilistic Attention Preservation for {KV}-Cache Compression},\nauthor={Jieui Kang and Jaeyoung Choi and Wonhui Roh and Jaehyeong Sim},\nyear={2025},\nurl={https://openreview.net/forum?id=WhO6Km5Rku}\n}"},"title":{"value":"QubitCache: Quantum-Inspired Probabilistic Attention Preservation for KV-Cache Compression"},"pdf":{"value":"/pdf/887ecca98a0f4ea6aa77df006d333141a01a65c2.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"kang|qubitcache_quantuminspired_probabilistic_attention_preservation_for_kvcache_compression"},"authorids":{"value":["~Jieui_Kang2","~Jaeyoung_Choi6","~Wonhui_Roh1","~Jaehyeong_Sim1"]},"authors":{"value":["Jieui Kang","Jaeyoung Choi","Wonhui Roh","Jaehyeong Sim"]}},"version":2},{"content":{"TLDR":{"value":"We discover a critical behavioral degradation phenomenon in quantized agentic coding and propose a pinpoint restoration strategy to mitigate it."},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["KV Cache Quantization","Agentic Coding","Behavioral Degradation Analysis"]},"supplementary_material":{"value":"/attachment/cc55e94837234686b97bc8cb4fa19f2a86bddb89.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Agentic coding has emerged as a prominent application of LLMs, but its multi-turn interaction trajectories continually expand the KV cache, imposing substantial memory demands during inference. KV cache quantization reduces per-token memory usage, but do these savings translate into more efficient agent trajectories? We find that low-bit quantization disproportionately amplifies three failures: repetitive action loops, improper tool use, and insufficient debugging, while increasing the actions and tokens required per task. To identify which attention heads are most sensitive to quantization around these failures, we measure changes in attention distributions and head outputs under controlled KV cache perturbations. Highly sensitive heads comprise only about 5% of all heads and cluster in a few layers. Guided by this localization, we introduce a simple and effective mixed-precision strategy that restores higher precision in selected layers while retaining low precision elsewhere. On SWE-bench Verified with Qwen3-Coder-30B-A3B-Instruct, increasing average precision from $3.0$ to $3.35$ bits substantially reduces the three behavioral failures and brings task resolution close to BF16. Despite higher per-token precision, observed KV cache usage per case decreases by 12% versus uniform 3-bit quantization. These results shift the focus of agent quantization from storing each token more cheaply to completing tasks more reliably with less memory."},"_bibtex":{"value":"@inproceedings{\nanonymous2026fewer,\ntitle={Fewer Bits, More Tokens: Rethinking {KV} Cache Quantization in Agentic Coding},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=Y0Ww5DnHS4},\nnote={under review}\n}"},"title":{"value":"Fewer Bits, More Tokens: Rethinking KV Cache Quantization in Agentic Coding"},"pdf":{"value":"/pdf/808e6c6028838eeda4029830a489b3025774b488.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791229643113,"tcdate":1789461811519,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission20488/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission20488/Authors"],"forum":"Y0Ww5DnHS4","license":"CC BY 4.0","number":20488,"cdate":1789461811519,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission20488/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791229643113,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"Y0Ww5DnHS4","version":2},{"content":{"research_area_keywords":{"value":"multi-agent systems,agent memory,agent communication"},"keywords":{"value":["LLM serving","KV cache","asymmetric granularity KV cache paging","shared prefix","multi-agent context","multi-agent systems","GPU memory management"]},"B_use_or_create_scientific_artifacts":{"value":"Yes"},"languages_studied":{"value":"Python"},"D4_ethics_review_board_approval":{"value":"N/A"},"C4_parameters_for_packages":{"value":"Yes"},"B6_elaboration":{"value":"4"},"B1_cite_creators_of_artifacts":{"value":"Yes"},"B2_discuss_the_license_for_artifacts":{"value":"No"},"B4_elaboration":{"value":"Ethics Statement"},"D_human_subjects_including_annotators":{"value":"No"},"A2_elaboration":{"value":"Ethics Statement"},"C3_elaboration":{"value":"4"},"B2_elaboration":{"value":"The license or terms for use will be disclosed on the github repository after it is disclosed on camera-ready."},"B3_elaboration":{"value":"Ethics Statement"},"C2_elaboration":{"value":"4"},"C4_elaboration":{"value":"4"},"E_ai_assistants_in_research_or_writing":{"value":"Yes"},"C1_elaboration":{"value":"4"},"B1_elaboration":{"value":"3.2"},"C2_experimental_setup_and_hyperparameters":{"value":"Yes"},"E1_elaboration":{"value":"Acknowledgements"},"C1_model_size_and_budget":{"value":"Yes"},"venue":{"value":"ACL ARR 2026 May Submission"},"D3_data_consent":{"value":"N/A"},"_bibtex":{"value":"@inproceedings{\nanonymous2026granikv,\ntitle={Grani{KV}: Asymmetric Granularity {KV}-Cache Paging for Multi-Agent Systems with Long Shared Prefix},\nauthor={Anonymous},\nbooktitle={Submitted to ACL Rolling Review - May 2026},\nyear={2026},\nurl={https://openreview.net/forum?id=4qZhFzZakX},\nnote={under review}\n}"},"title":{"value":"GraniKV: Asymmetric Granularity KV-Cache Paging for Multi-Agent Systems with Long Shared Prefix"},"C3_descriptive_statistics":{"value":"Yes"},"contribution_types":{"value":["NLP engineering experiment","Approaches to low-compute settings (efficiency)","Publicly available software and/or pre-trained models","Data analysis"]},"A1_limitations_section":{"value":"This paper has a limitations section."},"B6_statistics_for_data":{"value":"Yes"},"B4_data_contains_personally_identifying_info_or_offensive_content":{"value":"Yes"},"B3_artifact_use_consistent_with_intended_use":{"value":"Yes"},"A2_potential_risks":{"value":"Yes"},"E1_information_about_use_of_ai_assistants":{"value":"Yes"},"abstract":{"value":"Production paged-serving engines apply uniform paging granularity to\nthe KV cache, even though the two regions of a multi-agent\nworkload have opposite storage requirements: a long shared prefix\ndemands contiguity, while the per-request suffix demands fine-grained\nallocation. \n\nWe present \\textbf{GraniKV}, a KV-cache layer that\nallocates the shared prefix in a contiguous HOT pool and the suffix\nin a token-level COLD pool, combined with a per-step dispatcher\nwhich selects the appropriate backend among dual backends\nfor each regime (compute-, memory-, or communication-bound). \nTo the best of our knowledge, GraniKV is the first system to apply \nasymmetric paging granularity to the KV cache\nof a production paged-serving engine.\n\nOn ShareGPT-V3 user turns prepended with a fixed shared system\nprompt and served under Poisson arrivals at saturation rate, the\ngain grows monotonically with the shared-prefix length $L_p$ and is\nlargest at long prefixes --- exactly the regime that drives\nproduction multi-agent workloads. At $L_p{=}16$\\,K shared-prefix\ntokens GraniKV reaches $\\mathbf{2.16\\times}$, $\\mathbf{1.98\\times}$,\nand $\\mathbf{1.57\\times}$ the output-token throughput of the\nproduction baseline on Llama-3.1-8B/TP=1, Qwen-2.5-14B/TP=2, and\nQwen-2.5-32B/TP=4 (TP = tensor-parallel rank count), with no\nregression at short prefixes."},"D1_instructions_given_to_participants":{"value":"N/A"},"paper_type":{"value":"Long"},"C_computational_experiments":{"value":"Yes"},"pdf":{"value":"/pdf/9f037092b6d1b33fc530d186bccaadf26ad18ee6.pdf"},"research_area":{"value":"LLM agents"},"B5_documentation_of_artifacts":{"value":"N/A"},"EMNLP_2026_AI_Reviewing_Experiment":{"value":"yes"},"D2_recruitment_and_payment":{"value":"N/A"},"venueid":{"value":"aclweb.org/ACL/ARR/2026/May/Submission"}},"tmdate":1790696971606,"tcdate":1779720572706,"writers":["aclweb.org/ACL/ARR/2026/May","aclweb.org/ACL/ARR/2026/May/Submission8434/Authors"],"signatures":["aclweb.org/ACL/ARR/2026/May/Submission8434/Authors"],"forum":"4qZhFzZakX","license":"CC BY 4.0","number":8434,"cdate":1779720572706,"readers":["everyone"],"invitations":["aclweb.org/ACL/ARR/2026/May/-/Submission","aclweb.org/ACL/ARR/2026/May/-/Edit","aclweb.org/ACL/ARR/2026/May/-/Post_Submission","aclweb.org/ACL/ARR/2026/May/-/Preprint_Post_Submission","aclweb.org/ACL/ARR/2026/May/-/Desk_Reject_Verification_Flag"],"mdate":1790696971606,"odate":1780385598983,"domain":"aclweb.org/ACL/ARR/2026/May","id":"4qZhFzZakX","version":2},{"content":{"TLDR":{"value":"This paper proposes a training-free 2-bit KV cache quantization framework, QuantWM, which eliminates temporal flickering and visual degradition in video world models through attention preservation.."},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["Model quantization","KV cache","world model","video generation"]},"supplementary_material":{"value":"/attachment/e635aeb856c8acbf97e4c25281115539e1482e77.zip"},"primary_area":{"value":"generative models"},"abstract":{"value":"Video generation based world models achieve long-range temporal consistency by storing KV cache during generation, but the continuously growing cache makes KV cache memory become a major deployment bottleneck, which motivates low-bit quantization study for efficiency. Existing 2-bit KV cache quantization methods can achieve nearly quantitatively lossless performance on conventional video benchmarks such as VBench, however, when applied to video world models, we find that they still cause severe temporal flickering and visual degradation. Meanwhile, deeper investigates show that Key quantization produces smaller reconstruction errors than Value, but surprisingly leads to much larger output degradation. We trace this discrepancy to attention in video world models: Key perturbations can change the attention logits, \\textit{i.e.}, \\textit{QK$^\\top$}, and shift the temporal-spatial tokens selected by Queries. These observations motivate us to explicitly preserve attention logits and temporal-spatial token selection during KV cache quantization to alleviate the visual degradation problem. To address this issue, we present \\textbf{Quant\\textit{WM}}, a training-free 2-bit KV cache quantization framework for video world models. \\textbf{Quant\\textit{WM}} introduces two complementary techniques to mitigate the attention shifts. Firstly, \\textit{quantization-sensitivity-aware clustering (QSAC)} jointly considers historical Query sensitivity and residual ranges to select INT2-friendly Key centroids, which reduces quantization errors in channels that are more critical to attention. In addition, \\textit{principal-subspace attention compensation (PSAC)} restores the remaining Key errors along the dominant Query subspace using low-rank projections, which provides a direct and efficient correction to stabilize attention logits. Extensive experiments on LingBot-World-v2, HY-World 1.5, Matrix-Game-2, Longcat-Video and Causal-Forcing demonstrate that \\textbf{Quant\\textit{WM}} significantly improves visual quality and temporal consistency, while outperforming existing methods across image and video quality metrics with up to \\(6.20\\times\\) KV cache memory compression and limited additional overhead. The project page will be released soon."},"_bibtex":{"value":"@inproceedings{\nanonymous2026quantwm,\ntitle={Quant{WM}: Temporally Consistent 2-Bit {KV} Cache Quantization for Video World Models},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=B3TYCMjqzj},\nnote={under review}\n}"},"title":{"value":"QuantWM: Temporally Consistent 2-Bit KV Cache Quantization for Video World Models"},"pdf":{"value":"/pdf/b915c9e3c26138a616d19df10dda5be3621ac13c.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791225498764,"tcdate":1788967513281,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission12488/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission12488/Authors"],"forum":"B3TYCMjqzj","license":"CC BY 4.0","number":12488,"cdate":1788967513281,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/Submission12488/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791225498764,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"B3TYCMjqzj","version":2},{"content":{"summary":{"value":"Diffusion Large Language Models (dLLMs) have bidirectional attention strengths but face $O(L^3)$ computational complexity and inefficient static acceleration methods. Dynamic-dLLM, a training-free framework, solves this via two components: Dynamic Cache Updating (DCU) which adaptively allocates layer-wise cache budgets using layer input cosine distance (proxying intermediate changes) and a mandatory update window to avoid \"token stuck\", and Adaptive Parallel Decoding (APD) which adjusts thresholds via prediction concentration and temporal stability. Tested on NVIDIA Pro6000 with LLaDA-8B-Instruct/1.5, Dream-v0-7B-Instruct across 5 benchmarks, it achieves average 3× speedup (max 4.48× on GSM8K) while maintaining accuracy (difference ≤1% vs. baselines), outperforming dLLM-Cache, dKV-Cache, and Fast-dLLM."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"I would be happy to increase my rating if my views are given a thorough discussion."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"+ **Targeted Dynamic Adaptation Addresses Core Pain Points**：The paper is the first to systematically capture two key dynamic characteristics of dLLMs—heterogeneous inter-layer cache update demands and step-wise fluctuations in token confidence distribution—overcoming the limitations of existing static acceleration methods. For Dynamic Cache Updating (DCU), it uses layer inputs as a proxy for intermediate feature changes (Spearman correlation with Key/Value features > 0.94), avoiding the computational overhead of recomputing intermediate tensors while ensuring accurate update decisions. For Adaptive Parallel Decoding (APD), it adjusts thresholds by combining prediction distribution concentration and historical stability, solving the misjudgment issue of fixed-threshold methods (e.g., preventing premature unmasking of unstable tokens). \n+ **Training-Free Design Enables Strong Engineering Practicality**：Dynamic-dLLM requires no model fine-tuning or architectural modifications; it only optimizes inference via dynamic strategies, allowing direct integration as a \"plug-and-play\" module into open-source dLLMs (e.g., LLaDA-8B-Instruct, Dream-v0-7B-Instruct). The \"Mandatory Update Window\" (solving the \"token stuck in the mud\" problem) enhances engineering robustness, and no additional large caches are introduced, keeping memory overhead manageable and reducing deployment costs. \n+ **Comprehensive Experiments with High Persuasiveness**：Experiments cover 3 representative dLLMs, 5 task types (general reasoning, math, coding, QA), and 4 SOTA baselines (dLLM-Cache, dKV-Cache, Fast-dLLM). Results consistently show 2.45×–4.48× speedups with ≤1% accuracy loss (e.g., LLaDA-1.5’s average accuracy 60.67% vs. baseline 61.08%). Ablation studies (on $B_{\\text{layer}}$, $B_{\\text{window}}$, threshold types) fully validate the necessity of each component, and hyperparameter guidelines (e.g., optimal $B_{\\text{window}}=32$) improve result reproducibility. \n+ **Tight Integration of Empirical Observations and Theoretical Foundations**：The method design is rooted in solid empirical insights—such as the monotonic increase in inter-layer update demand from shallow to deep layers, and the local update pattern around key tokens—rather than pure heuristics. Quantitative metrics (cosine distance for token variation, distribution concentration for confidence) provide mathematical grounding, enhancing the method’s generality and interpretability."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"+ **Key parameters lack adaptability**：Mandatory update window size ($B_{\\text{window}}=32$) and APD hyperparameters ($\\alpha=0.001$, $\\beta=0.0008$) are fixed, failing to adapt to sequence length (e.g., redundant for short sequences, insufficient for ultra-long ones) or task differences, leading to suboptimal performance. \n+ **Extreme scenarios untested**：No validation on ultra-long sequences (>1K tokens) (to verify cache/memory stability) or low-resource hardware (e.g., RTX 3090) (to check cost-constrained deployment), limiting practicality. \n+ **Incomplete baseline comparisons**：Missing cross-paradigm methods (e.g., ES-dLLM’s early skipping) and industrial-grade dLLMs (e.g., Mercury), failing to fully prove relative advantages."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762922807684,"tcdate":1761991538513,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission11784/Reviewer_sidX"],"signatures":["ICLR.cc/2026/Conference/Submission11784/Reviewer_sidX"],"forum":"SdnkB5pGbq","number":3,"license":"CC BY 4.0","cdate":1761991538513,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission11784/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762922807684,"domain":"ICLR.cc/2026/Conference","replyto":"SdnkB5pGbq","id":"omrw1Dgri3","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["dLLMs","Inference Acceleration"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Diffusion Large Language Models (dLLMs) offer a promising alternative to autoregressive models, excelling in text generation tasks due to their bidirectional attention mechanisms. However, their computational complexity, scaling as $\\mathcal{O}(L^3)$ with sequence length $L$, poses significant challenges for long-sequence and real-time applications, primarily due to the lack of compatibility with key-value caching and the non-autoregressive nature of denoising steps. Existing acceleration methods rely on static caching or parallel decoding strategies, which fail to account for the dynamic behavior of token properties across layers and decoding steps. We propose \\textbf{Dynamic-dLLM}, a training-free framework that enhances dLLM inference efficiency through two components: Dynamic Cache Updating (DCU), which adaptively allocates cache-update budgets based on layer-wise token dynamics, and Adaptive Parallel Decoding (APD), which dynamically calibrates decoding thresholds to balance generation quality and efficiency. Extensive experiments on models like LLaDA-8B-Instruct, LLaDA-1.5, and Dream-v0-7B-Instruct across benchmarks such as MMLU, GSM8K, and HumanEval demonstrate that Dynamic-dLLM significantly improves inference speed, attaining an average speedup of exceeding 3$\\times$ while maintaining performance. Dynamic-dLLM outperforms state-of-the-art acceleration methods and provides a plug-and-play solution for efficient dLLM deployment without compromising performance. Code and models will be made publicly available."},"_bibtex":{"value":"@inproceedings{\nwu2026dynamicdllm,\ntitle={Dynamic-d{LLM}: Dynamic Cache-Budget and Adaptive Parallel Decoding for Training-Free Acceleration of Diffusion {LLM}},\nauthor={Tianyi Wu and Xiaoxi Sun and Yanhua Jiao and Yulin Li and Yixin Chen and Yun-Hao Cao and Yi-Qi Hu and Zhuotao Tian},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=SdnkB5pGbq}\n}"},"title":{"value":"Dynamic-dLLM: Dynamic Cache-Budget and Adaptive Parallel Decoding for Training-Free Acceleration of Diffusion LLM"},"pdf":{"value":"/pdf/f458189f5dfda8e18b960ad515402fb9546329f4.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"wu|dynamicdllm_dynamic_cachebudget_and_adaptive_parallel_decoding_for_trainingfree_acceleration_of_diffusion_llm"},"authorids":{"value":["~Tianyi_Wu6","~Xiaoxi_Sun2","~Yanhua_Jiao1","~Yulin_Li4","~Yixin_Chen4","~Yun-Hao_Cao1","~Yi-Qi_Hu2","~Zhuotao_Tian1"]},"authors":{"value":["Tianyi Wu","Xiaoxi Sun","Yanhua Jiao","Yulin Li","Yixin Chen","Yun-Hao Cao","Yi-Qi Hu","Zhuotao Tian"]}},"version":2},{"content":{"venue":{"value":"CoRR 2026"},"pdf":{"value":"https://arxiv.org/pdf/2602.02199v1"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"sood|more_than_a_quick_glance_overcoming_the_greedy_bias_in_kvcache_compression"},"html":{"value":"https://doi.org/10.48550/arXiv.2602.02199"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2602-02199,\n publtype={informal},\n author={Aryan Sood and Tanvi Sharma and Vansh Agrawal},\n title={More Than a Quick Glance: Overcoming the Greedy Bias in KV-Cache Compression},\n year={2026},\n month={February},\n cdate={1769904000000},\n journal={CoRR},\n volume={abs/2602.02199},\n url={https://doi.org/10.48550/arXiv.2602.02199}\n}\n"},"abstract":{"value":"While Large Language Models (LLMs) can theoretically support extensive context windows, their actual deployment is constrained by the linear growth of Key-Value (KV) cache memory. Prevailing compression strategies mitigate this through various pruning mechanisms, yet trade-off semantic recall for memory efficiency. In this work, we present LASER-KV (Layer Accumulated Selection with Exact-LSH Recall), a framework designed to test the limits of KV compression under a strict accumulative budgeting policy. We deviate from the standard fixed summary size approach by implementing a block-wise accumulation strategy governed by a protection divisor (n). This allows us to isolate the effects of compression from sliding window artifacts. Our experiments on the Babilong benchmark reveal performance degradation in previous compression methods by 15-30% on various long context tasks. LASER-KV maintains stable performance, achieving superior accuracies by a margin of upto 10% at 128k. These findings challenge the prevailing assumption that attention scores alone are a sufficient proxy for token utility."},"title":{"value":"More Than a Quick Glance: Overcoming the Greedy Bias in KV-Cache Compression"},"authors":{"value":[{"fullname":"Aryan Sood","username":"~Aryan_Sood3"},{"fullname":"Tanvi Sharma","username":""},{"fullname":"Vansh Agrawal","username":""}]}},"tmdate":1779826367824,"pdate":1798675200000,"externalIds":["dblp:journals/corr/abs-2602-02199"],"tcdate":1779784941771,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Vansh_Agrawal1"],"forum":"zceLVtd0CG","license":"CC BY-SA 4.0","number":27661,"cdate":1769904000000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit","OpenReview.net/Public_Article/-/Author_Removal","OpenReview.net/Public_Article/-/Authorship_Claim"],"mdate":1779826367824,"domain":"OpenReview.net/Public_Article","id":"zceLVtd0CG","version":2},{"content":{"data_release":{"value":"We authorize the release of our submission and author names to the public in the event of acceptance."},"TLDR":{"value":"We introduce a novel KV-cache compression method utilizing a hybrid selection mechanism"},"venue":{"value":"ICLR 2026 Workshop ICBINB"},"email_sharing":{"value":"We authorize the sharing of all author emails with Program Chairs."},"pdf":{"value":"/pdf/6d4185f923338288144b0ce702f8f3a8d7fc805f.pdf"},"keywords":{"value":["Large Language Models","Efficient Inference","KV Cache Compression. Long-Context Reasoning"]},"venueid":{"value":"ICLR.cc/2026/Workshop/ICBINB"},"paperhash":{"value":"sood|more_than_a_quick_glance_overcoming_the_greedy_bias_in_kvcache_compression"},"authorids":{"value":["~Aryan_Sood3","~Tanvi_Sharma3","~Vansh_Agrawal1"]},"abstract":{"value":"While Large Language Models (LLMs) can theoretically support extensive context windows, their actual deployment is constrained by the linear growth of Key-Value (KV) cache memory. Prevailing compression strategies mitigate this through various pruning mechanisms, yet trade-off semantic recall for memory efficiency. In this work, we present LASER-KV (Layer Accumulated Selection with Exact-LSH Recall), a framework designed to test the limits of KV compression under a strict accumulative budgeting policy. We deviate from the standard\nfixed summary size approach by implementing a block-wise accumulation strategy governed by a protection divisor (n). This allows us to isolate the effects of compression from sliding window artifacts. Our experiments on the Babilong benchmark reveal performance degradation in previous compression methods by 15–30% on various long context tasks. LASER-KV maintains stable performance,\nachieving superior accuracies by a margin of upto 10% at 128k. These findings challenge the prevailing assumption that attention scores alone are a sufficient proxy for token utility."},"title":{"value":"More Than a Quick Glance: Overcoming the Greedy Bias in KV-Cache Compression"},"authors":{"value":["Aryan Sood","Tanvi Sharma","Vansh Agrawal"]}},"tmdate":1772990680956,"pdate":1772479654705,"tcdate":1769936861112,"writers":["ICLR.cc/2026/Workshop/ICBINB","ICLR.cc/2026/Workshop/ICBINB/Submission100/Authors"],"signatures":["ICLR.cc/2026/Workshop/ICBINB/Submission100/Authors"],"forum":"fAeRWQLSmj","license":"CC BY 4.0","number":100,"cdate":1769936861112,"readers":["everyone"],"invitations":["ICLR.cc/2026/Workshop/ICBINB/-/Submission","ICLR.cc/2026/Workshop/ICBINB/-/Submission_Change_Before_Bidding","ICLR.cc/2026/Workshop/ICBINB/-/Submission_Change_Before_Reviewing","ICLR.cc/2026/Workshop/ICBINB/-/Submission_Release","ICLR.cc/2026/Workshop/ICBINB/Submission100/-/Camera_Ready_Revision"],"mdate":1772990680956,"odate":1772479654705,"domain":"ICLR.cc/2026/Workshop/ICBINB","id":"fAeRWQLSmj","version":2},{"content":{"TLDR":{"value":"Adversarial images can be trained to poison the KV cache and persist through attention masking."},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["vision-language models","adversarial attacks","prompt injection","KV cache"]},"primary_area":{"value":"alignment, fairness, safety, privacy, and societal considerations"},"abstract":{"value":"Modern language model systems operate autonomously over increasingly long contexts containing untrusted text and images. Can an adversarial input continue to steer a model even after that input is removed from its context? We show that attacks can be trained to persist through the key/value (KV) cache of subsequent tokens, allowing adversarial influence to outlive direct access to its source. We consider the Visual Memory Injection (VMI; Schlarmann & Hein, 2026) attack setting, in which an adversarial image that stays in the context plants a hidden backdoor: the model behaves normally until a chosen trigger elicits an attacker-chosen response. We first demonstrate persistence in this setting with optimized soft prompts, which remain effective after we mask the prompt from attention. We then introduce Persistent Visual Memory Injection (P-VMI), which optimizes images to preserve this adversarial behaviour after they are masked from attention. These attacks persist over conversations substantially longer than those used during optimization. On Qwen3-VL-8B-Instruct, P-VMI achieves up to approximately 90% target success in its strongest configuration and remains effective under a stricter removal setting that exposes the image only on the first turn. A cache-swap ablation localizes the persistent influence to the KV cache. Finally, we show that these attacks can be trained to survive compaction that retains the KV cache of a summary generated by the same model, demonstrating that adversarial behaviour can persist in cached state without continued access to its source."},"_bibtex":{"value":"@inproceedings{\nanonymous2026visual,\ntitle={Visual Memory Attacks Can Persist Through The {KV} Cache},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=BGfO0KuFnL},\nnote={under review}\n}"},"title":{"value":"Visual Memory Attacks Can Persist Through The KV Cache"},"pdf":{"value":"/pdf/0f91e588875db12abd7ec04f41038f7dc2817ee9.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791234208169,"tcdate":1789766857609,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission49101/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission49101/Authors"],"forum":"BGfO0KuFnL","license":"CC BY 4.0","number":49101,"cdate":1789766857609,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission49101/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791234208169,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"BGfO0KuFnL","version":2},{"content":{"venue":{"value":"ACL (1) 2025"},"pdf":{"value":"https://aclanthology.org/2025.acl-long.652.pdf"},"venueid":{"value":"dblp.org/conf/ACL/2025"},"paperhash":{"value":"li|madakv_adaptive_modalityperception_kv_cache_eviction_for_efficient_multimodal_longcontext_inference"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Kunxi_Li:","https://dblp.org/search/pid/api?q=author:Zhonghua_Jiang_0007:","~Zhouzhou_Shen1","https://dblp.org/search/pid/api?q=author:Zhaode_Wang:","https://dblp.org/search/pid/api?q=author:Chengfei_Lv:","~Shengyu_Zhang2","~Fan_Wu10","https://dblp.org/search/pid/api?q=author:Fei_Wu_0001:"]},"html":{"value":"https://aclanthology.org/2025.acl-long.652/"},"_bibtex":{"value":"@inproceedings{DBLP:conf/acl/Li0SWLZ0025,\n author={Kunxi Li and Zhonghua Jiang and Zhouzhou Shen and Zhaode Wang and Chengfei Lv and Shengyu Zhang and Fan Wu and Fei Wu},\n title={MadaKV: Adaptive Modality-Perception KV Cache Eviction for Efficient Multimodal Long-Context Inference},\n year={2025},\n cdate={1735689600000},\n pages={13306-13318},\n url={https://aclanthology.org/2025.acl-long.652/},\n booktitle={ACL (1)},\n crossref={conf/acl/2025-1}\n}\n"},"abstract":{"value":"This paper introduces MadaKV, a modality-adaptive key-value (KV) cache eviction strategy designed to enhance the efficiency of multimodal large language models (MLLMs) in long-context inference. In multimodal scenarios, attention heads exhibit varying preferences for different modalities, resulting in significant disparities in modality importance across attention heads. Traditional KV cache eviction methods, which are tailored for unimodal settings, fail to capture modality-specific information, thereby yielding suboptimal performance. MadaKV addresses these challenges through two key components: modality preference adaptation and hierarchical compression compensation. By dynamically sensing modality information within attention heads and adaptively retaining critical tokens, MadaKV achieves substantial reductions in KV cache memory footprint and model inference decoding latency (1.3 to 1.5 times improvement) while maintaining high accuracy across various multimodal long-context tasks. Extensive experiments on representative MLLMs and the MileBench benchmark demonstrate the effectiveness of MadaKV compared to existing KV cache eviction methods."},"title":{"value":"MadaKV: Adaptive Modality-Perception KV Cache Eviction for Efficient Multimodal Long-Context Inference"},"authors":{"value":["Kunxi Li","Zhonghua Jiang","Zhouzhou Shen","Zhaode Wang","Chengfei Lv","Shengyu Zhang","Fan Wu","Fei Wu"]}},"tmdate":1769419448039,"pdate":1735689600000,"externalIds":["dblp:conf/acl/Li0SWLZ0025"],"tcdate":1768177914843,"writers":["~"],"signatures":["~Shengyu_Zhang2"],"forum":"uZAqxb7Iad","license":"CC BY-SA 4.0","number":743328,"cdate":1735689600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1769419448039,"domain":"DBLP.org","id":"uZAqxb7Iad","version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2506.15724v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"li|madakv_adaptive_modalityperception_kv_cache_eviction_for_efficient_multimodal_longcontext_inference"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Kunxi_Li:","https://dblp.org/search/pid/api?q=author:Zhonghua_Jiang_0007:","~Zhouzhou_Shen1","https://dblp.org/search/pid/api?q=author:Zhaode_Wang:","https://dblp.org/search/pid/api?q=author:Chengfei_Lv:","~Shengyu_Zhang2","~Fan_Wu10","https://dblp.org/search/pid/api?q=author:Fei_Wu_0001:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2506.15724"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2506-15724,\n publtype={informal},\n author={Kunxi Li and Zhonghua Jiang and Zhouzhou Shen and Zhaode Wang and Chengfei Lv and Shengyu Zhang and Fan Wu and Fei Wu},\n title={MadaKV: Adaptive Modality-Perception KV Cache Eviction for Efficient Multimodal Long-Context Inference},\n year={2025},\n month={June},\n cdate={1748736000000},\n journal={CoRR},\n volume={abs/2506.15724},\n url={https://doi.org/10.48550/arXiv.2506.15724}\n}\n"},"abstract":{"value":"This paper introduces MadaKV, a modality-adaptive key-value (KV) cache eviction strategy designed to enhance the efficiency of multimodal large language models (MLLMs) in long-context inference. In multimodal scenarios, attention heads exhibit varying preferences for different modalities, resulting in significant disparities in modality importance across attention heads. Traditional KV cache eviction methods, which are tailored for unimodal settings, fail to capture modality-specific information, thereby yielding suboptimal performance. MadaKV addresses these challenges through two key components: modality preference adaptation and hierarchical compression compensation. By dynamically sensing modality information within attention heads and adaptively retaining critical tokens, MadaKV achieves substantial reductions in KV cache memory footprint and model inference decoding latency (1.3 to 1.5 times improvement) while maintaining high accuracy across various multimodal long-context tasks. Extensive experiments on representative MLLMs and the MileBench benchmark demonstrate the effectiveness of MadaKV compared to existing KV cache eviction methods."},"title":{"value":"MadaKV: Adaptive Modality-Perception KV Cache Eviction for Efficient Multimodal Long-Context Inference"},"authors":{"value":["Kunxi Li","Zhonghua Jiang","Zhouzhou Shen","Zhaode Wang","Chengfei Lv","Shengyu Zhang","Fan Wu","Fei Wu"]}},"tmdate":1769419447764,"pdate":1735689600000,"externalIds":["dblp:journals/corr/abs-2506-15724"],"tcdate":1768177917416,"writers":["~"],"signatures":["~Shengyu_Zhang2"],"forum":"vdIuluLoSo","license":"CC BY-SA 4.0","number":743356,"cdate":1748736000000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1769419447764,"domain":"DBLP.org","id":"vdIuluLoSo","version":2},{"content":{"venue":{"value":"ICLR 2027 Conference Desk Rejected Submission"},"keywords":{"value":["KV Cache Compression","Transformers","Meta-Learning"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Autoregressive decoding relies on the KV cache, whose memory footprint grows linearly with context length and caps batch size and throughput. Existing post-training methods exploit redundancy across tokens, layers, and feature dimensions, or lower numerical precision. We identify a complementary axis: within each attention head, values are largely predictable from their corresponding keys. We exploit this with value cache amortisation, which fits a lightweight per-head predictor at test time and stores residuals only for the tokens it reconstructs worst, giving $4\\times$ near-lossless value compression. The predictor can be initialised by meta-learning for negligible prefill overhead, or fitted from scratch in settings that allow a one-off cost, such as multi-query inference. We pair value amortisation with cross-layer low-rank key compression to produce LowRAM, an asymmetric compression pipeline that stays close to lossless across three model families. By reconstructing only a selected subset of KV pairs at each decoding step, LowRAM can keep decoding overhead minimal and achieve up to $5\\times$ higher peak decode throughput at $4\\times$ compression. LowRAM further composes with quantisation and eviction, reaching $26.9\\times$ compression at $94.5\\%$ of baseline performance on long contexts. These results show that treating keys and values as structurally distinct enables efficient, aggressive, and composable KV cache compression."},"_bibtex":{"value":"@misc{\nanonymous2026lowram,\ntitle={Low{RAM}: Low-Rank Keys and Amortised Values for Composable {KV} Cache Compression at Test-Time},\nauthor={Anonymous},\nyear={2026},\nurl={https://openreview.net/forum?id=BsMmyNjElt}\n}"},"title":{"value":"LowRAM: Low-Rank Keys and Amortised Values for Composable KV Cache Compression at Test-Time"},"pdf":{"value":"/pdf/2e72fc8c4029ddbbeac193371ef56847ec7876c8.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Desk_Rejected_Submission"}},"tmdate":1791218491853,"tcdate":1789772406162,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission50013/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission50013/Authors"],"forum":"BsMmyNjElt","license":"CC BY 4.0","number":50013,"cdate":1789772406162,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/Submission50013/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/-/Desk_Rejected_Submission","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791218491853,"domain":"ICLR.cc/2027/Conference","id":"BsMmyNjElt","version":2},{"content":{"venue":{"value":"CoRR 2026"},"pdf":{"value":"https://arxiv.org/pdf/2608.00902v1"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"liu|practical_online_kv_cache_compaction_for_llm_agents_an_empirical_study"},"html":{"value":"https://doi.org/10.48550/arXiv.2608.00902"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2608-00902,\n publtype={informal},\n author={Yujian Liu and Jiabao Ji and Li An and Rohit Jain and Gungor Polatkan and Siyu Zhu and Shiyu Chang},\n title={Practical Online KV Cache Compaction for LLM Agents: An Empirical Study},\n year={2026},\n month={August},\n cdate={1785542400000},\n journal={CoRR},\n volume={abs/2608.00902},\n url={https://doi.org/10.48550/arXiv.2608.00902}\n}\n"},"abstract":{"value":"LLM agents accumulate long trajectories of reasoning steps, tool calls, and environment feedback, making the KV cache a major inference bottleneck. KV cache compaction can reduce this cost, but most prior methods assume a static context where future queries are known or can be approximated offline. Agents instead require online compaction: new information must be compressed before future relevance is known, using proxy queries cheap enough for the inference path. We study online compaction across token eviction (TE) and attention matching (AM), adapting both to compact agent turns and comparing cheap proxy sources such as boundary, repeat-prefill, and delayed future-generation queries. Experiments on BrowseComp-Plus and WideSearch show that immediate compaction often hurts performance, whereas delaying compaction to use the agent's future queries recovers much of the gap. Moreover, TE is often more robust than AM under imperfect proxies. Across models at different scales, TE preserves most of the accuracy while reducing KV cache by 80%, and can improve throughput over the no compaction baseline. These results position proxy-query selection as a core design choice for practical online KV compaction."},"title":{"value":"Practical Online KV Cache Compaction for LLM Agents: An Empirical Study"},"authors":{"value":[{"fullname":"Yujian Liu","username":"~Yujian_Liu1"},{"fullname":"Jiabao Ji","username":""},{"fullname":"Li An","username":""},{"fullname":"Rohit Jain","username":"~Rohit_Jain2"},{"fullname":"Gungor Polatkan","username":""},{"fullname":"Siyu Zhu","username":""},{"fullname":"Shiyu Chang","username":""}]}},"tmdate":1792054353718,"pdate":1798675200000,"externalIds":["dblp:journals/corr/abs-2608-00902"],"tcdate":1792039052373,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Yujian_Liu1"],"forum":"J3b1S0IMWT","license":"CC BY-SA 4.0","number":186658,"cdate":1785542400000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit","OpenReview.net/Public_Article/-/Authorship_Claim"],"mdate":1792054353718,"domain":"OpenReview.net/Public_Article","id":"J3b1S0IMWT","version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2508.08438v2"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"chu|selective_kvcache_sharing_to_mitigate_timing_sidechannels_in_llm_inference"},"authorids":{"value":["~Kexin_Chu1","https://dblp.org/search/pid/api?q=author:Zecheng_Lin:","~Dawei_Xiang1","https://dblp.org/search/pid/api?q=author:Zixu_Shen:","~Jianchang_Su1","https://dblp.org/search/pid/api?q=author:Cheng_Chu:","https://dblp.org/search/pid/api?q=author:Yiwei_Yang_0002:","https://dblp.org/search/pid/api?q=author:Wenhui_Zhang:","https://dblp.org/search/pid/api?q=author:Wenfei_Wu:","https://dblp.org/search/pid/api?q=author:Wei_Zhang:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2508.08438"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2508-08438,\n publtype={informal},\n author={Kexin Chu and Zecheng Lin and Dawei Xiang and Zixu Shen and Jianchang Su and Cheng Chu and Yiwei Yang and Wenhui Zhang and Wenfei Wu and Wei Zhang},\n title={Selective KV-Cache Sharing to Mitigate Timing Side-Channels in LLM Inference},\n year={2025},\n month={August},\n cdate={1754006400000},\n journal={CoRR},\n volume={abs/2508.08438},\n url={https://doi.org/10.48550/arXiv.2508.08438}\n}\n"},"abstract":{"value":"Global KV-cache sharing is an effective optimization for accelerating large language model (LLM) inference, yet it introduces an API-visible timing side channel that lets adversaries infer sensitive user inputs from shared entries, leading to cross-tenant privacy risks. To address this problem, we introduce SafeKV (Secure and Flexible KV-cache Sharing), a system-level co-design of privacy enforcement and KV-cache management. SafeKV integrates lightweight detection and isolation directly into the serving runtime to eliminate cross-tenant reuse of sensitive KV-cache blocks under our threat model, while recovering most of the performance benefits of global sharing. Our key contributions are: (1) a three-tier asynchronous detection pipeline that decouples privacy classification from inference and supports streaming workloads, (2) a unified radix-tree-based memory manager with path compression and sensitivity-aware eviction for scalable selective isolation, and (3) an RDR-guided (Reuse Diversity Ratio) runtime safeguard that detects and bounds residual leakage. On large LLM backends, SafeKV reduces the time-to-first-token (TTFT) overhead compared to full isolation by up to 40.58% and raises throughput by up to 2.66x. Overall, SafeKV restores the efficiency of KV reuse while enforcing strong, practical privacy for multi-tenant LLM inference."},"title":{"value":"Selective KV-Cache Sharing to Mitigate Timing Side-Channels in LLM Inference"},"authors":{"value":["Kexin Chu","Zecheng Lin","Dawei Xiang","Zixu Shen","Jianchang Su","Cheng Chu","Yiwei Yang","Wenhui Zhang","Wenfei Wu","Wei Zhang"]}},"tmdate":1779775978725,"pdate":1767139200000,"externalIds":["dblp:journals/corr/abs-2508-08438"],"tcdate":1771302817968,"writers":["~"],"signatures":["~Dawei_Xiang2"],"forum":"6rn1Mkam7e","license":"CC BY-SA 4.0","number":825012,"cdate":1754006400000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1779775978725,"domain":"DBLP.org","id":"6rn1Mkam7e","version":2},{"content":{"TLDR":{"value":"We identify KV-cache error accumulation as a non-negligible source of accuracy loss in quantized LLMs for long-horizon tasks and propose iKVC, a novel inference method that periodically corrects the KV cache to balance accuracy and throughput."},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["LLM","Quantization","KV cache"]},"primary_area":{"value":"generative models"},"abstract":{"value":"Post-training quantization (PTQ) reduces the memory footprint of large language models (LLMs) and can improve inference throughput. \nHowever, the resulting quantization errors can significantly degrade accuracy, creating a fundamental trade-off between the efficiency gained from quantization and the accuracy retained from the full-precision model, particularly in long-horizon generation.\nPrior work mitigates this degradation by preserving the KV cache of selected prompt tokens at full precision, thereby reducing errors introduced during prefill. \nThis strategy, however, does not address errors introduced during decoding, as quantized weights continuously perturb the key and value representations of newly generated tokens, and these errors accumulate in the KV cache, altering subsequent predictions. \nWe empirically observe that this error accumulation becomes non-negligible as a large number of tokens are generated, and we theoretically derive that the accumulated KV-cache errors can cause the attention output to deviate from the values computed by the original full-precision model.\nMotivated by this finding, we propose inference-time KV-cache Correction (iKVC), a training-free inference method that periodically corrects the KV cache for both prompt and generated tokens before decoding errors substantially accumulate. \nThe correction interval is configurable, allowing iKVC to explicitly control the accuracy--throughput trade-off. \nAcross experiments, iKVC consistently improves the trade-off over PTQ alone in long-horizon tasks. \nOur results identify KV-cache error accumulation as an important, previously underexplored factor in achieving accurate and efficient long-horizon tasks with quantized LLMs, suggesting that managing such errors during inference can complement PTQ techniques."},"_bibtex":{"value":"@inproceedings{\nanonymous2026ikvc,\ntitle={i{KVC}: Inference-Time {KV}-Cache Correction for Quantized {LLM}s in Long-Horizon Tasks},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=bhD9D7FTiM},\nnote={under review}\n}"},"title":{"value":"iKVC: Inference-Time KV-Cache Correction for Quantized LLMs in Long-Horizon Tasks"},"pdf":{"value":"/pdf/dc521d84b438494b9a5492163b702aa5a2ddfbad.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791231371536,"tcdate":1789648970529,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission31362/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission31362/Authors"],"forum":"bhD9D7FTiM","license":"CC BY 4.0","number":31362,"cdate":1789648970529,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Edit","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission31362/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing"],"mdate":1791231371536,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"bhD9D7FTiM","version":2},{"content":{"TLDR":{"value":"L2 gradient-norm is the only single-pass proxy that ranks KV layers by 2-bit damage; it beats KVTuner search at 3.25-bit and matches or beats it at 3.5-bit, at 100-1000x lower cost. Wide candidate sets make greedy water-filling worse than uniform."},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["KV cache","quantization","bit allocation","KVTuner","gradient sensitivity","mixed precision"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Layer-wise mixed-precision KV-cache quantization needs a per-layer sensitivity signal to decide where to spend bits. Search-based methods such as KVTuner measure damage for every (layer, bit-width) pair on calibration data; proxy-based methods replace the search with a cheap statistic. We evaluate three single-pass proxies—attention mass on outlier positions, key-quantization MSE, and the L2 norm of gradients with respect to the K/V projections—inside one matched harness on Phi-4-mini-instruct (3.8B) and Llama-3.2-1B, against our own re-implementation of the KVTuner sensitivity search as the published baseline. The gradient proxy is the only signal whose layer ranking correlates with measured 2-bit damage (Spearman ρ = 0.60 and 0.76 on the two backbones, vs. ≤0.21 for the others), and its allocations beat both activation-based proxies and the search-based allocation at the aggressive 3.25-bit budget, and beat the search on Phi and match it on Llama at 3.5 bits—with the advantage growing as the budget tightens—at two to three orders of magnitude lower calibration cost. This confirms, head-to-head and at matched allocation algorithm, recent rate-distortion analyses that claim gradient sensitivity dominates activation-based heuristics. We further document a failure mode of greedy water-filling on measured damage: when the candidate bit set extends above the budget (e.g. 8-bit candidates at a 4.0-bit budget), the allocator promotes insensitive layers to high precision and pushes sensitive layers into super-linearly damaging 2-bit territory, producing allocations worse than uniform quantization. A restricted-candidate control isolates the failure to the candidate-set width rather than the damage estimates themselves."},"_bibtex":{"value":"@inproceedings{\nanonymous2026gradient,\ntitle={Gradient Norms, Not Attention Mass: Cheap Proxies for Layer-Wise {KV}-Cache Bit Allocation},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=G3Bjxzn3w6},\nnote={under review}\n}"},"title":{"value":"Gradient Norms, Not Attention Mass: Cheap Proxies for Layer-Wise KV-Cache Bit Allocation"},"pdf":{"value":"/pdf/e7eee51290ba50e27221b4a3d3893f5eea46a561.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791230985323,"tcdate":1789626752039,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission28959/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission28959/Authors"],"forum":"G3Bjxzn3w6","license":"CC BY 4.0","number":28959,"cdate":1789626752039,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/Submission28959/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791230985323,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"G3Bjxzn3w6","version":2},{"content":{"venue":{"value":"Proceedings of Machine Learning and Systems"},"venueid":{"value":"OpenReview.net/Archive"},"paperhash":{"value":"adnan|keyformer_kv_cache_reduction_through_key_tokens_selection_for_efficient_generative_inference"},"authorids":{"value":["~Muhammad_Adnan1","~Akhil_Arunkumar1","~Gaurav_Jain5","~Prashant_J._Nair1","~Ilya_Soloveychik1","~Purushotham_Kamath1"]},"html":{"value":"https://proceedings.mlsys.org/paper_files/paper/2024/file/48fecef47b19fe501d27d338b6d52582-Paper-Conference.pdf"},"abstract":{"value":"Transformers have emerged as the underpinning architecture for Large Language Models (LLMs). In generative\nlanguage models, the inference process involves two primary phases: prompt processing and token generation.\nToken generation, which constitutes the majority of the computational workload, primarily entails vector-matrix\nmultiplications and interactions with the Key-Value (KV) Cache. This phase is constrained by memory bandwidth\ndue to the overhead of transferring weights and KV cache values from the memory system to the computing units.\nThis memory bottleneck becomes particularly pronounced in applications that require long-context and extensive\ntext generation, both of which are increasingly crucial for LLMs.\nThis paper introduces “Keyformer”, an innovative inference-time approach, to mitigate the challenges associated\nwith KV cache size. Keyformer leverages the observation that approximately 90% of the attention weight in\ngenerative inference focuses on a specific subset of tokens, referred to as “key” tokens. Keyformer retains only\nthe key tokens in the KV cache by identifying these crucial tokens using a novel score function. This approach\nreduces both the KV cache size and memory bandwidth usage without compromising model accuracy. We\nevaluate Keyformer’s performance across three foundational models: GPT-J, Cerebras-GPT, and MPT, which\nemploy various positional embedding algorithms. Our assessment uses a variety of tasks, with an emphasis on\nsummarization and conversation tasks involving extended contexts. We show that Keyformer reduces inference\nlatency by 2.1× and improves token generation throughput by 2.4×, while preserving the model’s accuracy"},"title":{"value":"Keyformer: KV Cache reduction through key tokens selection for Efficient Generative Inference"},"authors":{"value":["Muhammad Adnan","Akhil Arunkumar","Gaurav Jain","Prashant J. Nair","Ilya Soloveychik","Purushotham Kamath"]}},"tmdate":1777934575604,"pdate":1627801200000,"tcdate":1777934575604,"writers":["~Muhammad_Adnan1","~Akhil_Arunkumar1","~Gaurav_Jain5","~Prashant_J._Nair1","~Ilya_Soloveychik1","~Purushotham_Kamath1"],"signatures":["~Muhammad_Adnan1"],"forum":"06Lor2RT3d","license":"CC BY-NC 4.0","number":47916,"cdate":1777934575604,"readers":["everyone"],"invitations":["OpenReview.net/Archive/-/Direct_Upload"],"mdate":1777934575604,"domain":"OpenReview.net/Archive","id":"06Lor2RT3d","version":2},{"content":{"venue":{"value":"ICML 2026 regular"},"keywords":{"value":["Efficient LLM","KV Cache Eviction","Long Context","Inference Acceleration","Attention Head Heterogeneity","Long-horizon Marginal Utility"]},"_bibtex":{"value":"@inproceedings{\ntang2026predicting,\ntitle={Predicting Future Utility: Global Combinatorial Optimization for Task-Agnostic {KV} Cache Eviction},\nauthor={Ziyao Tang and Pengkun Jiao and Xinhang Chen and Wei Liu and Shiyong Li and Jingjing Chen},\nbooktitle={Forty-third International Conference on Machine Learning},\nyear={2026},\nurl={https://openreview.net/forum?id=FQLxcBsKIb}\n}"},"title":{"value":"Predicting Future Utility: Global Combinatorial Optimization for Task-Agnostic KV Cache Eviction"},"paperhash":{"value":"tang|predicting_future_utility_global_combinatorial_optimization_for_taskagnostic_kv_cache_eviction"},"originally_submitted_PDF":{"value":"/pdf/fb2f34e9e926fb6936c7caf66b4db0dd12e74f8c.pdf"},"primary_area":{"value":"deep_learning->large_language_models"},"abstract":{"value":"Given the quadratic complexity of attention, KV cache eviction is vital to accelerate model inference.\nCurrent KV cache eviction methods typically rely on instantaneous heuristic metrics, implicitly assuming that score magnitudes are consistent proxies for importance across all heads. \nHowever, this overlooks the heterogeneity in predictive fidelity across attention heads. While certain heads prioritize the instantaneous contribution of tokens, others are dedicated to capturing long-horizon utility.\nIn this paper, we propose that optimal budget allocation should be governed by the marginal utility in preserving long-term semantic information. \nBuilding on this insight, we propose LU-KV, a novel framework that formulates head-level budget allocation as a global combinatorial optimization problem to maximize the long-horizon marginal contribution of reserved tokens. To solve this non-convex problem, we employ a convex-hull relaxation and a marginal-utility-based greedy solver, achieving near-optimal solutions. Furthermore, we implement a data-driven offline profiling protocol to facilitate the practical deployment of LU-KV.\nEvaluations on LongBench and RULER benchmarks demonstrate that LU-KV reduces KV cache size by 80\\% with minimal performance degradation, while also decreasing inference latency and GPU memory footprint."},"link_to_code":{"value":"https://github.com/baidu-baige/LU-KV"},"pdf":{"value":"/pdf/bc459e75e628ef8aa84e2c2228bc57e10a04424e.pdf"},"lay_summary":{"value":"Modern AI assistants, such as chatbots and document-analysis tools, need to process and remember long conversations or documents. To do this efficiently, they store intermediate information called a \"KV cache\" — essentially a memory bank that helps the model recall what it has already read. However, this memory grows rapidly with text length, consuming enormous amounts of computing resources and slowing down responses.\n\nTo address this, researchers have developed methods that selectively discard less important memories. Current approaches treat all parts of the AI model equally, using simple rules of thumb — such as \"keep whatever seemed most relevant just now\" — to decide what to forget. This works reasonably well, but ignores the fact that different components of the model play very different roles: some focus on what is immediately useful, while others track information that only becomes important much later in a conversation.\n\nIn this paper, we introduce LU-KV, a smarter memory management system that respects these differences. Rather than applying the same deletion rule everywhere, LU-KV asks: which memories, if kept, will provide the greatest benefit over the long run? It allocates more storage budget to the model components that handle long-range information, and less to those that only need short-term context — much like prioritizing a long-term planner over a short-term to-do list.\n\nWe show that LU-KV can reduce KV cache by 80% while preserving the quality of the model's responses, making AI assistants faster, cheaper to run, and more accessible — without sacrificing their ability to understand long documents."},"venueid":{"value":"ICML.cc/2026/Conference"},"authorids":{"value":["~Ziyao_Tang3","~Pengkun_Jiao1","~Xinhang_Chen1","~Wei_Liu123","~Shiyong_Li3","~Jingjing_Chen3"]},"authors":{"value":["Ziyao Tang","Pengkun Jiao","Xinhang Chen","Wei Liu","Shiyong Li","Jingjing Chen"]}},"tmdate":1790067317456,"pdate":1777576284184,"tcdate":1768802082447,"writers":["ICML.cc/2026/Conference","ICML.cc/2026/Conference/Submission7637/Authors"],"signatures":["ICML.cc/2026/Conference/Submission7637/Authors"],"forum":"FQLxcBsKIb","license":"CC BY 4.0","number":7637,"cdate":1768802082447,"readers":["everyone"],"invitations":["ICML.cc/2026/Conference/-/Submission","ICML.cc/2026/Conference/-/Post_Submission","ICML.cc/2026/Conference/Submission7637/-/Full_Submission","ICML.cc/2026/Conference/Submission7637/-/Reciprocal_Reviewing_Correction","ICML.cc/2026/Conference/-/Edit","ICML.cc/2026/Conference/Submission7637/-/Camera_Ready_Revision"],"mdate":1790067317456,"odate":1782341927026,"domain":"ICML.cc/2026/Conference","id":"FQLxcBsKIb","version":2},{"content":{"research_area_keywords":{"value":"pruning; quantization"},"languages_studied":{"value":"English"},"venue":{"value":"ACL ARR 2025 May Submission"},"_bibtex":{"value":"@inproceedings{\nanonymous2025more,\ntitle={More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in {KV} Cache Compression},\nauthor={Anonymous},\nbooktitle={Submitted to ACL Rolling Review - May 2025},\nyear={2025},\nurl={https://openreview.net/forum?id=j9wiK7iPk0},\nnote={under review}\n}"},"title":{"value":"More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression"},"contribution_types":{"value":["NLP engineering experiment","Approaches low compute settings-efficiency","Data analysis"]},"abstract":{"value":"As large language models (LLMs) process increasing context windows, the memory usage of KV cache has become a critical bottleneck during inference. The mainstream KV compression methods, including KV pruning and KV quantization, primarily focus on either token or precision dimension separately. However, these works leaving the trade-off between these two orthogonal dimensions largely under-explored. \nIn this paper, we leverage the Information Bottleneck principle to formulate KV cache compression within a unified theoretical framework. We demonstrate that a carefully managed token-precision trade-off can achieve an optimal point within the Information Bottleneck compared to standalone KV pruning or KV quantization. Experiments reveal that storing more tokens in the KV cache at lower precision—a strategy we term quantized pruning—can significantly enhance the long-context performance of LLMs. An in-depth analysis of this token-precision trade-off across key aspects shows that quantized pruning achieves substantial improvements in retrieval-related tasks and consistently performs well across varying input lengths. Furthermore, quantized pruning exhibits notable stability and effectiveness across different KV pruning methods, quantization strategies, and model scales. These findings offer valuable insights into optimizing KV cache compression through balanced token-precision trade-off strategies."},"paper_type":{"value":"Long"},"pdf":{"value":"/pdf/fb207fa86f12f34eeec445e9f562da0be5937bb3.pdf"},"research_area":{"value":"Efficient/Low-Resource Methods for NLP"},"venueid":{"value":"aclweb.org/ACL/ARR/2025/May/Submission"}},"tmdate":1782962249072,"tcdate":1747741606776,"writers":["aclweb.org/ACL/ARR/2025/May","aclweb.org/ACL/ARR/2025/May/Submission7681/Authors"],"signatures":["aclweb.org/ACL/ARR/2025/May/Submission7681/Authors"],"forum":"j9wiK7iPk0","license":"CC BY 4.0","number":7681,"cdate":1747741606776,"readers":["everyone"],"invitations":["aclweb.org/ACL/ARR/2025/May/-/Submission","aclweb.org/ACL/ARR/2025/May/-/Post_Submission","aclweb.org/ACL/ARR/2025/May/-/Edit","aclweb.org/ACL/ARR/2025/May/-/Desk_Reject_Verification_Flag","aclweb.org/ACL/ARR/2025/May/-/Preprint_Release_Submission","aclweb.org/ACL/ARR/2025/May/-/Preprint_Post_Submission"],"mdate":1782962249072,"odate":1753766020949,"domain":"aclweb.org/ACL/ARR/2025/May","id":"j9wiK7iPk0","version":2},{"content":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["LLM","KV cache compression","CGE","RGL"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Large Language Models (LLMs) have demonstrated exceptional capabilities in integrating contextual knowledge, but their deployment is often constrained by the substantial computational resources required for long text sequences. To mitigate the inference time cost associated with attention mechanisms, LLMs utilize key-value embedding caching techniques (KV cache), which introduce significant storage pressure. In this paper, we propose IntelLLM, a novel and efficient approach to KV cache compression that strikes a balance between compression rate and performance. Drawing inspiration from sparse attention mechanism, we observe that only a small subset of tokens in lengthy texts capture the majority of attention weights. This sparsity, intrinsic to the attention mechanism, serves as the foundation for improving the KV compression ratio through a strategic eviction method. IntelLLM is composed of center of gravity eviction (CGE) strategy and remote gap localization (RGL) strategy. CGE is designed to address the potential loss of important semantic dependencies when evicting high-sparsity tokens, which prioritizes the retention of key tokens by shielding the center of gravity of attention during inference, thereby preserving critical information and optimizing the efficiency of attention computation. Additionally, RGL is proposed to leverage implicit positional features to maintain long-range dependencies, inspired by advancements in location encoding research. Our KV compression approach integrates seamlessly with existing LLMs, requiring minimal code modifications without the need for fine-tuning or model parameter changes. IntelLLM not only significantly reduces the storage requirements for KV cache but also consistently outperforms full KV models in long text processing tasks, while utilizing only 50% of the typical KV cache expenses."},"_bibtex":{"value":"@misc{\nli2024intelllm,\ntitle={Intel{LLM}: Little Hints Make a Big Difference for {LLM} {KV} Cache Compression},\nauthor={TingLong Li and Qiuyu Shao},\nyear={2024},\nurl={https://openreview.net/forum?id=4QWPCTLq20}\n}"},"title":{"value":"IntelLLM: Little Hints Make a Big Difference for LLM KV Cache Compression"},"pdf":{"value":"/pdf/e70bdc24592a789f5a144d6ea6d61377169446db.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"li|intelllm_little_hints_make_a_big_difference_for_llm_kv_cache_compression"},"authorids":{"value":["~TingLong_Li1","~Qiuyu_Shao1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["TingLong Li","Qiuyu Shao"]}},"tmdate":1732608627190,"tcdate":1727426433448,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission9602/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission9602/Authors"],"forum":"4QWPCTLq20","license":"CC BY 4.0","number":9602,"cdate":1727426433448,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/-/Submission","ICLR.cc/2025/Conference/-/Post_Submission","ICLR.cc/2025/Conference/Submission9602/-/Full_Submission","ICLR.cc/2025/Conference/-/Withdrawn_Submission"],"mdate":1732608627190,"odate":1728008565725,"domain":"ICLR.cc/2025/Conference","id":"4QWPCTLq20","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2410.12876v2"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"zeng|incontext_kvcache_eviction_for_llms_via_attentiongate"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Zihao_Zeng:","~Bokai_Lin1","https://dblp.org/search/pid/api?q=author:Tianqi_Hou:","https://dblp.org/search/pid/api?q=author:Hao_Zhang:","~Zhijie_Deng1"]},"html":{"value":"https://doi.org/10.48550/arXiv.2410.12876"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2410-12876,\n publtype={informal},\n author={Zihao Zeng and Bokai Lin and Tianqi Hou and Hao Zhang and Zhijie Deng},\n title={In-context KV-Cache Eviction for LLMs via Attention-Gate},\n year={2024},\n cdate={1704067200000},\n journal={CoRR},\n volume={abs/2410.12876},\n url={https://doi.org/10.48550/arXiv.2410.12876}\n}\n"},"abstract":{"value":"The KV-Cache technique has become the standard for the inference of large language models (LLMs). It caches states of self-attention to avoid recomputation. Yet, it is widely criticized that KV-Cache can become a bottleneck of the LLM inference system, especially when confronted with ultra-large models and long-context queries. A natural remedy is to discard the KV-Cache for less important tokens, with StreamingLLM as an example, but the used static eviction strategies cannot flexibly adapt to varying contexts. Remedies like H2O leverage accumulative attention scores to perform dynamic eviction but suffer from the attention bias issue in capturing contextual information. This paper bridges this gap by devising a parameterized KV-Cache eviction mechanism, dubbed as Attention-Gate, which accepts the whole context as input and yields eviction flags for each token to realize in-context eviction. The subsequent self-attention module proceeds according to the flags and only the KV states for the remaining tokens need to be cached. The Attention-Gates can vary among different heads and layers and be trivially plugged into pre-trained LLMs, tuned by cost-effective continual pre-training or supervised fine-tuning objectives to acquire what to discard. The computational and memory overhead introduced by Attention-Gates is minimal. Our method is validated across multiple tasks, demonstrating both efficiency and adaptability. After a highly efficient continual pre-training, it achieves higher average accuracy and evicts more tokens compared to traditional training-free methods. In supervised fine-tuning, it not only evicts many tokens but also outperforms LoRA-finetuned LLMs on some datasets, such as RTE, where it improves accuracy by 13.9% while evicting 62.8% of tokens, showing that effective eviction of redundant tokens can even enhance performance."},"title":{"value":"In-context KV-Cache Eviction for LLMs via Attention-Gate"},"authors":{"value":["Zihao Zeng","Bokai Lin","Tianqi Hou","Hao Zhang","Zhijie Deng"]}},"tmdate":1759667249997,"pdate":1704067200000,"tcdate":1739939562690,"writers":["~"],"signatures":["~Zhijie_Deng1"],"forum":"9T23spFddZ","license":"CC BY-SA 4.0","number":326017,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1759667249997,"domain":"DBLP.org","id":"9T23spFddZ","version":2},{"content":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["KV-Cache eviction","Attention-Gate","LLM inference acceleration"]},"supplementary_material":{"value":"/attachment/147f28fa623cf49cbf5091a5ce45aec09d8eaa7a.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The KV-Cache technique has become the standard for the inference of large language models (LLMs).\nIt caches states of self-attention to avoid recomputation. \nYet, it is widely criticized that KV-Cache can become a bottleneck of the LLM inference system, especially when confronted with ultra-large models and long-context queries. \nA natural remedy is to discard the KV-Cache for less important tokens, with StreamingLLM as an example, but the used static eviction strategies cannot flexibly adapt to varying contexts. \nRemedies like H2O leverage accumulative attention scores to perform dynamic eviction but suffer from the attention bias issue in capturing contextual information. \nThis paper bridges this gap by devising a parameterized KV-Cache eviction mechanism, dubbed as *Attention-Gate*, which accepts the whole context as input and yields eviction flags for each token to realize *in-context* eviction. \nThe subsequent self-attention module proceeds according to the flags and only the KV states for the remaining tokens need to be cached. \nThe Attention-Gates can vary among different heads and layers and be trivially plugged into pre-trained LLMs, tuned by cost-effective continual pre-training or supervised fine-tuning objectives to acquire what to discard. \nThe computational and memory overhead introduced by Attention-Gates is minimal.\nOur method is validated across multiple tasks, demonstrating both efficiency and adaptability.\nAfter a highly efficient continual pre-training, it achieves higher average accuracy and evicts more tokens compared to traditional training-free methods. \nIn supervised fine-tuning, it not only evicts many tokens but also outperforms LoRA-finetuned LLMs on some datasets, such as RTE, where it improves accuracy by 13.9% while evicting 62.8% of tokens, showing that effective eviction of redundant tokens can even enhance performance."},"_bibtex":{"value":"@misc{\nzeng2025incontext,\ntitle={In-context {KV}-Cache Eviction for {LLM}s via Attention-Gate},\nauthor={Zihao Zeng and Bokai Lin and TianQi Hou and Hao Zhang and Zhijie Deng},\nyear={2025},\nurl={https://openreview.net/forum?id=tvQNysCP7C}\n}"},"title":{"value":"In-context KV-Cache Eviction for LLMs via Attention-Gate"},"pdf":{"value":"/pdf/28994ecde81dc28a4b4f92441562ed2f2f02c0bb.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"zeng|incontext_kvcache_eviction_for_llms_via_attentiongate"},"authorids":{"value":["~Zihao_Zeng1","~Bokai_Lin1","~TianQi_Hou1","~Hao_Zhang2","~Zhijie_Deng1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Zihao Zeng","Bokai Lin","TianQi Hou","Hao Zhang","Zhijie Deng"]}},"tmdate":1738735782166,"tcdate":1727410844375,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission8951/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission8951/Authors"],"forum":"tvQNysCP7C","license":"CC BY 4.0","number":8951,"cdate":1727410844375,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/-/Submission","ICLR.cc/2025/Conference/-/Post_Submission","ICLR.cc/2025/Conference/Submission8951/-/Full_Submission","ICLR.cc/2025/Conference/Submission8951/-/Rebuttal_Revision","ICLR.cc/2025/Conference/-/Edit"],"mdate":1738735782166,"odate":1728008565725,"domain":"ICLR.cc/2025/Conference","id":"tvQNysCP7C","version":2},{"content":{"research_area_keywords":{"value":"long-context model, efficiency, Key-Value cache"},"languages_studied":{"value":"English"},"venue":{"value":"ACL ARR 2025 February Submission"},"_bibtex":{"value":"@inproceedings{\nanonymous2025refreshkv,\ntitle={Refresh{KV}: Updating Small {KV} Cache During Long-form Generation},\nauthor={Anonymous},\nbooktitle={Submitted to ACL Rolling Review - February 2025},\nyear={2025},\nurl={https://openreview.net/forum?id=bUuTW1DScu},\nnote={under review}\n}"},"title":{"value":"RefreshKV: Updating Small KV Cache During Long-form Generation"},"contribution_types":{"value":["Approaches low compute settings-efficiency"]},"abstract":{"value":"Generating long sequences of tokens given a long-context input is a very compute-intensive inference scenario for large language models (LLMs). One prominent inference speed-up approach is constructing a smaller key-value (KV) cache, relieving LLMs from computing attention over a long sequence of tokens. While such methods work well to generate short sequences, their performance degrades rapidly for long-form generation. Most KV compression happens once, prematurely removing tokens that can be useful later in the generation. We propose a new inference-time method, RefreshKV, that flexibly alternates between full context attention and attention over a subset of input tokens during generation. After each full attention step, we update the smaller KV cache based on the attention pattern over the entire input. Applying our method to off-the-shelf LLMs achieves comparable speedup to eviction-based methods while improving performance for various long-form generation tasks. Lastly, we show that continued pretraining with our inference setting brings further gains in performance."},"paper_type":{"value":"Long"},"pdf":{"value":"/pdf/2413059a2d8ed5a77c44125b4647a89c648ebcfc.pdf"},"research_area":{"value":"Efficient/Low-Resource Methods for NLP"},"venueid":{"value":"aclweb.org/ACL/ARR/2025/February/Submission"}},"tmdate":1782962039683,"tcdate":1739672909011,"writers":["aclweb.org/ACL/ARR/2025/February","aclweb.org/ACL/ARR/2025/February/Submission4972/Authors"],"signatures":["aclweb.org/ACL/ARR/2025/February/Submission4972/Authors"],"forum":"bUuTW1DScu","license":"CC BY 4.0","number":4972,"cdate":1739672909011,"readers":["everyone"],"invitations":["aclweb.org/ACL/ARR/2025/February/-/Submission","aclweb.org/ACL/ARR/2025/February/-/Edit","aclweb.org/ACL/ARR/2025/February/-/Post_Submission","aclweb.org/ACL/ARR/2025/February/-/Desk_Reject_Verification_Flag","aclweb.org/ACL/ARR/2025/February/-/Preprint_Release_Submission","aclweb.org/ACL/ARR/2025/February/-/Preprint_Post_Submission"],"mdate":1782962039683,"odate":1746821355472,"domain":"aclweb.org/ACL/ARR/2025/February","id":"bUuTW1DScu","version":2},{"content":{"data_release":{"value":"We authorize the release of our submission and author names to the public in the event of acceptance."},"TLDR":{"value":"We propose RepairKV, a runtime mechanism which restores previously evicted (offloaded) tokens to the active KV cache when recent context makes them relevant again."},"venue":{"value":"AdaptFM Poster"},"email_sharing":{"value":"We authorize the sharing of all author emails with Program Chairs."},"keywords":{"value":["KV cache","KV cache compression","LLM inference","long-context language models","multi-turn LLM","efficient inference","agentic AI","attention"]},"abstract":{"value":"As large language models (LLMs) support longer contexts and more capable agentic sessions, what matters in the accumulated context can change with each new query, tool output, or test result. At the same time, inference still depends on a limited active key-value (KV) cache, and many existing KV-cache compression methods decide which past tokens stay active from the current prefix. As a result, tokens that matter in later turns may already have been evicted or compressed away in earlier compression decisions. We argue the KV cache should be multi-tiered and governed by a two-way mechanism: both eviction and repair, not just eviction. We propose RepairKV, a runtime operator for multi-turn agentic inference that reevaluates which evicted KV rows are currently important and restores a budgeted subset from host memory back into the GPU KV cache. Controlled experiments show that RepairKV's restoration of evicted tokens improves retrieval accuracy, with the effect persisting across relevance changes and different initial eviction policies, and a preliminary diagnostic on open-source repositories shows the same pattern; on Qwen2.5-7B-Instruct at 32K context, RepairKV achieves 91.0% retrieval on a four-query needle-in-a-haystack task versus 24.5% for the matched no-repair baseline at the same active-cache budget, with only 96 promoted tokens."},"_bibtex":{"value":"@inproceedings{\nrusli2026cache,\ntitle={Cache You Later: Post-Compression {KV} Repair for Long-Context Agentic {LLM} Inference},\nauthor={Andrew Rusli and Shreyan Paliwal and Michael Jiao},\nbooktitle={AdaptFM: Resource-Adaptive Foundation Model Inference},\nyear={2026},\nurl={https://openreview.net/forum?id=LsrmZrp7tW}\n}"},"title":{"value":"Cache You Later: Post-Compression KV Repair for Long-Context Agentic LLM Inference"},"pdf":{"value":"/pdf/b409204ee8a70b5cd1e33e733d2eb3c77f591bc4.pdf"},"venueid":{"value":"ICML.cc/2026/Workshop/AdaptFM"},"paperhash":{"value":"rusli|cache_you_later_postcompression_kv_repair_for_longcontext_agentic_llm_inference"},"authorids":{"value":["~Andrew_Rusli1","~Shreyan_Paliwal1","~Michael_Jiao1"]},"authors":{"value":["Andrew Rusli","Shreyan Paliwal","Michael Jiao"]}},"tmdate":1781192755402,"pdate":1780353776095,"tcdate":1778294071778,"writers":["ICML.cc/2026/Workshop/AdaptFM","ICML.cc/2026/Workshop/AdaptFM/Submission171/Authors"],"signatures":["ICML.cc/2026/Workshop/AdaptFM/Submission171/Authors"],"forum":"LsrmZrp7tW","number":171,"license":"CC BY 4.0","cdate":1778294071778,"readers":["everyone"],"invitations":["ICML.cc/2026/Workshop/AdaptFM/-/Submission","ICML.cc/2026/Workshop/AdaptFM/-/Submission_Change_Before_Reviewing","ICML.cc/2026/Workshop/AdaptFM/-/Submission_Release","ICML.cc/2026/Workshop/AdaptFM/Submission171/-/Camera_Ready_Revision","ICML.cc/2026/Workshop/AdaptFM/-/Edit"],"mdate":1781192755402,"odate":1780353776095,"domain":"ICML.cc/2026/Workshop/AdaptFM","id":"LsrmZrp7tW","version":2},{"content":{"venue":{"value":"COLM 2026 ER Workshop"},"pdf":{"value":"/pdf/8bab34e33c8d0ef9a94ea369cb2ef873444aba64.pdf"},"keywords":{"value":["prompt engineering; inference acceleration"]},"venueid":{"value":"colmweb.org/COLM/2026/Workshop/Efficient_Reasoning"},"paperhash":{"value":"qin|promptcooperative_kv_cache_reuse_for_efficient_multiagent_llm_inference"},"authorids":{"value":["~Jiayu_Qin1","~Ankith_M_S1","~Harpreet_Kaur2","~Kevin_Zhai1","~Sumit_Negi2","~Changyou_Chen1"]},"abstract":{"value":"Multi-agent LLM systems suffer from redundant prefill computation as agents repeatedly process overlapping context---shared questions, peer outputs, and instructions. Prior methods address this by correcting KV cache offsets at inference time, requiring online anchor pools or selective recomputation. We propose a simpler alternative: \\textit{prompt-cooperative design}, where agent prompts are structured to minimize KV cache divergence across agents, enabling direct cache reuse without correction. Our key insight is that when system prompts differ by only 2--3 tokens (short role names), the resulting KV cache deviation for shared content remains below 2\\% in cosine distance across all layers, making uncorrected reuse safe. We implement this in vLLM via a greedy token aligner that identifies reusable spans and skips their forward computation. On three benchmarks (MMLU, GSM8K, HumanEval) with Qwen2.5-7B-Instruct and Qwen2.5-Coder-7B-Instruct, our approach achieves $\\sim$50\\% prefill token reduction with $<$0.8 percentage point quality loss compared to standard prefix caching---matching the quality preservation of more complex methods like KVComm while requiring no offset correction machinery."},"_bibtex":{"value":"@inproceedings{\nqin2026promptcooperative,\ntitle={Prompt-Cooperative {KV} Cache Reuse for Efficient Multi-Agent {LLM} Inference},\nauthor={Jiayu Qin and Ankith M S and Harpreet Kaur and Kevin Zhai and Sumit Negi and Changyou Chen},\nbooktitle={COLM 2026 Workshop on Efficient Reasoning},\nyear={2026},\nurl={https://openreview.net/forum?id=IYEaEZ6ZTH}\n}"},"title":{"value":"Prompt-Cooperative KV Cache Reuse for Efficient Multi-Agent LLM Inference"},"authors":{"value":["Jiayu Qin","Ankith M S","Harpreet Kaur","Kevin Zhai","Sumit Negi","Changyou Chen"]}},"tmdate":1786485565782,"pdate":1786144114341,"tcdate":1784536924338,"writers":["colmweb.org/COLM/2026/Workshop/Efficient_Reasoning","colmweb.org/COLM/2026/Workshop/Efficient_Reasoning/Submission188/Authors"],"signatures":["colmweb.org/COLM/2026/Workshop/Efficient_Reasoning/Submission188/Authors"],"forum":"IYEaEZ6ZTH","license":"CC BY 4.0","number":188,"cdate":1784536924338,"readers":["everyone"],"invitations":["colmweb.org/COLM/2026/Workshop/Efficient_Reasoning/-/Submission","colmweb.org/COLM/2026/Workshop/Efficient_Reasoning/-/Post_Submission","colmweb.org/COLM/2026/Workshop/Efficient_Reasoning/-/Edit"],"mdate":1786485565782,"odate":1786144114341,"domain":"colmweb.org/COLM/2026/Workshop/Efficient_Reasoning","id":"IYEaEZ6ZTH","version":2},{"content":{"TLDR":{"value":"ROKI combines weight-foldable isofrequency group rotations with importance-aware mixed precision to improve low-bit KV-cache quantization accuracy without additional online transformations"},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["KV cache quantization"]},"primary_area":{"value":"infrastructure, software libraries, hardware, systems, etc."},"abstract":{"value":"Low-bit KV-cache quantization reduces the memory and bandwidth costs of long-context LLM inference, but post-RoPE per-token quantization is sensitive to large activations that dominate the shared quantization range. Cross-channel rotations can reduce these outliers, yet general cross-pair rotations do not commute with Rotary Position Embedding (RoPE) and therefore cannot be folded into Q/K projection weights. We propose ROKI, a RoPE-aware KV-cache quantization framework based on Isofrequency Commutant Expansion (ICE). Rather than restricting transformations to the fixed RoPE operator, ICE selectively ties nearby low-frequency RoPE pairs, creating isofrequency groups that enlarge the exact RoPE-commuting transformation space from pair-wise to cross-pair rotations. ROKI uses this expanded space to redistribute Q/K activations and applies transformation-aware mixed-precision key quantization based on post-transformation QK contributions. All Q/K transformations are optimized offline and folded into the projection weights, requiring no additional online transformation. At an average 3-bit KV-cache budget, ROKI achieves the highest average LongBench-E score among the evaluated quantized baselines on three Llama/Qwen models while remaining close to FP16. Our ROKI\nimplementation also achieves up to 4.22$\\times$ decode speedup over the\nevaluated low-bit baselines at long contexts."},"_bibtex":{"value":"@inproceedings{\nanonymous2026roki,\ntitle={{ROKI}: Ro{PE}-Aware {KV} Cache Quantization via Isofrequency Grouping},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=sUzaZTsvLQ},\nnote={under review}\n}"},"title":{"value":"ROKI: RoPE-Aware KV Cache Quantization via Isofrequency Grouping"},"pdf":{"value":"/pdf/2ab4c960783dafa6dab073f44c9db5c0857c95a9.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791234596203,"tcdate":1789783056374,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission51906/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission51906/Authors"],"forum":"sUzaZTsvLQ","license":"CC BY 4.0","number":51906,"cdate":1789783056374,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission51906/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791234596203,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"sUzaZTsvLQ","version":2},{"content":{"venue":{"value":"COLM 2026 ER Workshop"},"pdf":{"value":"/pdf/7f3e7edbf9d17574ce7e490b7671f650977420c1.pdf"},"keywords":{"value":["KV cache eviction","KV cache compression","LLM inference","efficient NLP","memory-efficient decoding","reasoning models","value payload"]},"venueid":{"value":"colmweb.org/COLM/2026/Workshop/Efficient_Reasoning"},"paperhash":{"value":"yu|avokve_payloadaware_kv_cache_eviction_for_long_reasoning"},"authorids":{"value":["~Han_Yu6","~Wenhui_Zhu1","~Xiwen_Chen3","~Zhipeng_Wang1","~Hejian_Sang2","~Han_Shi8","~Menglin_Zhou1","~XUANZHAO_DONG1","~Minzhou_Huang1","~Rui_Cai3","~Hao_Wang68","~Alborz_Geramifard1"]},"abstract":{"value":"Long-output reasoning shifts the KV-cache bottleneck from prompt storage to generated reasoning traces. Existing reasoning-cache eviction methods mainly estimate routing utility: whether an old key will still be read, will recur, or can be replaced. We show that this routing-only view misses two effects in long reasoning. First, some low-attention entries carry large value payloads whose removal causes high future KL divergence. Second, newly generated reasoning states can appear stale before later tokens have had time to read them. We propose AvoKV-E, a training-free eviction policy that delays eligibility for newly generated states and ranks eligible entries by read pressure, key redundancy, and value-payload potential after normalizing each signal within the candidate set. In empirical evaluations, AvoKV-E matches or exceeds redundancy-aware, recurrence-based, and thought-adaptive eviction baselines under matched active-KV budgets, with the largest gains in the tightest-cache regime. The results suggest that long-reasoning KV eviction should preserve not only keys that are read, but also value payloads that sustain reasoning."},"_bibtex":{"value":"@inproceedings{\nyu2026avokve,\ntitle={Avo{KV}-E: Payload-Aware {KV} Cache Eviction for Long Reasoning},\nauthor={Han Yu and Wenhui Zhu and Xiwen Chen and Zhipeng Wang and Hejian Sang and Han Shi and Menglin Zhou and XUANZHAO DONG and Minzhou Huang and Rui Cai and Hao Wang and Alborz Geramifard},\nbooktitle={COLM 2026 Workshop on Efficient Reasoning},\nyear={2026},\nurl={https://openreview.net/forum?id=tFmJWIa3Tj}\n}"},"title":{"value":"AvoKV-E: Payload-Aware KV Cache Eviction for Long Reasoning"},"authors":{"value":["Han Yu","Wenhui Zhu","Xiwen Chen","Zhipeng Wang","Hejian Sang","Han Shi","Menglin Zhou","XUANZHAO DONG","Minzhou Huang","Rui Cai","Hao Wang","Alborz Geramifard"]}},"tmdate":1786485563574,"pdate":1786144108147,"tcdate":1784246378007,"writers":["colmweb.org/COLM/2026/Workshop/Efficient_Reasoning","colmweb.org/COLM/2026/Workshop/Efficient_Reasoning/Submission93/Authors"],"signatures":["colmweb.org/COLM/2026/Workshop/Efficient_Reasoning/Submission93/Authors"],"forum":"tFmJWIa3Tj","license":"CC BY 4.0","number":93,"cdate":1784246378007,"readers":["everyone"],"invitations":["colmweb.org/COLM/2026/Workshop/Efficient_Reasoning/-/Submission","colmweb.org/COLM/2026/Workshop/Efficient_Reasoning/-/Post_Submission","colmweb.org/COLM/2026/Workshop/Efficient_Reasoning/-/Edit"],"mdate":1786485563574,"odate":1786144108147,"domain":"colmweb.org/COLM/2026/Workshop/Efficient_Reasoning","id":"tFmJWIa3Tj","version":2},{"content":{"research_area_keywords":{"value":"pruning; quantization"},"languages_studied":{"value":"English"},"venue":{"value":"ACL ARR 2025 February Submission"},"_bibtex":{"value":"@inproceedings{\nanonymous2025more,\ntitle={More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in {KV} Cache Compression},\nauthor={Anonymous},\nbooktitle={Submitted to ACL Rolling Review - February 2025},\nyear={2025},\nurl={https://openreview.net/forum?id=yrGUpvcSEV},\nnote={under review}\n}"},"title":{"value":"More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression"},"contribution_types":{"value":["NLP engineering experiment","Approaches low compute settings-efficiency","Data analysis"]},"abstract":{"value":"As large language models (LLMs) process increasing context windows, the memory usage of KV cache has become a critical bottleneck during inference. The mainstream KV compression methods, including KV pruning and KV quantization, primarily focus on either token or precision dimension separately. However, these works leaving the trade-off between these two orthogonal dimensions largely under-explored. \nIn this paper, we comprehensively investigate the token-precision trade-off in KV cache compression. \nExperiments demonstrate that storing more tokens \nin the KV cache with lower precision, \n a strategy we term quantized pruning, can significantly enhance the long-context performance of LLMs. \nIn-depth analysis of the token-precision trade-off across key aspects demonstrates that, quantized pruning achieves substantial improvements in retrieval-related tasks and consistently performs well across varying input lengths. Furthermore,\nquantized pruning demonstrates notable stability and effectiveness across different KV pruning methods, quantization strategies, and model scales. These findings offer valuable insights into optimizing KV cache compression through balanced token-precision trade-off strategies. We plan to release our code soon."},"paper_type":{"value":"Long"},"pdf":{"value":"/pdf/8d2f5ca770d27a129ac52ccc18989032a32a66a5.pdf"},"research_area":{"value":"Efficient/Low-Resource Methods for NLP"},"venueid":{"value":"aclweb.org/ACL/ARR/2025/February/Submission"}},"tmdate":1782962101189,"tcdate":1739701772958,"writers":["aclweb.org/ACL/ARR/2025/February","aclweb.org/ACL/ARR/2025/February/Submission7133/Authors"],"signatures":["aclweb.org/ACL/ARR/2025/February/Submission7133/Authors"],"forum":"yrGUpvcSEV","license":"CC BY 4.0","number":7133,"cdate":1739701772958,"readers":["everyone"],"invitations":["aclweb.org/ACL/ARR/2025/February/-/Submission","aclweb.org/ACL/ARR/2025/February/-/Edit","aclweb.org/ACL/ARR/2025/February/-/Post_Submission","aclweb.org/ACL/ARR/2025/February/-/Desk_Reject_Verification_Flag","aclweb.org/ACL/ARR/2025/February/-/Preprint_Release_Submission","aclweb.org/ACL/ARR/2025/February/-/Preprint_Post_Submission"],"mdate":1782962101189,"odate":1746821355472,"domain":"aclweb.org/ACL/ARR/2025/February","id":"yrGUpvcSEV","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"https://arxiv.org/pdf/2412.12706v2"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"zhang|more_tokens_lower_precision_towards_the_optimal_tokenprecision_tradeoff_in_kv_cache_compression"},"authorids":{"value":["","","","","","","~Lifeng_Shang1","",""]},"html":{"value":"https://doi.org/10.48550/arXiv.2412.12706"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2412-12706,\n publtype={informal},\n author={Jiebin Zhang and Dawei Zhu and Yifan Song and Wenhao Wu and Chuqiao Kuang and Xiaoguang Li and Lifeng Shang and Qun Liu and Sujian Li},\n title={More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression},\n year={2024},\n cdate={1704067200000},\n journal={CoRR},\n volume={abs/2412.12706},\n url={https://doi.org/10.48550/arXiv.2412.12706}\n}\n"},"abstract":{"value":"As large language models (LLMs) process increasing context windows, the memory usage of KV cache has become a critical bottleneck during inference. The mainstream KV compression methods, including KV pruning and KV quantization, primarily focus on either token or precision dimension separately. However, these works leaving the trade-off between these two orthogonal dimensions largely under-explored. In this paper, we comprehensively investigate the token-precision trade-off in KV cache compression.Experiments demonstrate that storing more tokens in the KV cache with lower precision,a strategy we term quantized pruning, can significantly enhance the long-context performance of LLMs. In-depth analysis of the token-precision trade-off across key aspects demonstrates that, quantized pruning achieves substantial improvements in retrieval-related tasks and consistently performs well across varying input lengths. Furthermore, quantized pruning demonstrates notable stability and effectiveness across different KV pruning methods, quantization strategies, and model scales. These findings offer valuable insights into optimizing KV cache compression through balanced token-precision trade-off strategies. Our code is available at https://github.com/zhzihao/QPruningKV."},"title":{"value":"More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression"},"authors":{"value":["Jiebin Zhang","Dawei Zhu","Yifan Song","Wenhao Wu","Chuqiao Kuang","Xiaoguang Li","Lifeng Shang","Qun Liu","Sujian Li"]}},"tmdate":1773712241969,"pdate":1735603200000,"externalIds":["dblp:journals/corr/abs-2412-12706"],"tcdate":1773712218557,"writers":["~"],"signatures":["~Lifeng_Shang1"],"forum":"frbiVLAI4m","license":"CC BY-SA 4.0","number":853079,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1773712241969,"domain":"DBLP.org","id":"frbiVLAI4m","version":2},{"content":{"research_area_keywords":{"value":"Multimodal Long-context compression, multimodal KV cache optimization, multimodal large language model, efficient ML"},"languages_studied":{"value":"English"},"venue":{"value":"ACL ARR 2024 October Submission"},"_bibtex":{"value":"@inproceedings{\nanonymous2025attention,\ntitle={Attention Entropy-Guided Dynamic Cache Allocation for Efficient Multimodal Long-Context Inference},\nauthor={Anonymous},\nbooktitle={Submitted to ACL Rolling Review - October 2024},\nyear={2025},\nurl={https://openreview.net/forum?id=JJVYGwMAtQ},\nnote={under review}\n}"},"title":{"value":"Attention Entropy-Guided Dynamic Cache Allocation for Efficient Multimodal Long-Context Inference"},"contribution_types":{"value":["Model analysis & interpretability","NLP engineering experiment","Reproduction study","Approaches to low-resource settings","Approaches low compute settings-efficiency"]},"abstract":{"value":"Long-context Multimodal Large Language Models (MLLMs), incorporating long text-image and text-video modalities, demand substantial computational resources as their multimodal Key-Value (KV) cache grows with increasing input lengths, challenging memory and time efficiency. For multimodal scenarios, the cross-modal interactions inevitablely increase complexity, and prior methods for KV cache compression, in both text-only and multimodal LLMs, have neglected attention density variations across layers, often adopting uniform or progressive reduction strategis for layer-wise cache allocation. This results in precision loss and suboptimal performance. We propose \\textbf{MEDA} (\\textbf{M}ultimodal Attention \\textbf{E}ntropy-Guided \\textbf{D}ynamic \\textbf{A}llocation), a novel approach specifically designed for the complexities of multimodal settings, dynamically allocating KV cache sizes based on attention entropy to better adapt to multimodal interactions. Through a dynamic multimodal KV cache allocation strategy, \\textbf{MEDA} compresses the KV cache, adaptively retains sufficient multimodal information at each layer. Meanwhile, to mitigate the degradation of contextual information due to cache compression, we also integrate KV pairs merging techniques to maintain coherence. \\textbf{MEDA} achieves up to \\textbf{72\\%} KV cache memory reduction and \\textbf{2.82} faster decoding speeds in some cases, while maintaining or enhancing performance on various multimodal tasks in a long context, including multi-image and long video scenarios."},"paper_type":{"value":"Long"},"pdf":{"value":"/pdf/fe2ebd313aa530eb477069e0185679519cd1b32b.pdf"},"research_area":{"value":"Machine Learning for NLP"},"venueid":{"value":"aclweb.org/ACL/ARR/2024/October/Submission"}},"tmdate":1782962365273,"tcdate":1729051658698,"writers":["aclweb.org/ACL/ARR/2024/October","aclweb.org/ACL/ARR/2024/October/Submission1823/Authors"],"signatures":["aclweb.org/ACL/ARR/2024/October/Submission1823/Authors"],"forum":"JJVYGwMAtQ","license":"CC BY 4.0","number":1823,"cdate":1729051658698,"readers":["everyone"],"invitations":["aclweb.org/ACL/ARR/2024/October/-/Submission","aclweb.org/ACL/ARR/2024/October/-/Edit","aclweb.org/ACL/ARR/2024/October/-/Post_Submission","aclweb.org/ACL/ARR/2024/October/-/Preprint_Release_Submission","aclweb.org/ACL/ARR/2024/October/-/Preprint_Post_Submission"],"mdate":1782962365273,"odate":1736460501709,"domain":"aclweb.org/ACL/ARR/2024/October","id":"JJVYGwMAtQ","version":2},{"content":{"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["KV-cache quantization","efficient LLM inference","long-context language models","mixed-precision bit allocation","rotary position embeddings (RoPE)"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Larger models and longer contexts make the KV cache a memory and bandwidth\nbottleneck. We introduce \\emph{Block-GTQ}, a framework that uses the\ntwo-dimensional frequency blocks of rotary position embeddings (RoPE) to\nallocate cached-key precision. A blockwise logit-error bound motivates a\nlabel-free query--key energy score, which we combine with codec-dependent\nrate penalties. Under equal-cost budgets and diminishing marginal gains,\nlargest-marginal-gain allocation exactly minimizes the separable integer\nproxy. Our TurboQuant-MSE (TQ-MSE) instantiation groups equal-rate blocks\nand supports attention over packed caches without materializing a decoded\nfp16 KV cache. Across ten model/path configurations, K-only diagnostics\nreduce layer-averaged logit mean absolute error by about $32$--$80\\%$\nrelative to uniform TQ-MSE, improving all $367$ layers at both 2- and 3-bit\naverage logical key rates. At K2V2 (2 average logical bits per K/V\ncoordinate) on Llama-3.1-8B-Instruct, Block-GTQ raises the six-task\nneedle-in-a-haystack (NIAH) average from $70.6$ to $97.4$ and LongBench-EN\nfrom $33.7$ to $44.9$; K2V2 gains also hold on the tested 32B and\n70B models. On DeepSeek-R1-Distill-Qwen-7B at K3V2, without fp16 sink or\nrecent-token buffers, it achieves $49.6/38.8\\%$ avg@8 on AIME 2024/2025 versus fp16’s $54.2/37.9\\%$,\nwhere uniform TQ-MSE scores zero on both. On Qwen2.5-3B-Instruct, packed\nK3V3 serving on one H800 reduces resident KV footprint by a factor of\n$3.24$ and retains perplexity close to fp16 at 128K, while uniform TQ-MSE\ndegrades severely. At a 1M-token context, Block-GTQ reduces decode latency by 8.0\\%\nrelative to FP16 FlashAttention-2."},"_bibtex":{"value":"@inproceedings{\nanonymous2026blockgtq,\ntitle={Block-{GTQ}: Ro{PE}-Block Rate Allocation for {KV}-Cache Quantization},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=rS9IUWWuiX},\nnote={under review}\n}"},"title":{"value":"Block-GTQ: RoPE-Block Rate Allocation for KV-Cache Quantization"},"pdf":{"value":"/pdf/1697b6d115851ebd6a8d38acade8433cb2d72608.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791231643125,"tcdate":1789662199704,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission33096/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission33096/Authors"],"forum":"rS9IUWWuiX","license":"CC BY 4.0","number":33096,"cdate":1789662199704,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission33096/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791231643125,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"rS9IUWWuiX","version":2},{"content":{"summary":{"value":"The paper targets KV-cache compression for auto-regressive LLMs. Observing that (i) positional encodings dominate query representations and (ii) ground-truth decoding queries are unavailable at prefill, the authors propose DapQ: a lightweight prefill-only method that constructs “position-aware pseudo-queries” by appending synthetic tokens whose position IDs match the next N decoding positions. Attention from these pseudo-queries to the prompt keys yields an importance score; Top-K keys are retained. Extensive experiments on LongBench, Ruler, HELMET and Needle-in-a-Haystack (4 models, cache budgets down to 3 %) show DapQ consistently outperforming SnapKV, PyramidKV, H2O, StreamingLLM, etc., often approaching full-cache accuracy (e.g. 99.5 % on NIAH with only 3 % KV budget). The paper claims 4.7× speed-up over prior retrieval-based methods and near-lossless quality."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"1.\tCould the authors add experiments that compare DaqQ with more recent state-of-the-art approaches?\n2.\tCould the authors include ablation experiments to examine the effect of placing pseudo-queries at different positions, not just following prompt?"},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"Interesting idea that uses position-aware pseudo-queries to simulate the tokens to be generated and predict attention patterns, thereby guiding KV-cache compression."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- Figure 1 in the paper contains obvious errors, which detract from the overall quality of the manuscript.\n- The paper sets the position IDs of the pseudo-queries to a short span immediately after the prompt, highly close to SnapKV’s observation window. Consequently, DaqQ fails to outperform SnapKV by a noticeable margin, especially in Figures 4(b) and 4(c).\n- Most experiments compare DaqQ against dated baselines such as SnapKV and H2O (published one to two years ago). Although the very recent LaCache is also included, it consistently performs poorly on all benchmarks. Comparing DaqQ with more recent state-of-the-art methods (e.g., AdaKV) would provide a stronger demonstration of its effectiveness."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1764359104003,"tcdate":1761929483639,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission16503/Reviewer_qbfR"],"signatures":["ICLR.cc/2026/Conference/Submission16503/Reviewer_qbfR"],"forum":"ZpgyPxdxiL","number":3,"license":"CC BY 4.0","cdate":1761929483639,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission16503/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1764359104003,"domain":"ICLR.cc/2026/Conference","replyto":"ZpgyPxdxiL","id":"2Qn7K5zhi2","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["LLM","KV cache compression","Long-context"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"The Key-Value (KV) cache is crucial for efficient Large Language Models (LLMs) inference, but excessively long contexts drastically increase KV cache memory footprint. Existing KV cache compression methods typically rely on input-side attention patterns within a prompt observation window to estimate token importance during the prefill stage. They fail to preserve critical tokens for future generation since these assessments are not derived from the decoding process. Intuitively, an effective observation window should mirror the decoding-stage queries to accurately reflect which tokens the generation process will attend to. However, ground-truth decoding queries are inherently unavailable during inference. For constructing pseudo-queries to approximate them, we find that positional information plays a more critical role than semantic content. Motivated by this insight, we propose decoding-aligned KV cache compression via position-aware pseudo-queries (DapQ), a novel and lightweight eviction framework that leverages position-aware pseudo-queries to simulate the output tokens, thereby establishing an effective observation window for importance assessment. It enables precise token eviction that aligns closely with the actual generation context. Extensive evaluation across multiple benchmarks and LLMs demonstrates that DapQ achieves superior performance, particularly under strict memory constraints (e.g., up to nearly lossless performance 99.5\\% on NIAH with 3\\% KV cache budgets)."},"_bibtex":{"value":"@misc{\ntian2025where,\ntitle={Where Matters More Than What: Decoding-aligned {KV} Cache Compression via Position-aware Pseudo-queries},\nauthor={Zhenxu Tian and Yi Su and Juntao Li and Min Zhang},\nyear={2025},\nurl={https://openreview.net/forum?id=ZpgyPxdxiL}\n}"},"title":{"value":"Where Matters More Than What: Decoding-aligned KV Cache Compression via Position-aware Pseudo-queries"},"pdf":{"value":"/pdf/ad512fcca70f8b5e5368f1ae98d812868865ee6c.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"tian|where_matters_more_than_what_decodingaligned_kv_cache_compression_via_positionaware_pseudoqueries"},"authorids":{"value":["~Zhenxu_Tian1","~Yi_Su3","~Juntao_Li2","~Min_Zhang9"]},"authors":{"value":["Zhenxu Tian","Yi Su","Juntao Li","Min Zhang"]}},"version":2},{"content":{"venue":{"value":"CoRR 2026"},"pdf":{"value":"https://arxiv.org/pdf/2601.21686v1"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"benfenati|dont_be_so_stief_learning_kv_cache_lowrank_approximation_over_the_stiefel_manifold"},"html":{"value":"https://doi.org/10.48550/arXiv.2601.21686"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2601-21686,\n publtype={informal},\n author={Luca Benfenati and Matteo Risso and Andrea Vannozzi and Ahmet Caner Yüzügüler and Lukas Cavigelli and Enrico Macii and Daniele Jahier Pagliari and Alessio Burrello},\n title={Don't be so Stief! Learning KV Cache low-rank approximation over the Stiefel manifold},\n year={2026},\n month={January},\n cdate={1767225600000},\n journal={CoRR},\n volume={abs/2601.21686},\n url={https://doi.org/10.48550/arXiv.2601.21686}\n}\n"},"abstract":{"value":"Key--value (KV) caching enables fast autoregressive decoding but at long contexts becomes a dominant bottleneck in High Bandwidth Memory (HBM) capacity and bandwidth. A common mitigation is to compress cached keys and values by projecting per-head matrixes to a lower rank, storing only the projections in the HBM. However, existing post-training approaches typically fit these projections using SVD-style proxy objectives, which may poorly reflect end-to-end reconstruction after softmax, value mixing, and subsequent decoder-layer transformations. For these reasons, we introduce StiefAttention, a post-training KV-cache compression method that learns \\emph{orthonormal} projection bases by directly minimizing \\emph{decoder-layer output reconstruction error}. StiefAttention additionally precomputes, for each layer, an error-rank profile over candidate ranks, enabling flexible layer-wise rank allocation under a user-specified error budget. Noteworthy, on Llama3-8B under the same conditions, StiefAttention outperforms EigenAttention by $11.9$ points on C4 perplexity and $5.4\\%$ on 0-shot MMLU accuracy at iso-compression, yielding lower relative error and higher cosine similarity with respect to the original decoder-layer outputs."},"title":{"value":"Don't be so Stief! Learning KV Cache low-rank approximation over the Stiefel manifold"},"authors":{"value":[{"fullname":"Luca Benfenati","username":"~Luca_Benfenati1"},{"fullname":"Matteo Risso","username":"~Matteo_Risso1"},{"fullname":"Andrea Vannozzi","username":""},{"fullname":"Ahmet Caner Yüzügüler","username":""},{"fullname":"Lukas Cavigelli","username":""},{"fullname":"Enrico Macii","username":""},{"fullname":"Daniele Jahier Pagliari","username":""},{"fullname":"Alessio Burrello","username":""}]}},"tmdate":1780301319643,"pdate":1798675200000,"externalIds":["dblp:journals/corr/abs-2601-21686"],"tcdate":1779439241355,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Matteo_Risso1"],"forum":"oL6vdh5W5Y","license":"CC BY-SA 4.0","number":19808,"cdate":1767225600000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit","OpenReview.net/Public_Article/-/Authorship_Claim"],"mdate":1780301319643,"domain":"OpenReview.net/Public_Article","id":"oL6vdh5W5Y","version":2},{"content":{"TLDR":{"value":"We show when compressed-cache geometry turns otherwise insufficient current-output fidelity into reliable transition control for KV-cache replacement."},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["KV cache compression","long-context LLM inference","attention","transformer inference","theoretical analysis","cache replacement","efficient inference"]},"supplementary_material":{"value":"/attachment/b9f61d6c82d945beb764c23a66de014aa4286d6d.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Long-context autoregressive inference is increasingly constrained by KV caches, whose memory footprint grows with sequence length, motivating compression for memory-efficient inference. However, when compression replaces the full cache with a smaller one that continued decoding must use and update, matching the current attention output is insufficient in general: two cache states can agree now yet diverge after the same append because softmax hides normalization mass. We characterize when current-output error can nevertheless control cache-extension error. For unchanged-row selection, this control depends on omitted attention mass and the separation between retained and omitted value means. Cancellation is the failure boundary: substantial omitted mass can remain nearly invisible in the current output. Away from it, current error controls the hidden mass exposed by an append, yielding a sharp, attainable one-append guarantee. The same control extends perturbatively to more general compressed caches that remain close, in response and normalization mass, to an unchanged-row selection on the same support. Fresh Qwen and Mistral evaluations support this mechanism: transition-aware scoring improves pairwise post-extension concordance over current-output scoring by .320 and .211, respectively, with 95% confidence intervals excluding zero in both model families. CaM and KeepKV test perturbative transfer at different scales. Finally, we characterize exactly when these headwise guarantees survive fixed output projection. Together, the results show how the geometry of compressed caches can make current-output error informative for reliable KV-cache replacement despite its global insufficiency."},"_bibtex":{"value":"@inproceedings{\nanonymous2026kv,\ntitle={{KV} Cache Compression as State Substitution: When Does {KV} Compression Yield a Reliable Replacement for the Full Cache?},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=TgUIFZtPO0},\nnote={under review}\n}"},"title":{"value":"KV Cache Compression as State Substitution: When Does KV Compression Yield a Reliable Replacement for the Full Cache?"},"pdf":{"value":"/pdf/f8ef9853f32b56c1cb75803554527d732de59ac4.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791233261794,"tcdate":1789736111639,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission42856/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission42856/Authors"],"forum":"TgUIFZtPO0","license":"CC BY 4.0","number":42856,"cdate":1789736111639,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Edit","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission42856/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing"],"mdate":1791233261794,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"TgUIFZtPO0","version":2},{"content":{"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["llm; infra"]},"primary_area":{"value":"infrastructure, software libraries, hardware, systems, etc."},"abstract":{"value":"Long-horizon language-model agents place unique demands on LLM serving systems because they repeatedly return from tool calls with increasingly long contexts. At high concurrency, these growing contexts can evict one another’s reusable KV cache, forcing repeated prefill, slowing completion, and keeping even more contexts resident. We call this positive-feedback failure KV-cache collapse. We show that the onset of collapse can be predicted from the relationship between available KV capacity and the average context demand of each instance. We introduce AGENTKV, a predictive admission controller that forecasts shorthorizon context growth, maintains a lifecycle-safe working set under an adaptive high-bandwidth memory (HBM) budget, and replans from live cache telemetry. Across ten Qwen3-8B and Qwen3.8-27B campaigns on four RTX PRO 6000 GPUs, the predicted capacity boundary closely matches the observed transition from stable prefix reuse to cache collapse. Beyond this boundary, AGENTKV outperforms vLLM, ThunderAgent, InferCept, Continuum-TTL, and cache-aware vLLM-router in every measured cell, reaching up to 3.7× the throughput of stock vLLM. At 240 concurrent instances, AGENTKV retains roughly 75% prefix reuse for Qwen3-8B workloads while stock vLLM falls to at most 0.4%, and removing predictive admission leaves only 27–77% of full-system throughput. These results show that predictive working-set control can prevent a high-load cache-collapse regime that allocation, routing, and persistence alone do not fully address."},"_bibtex":{"value":"@inproceedings{\nanonymous2026agentkv,\ntitle={Agent{KV}: Predictive Admission Prevents {KV}-Cache Collapse in Agentic {LLM} Serving},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=zLtopd1ATE},\nnote={under review}\n}"},"title":{"value":"AgentKV: Predictive Admission Prevents KV-Cache Collapse in Agentic LLM Serving"},"pdf":{"value":"/pdf/26718e436b30d9794b6a6d336336b9f1390fe1f5.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791233950413,"tcdate":1789756960310,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission47397/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission47397/Authors"],"forum":"zLtopd1ATE","license":"CC BY 4.0","number":47397,"cdate":1789756960310,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission47397/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791233950413,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"zLtopd1ATE","version":2},{"content":{"TLDR":{"value":"We introduce OCP, a first-order framework that predicts whether KV-cache compression preserves the dense-model output and when compressed generation will first diverge."},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["KV-cache compression","output preservation","first divergence","output crossing","efficient LLM inference"]},"primary_area":{"value":"infrastructure, software libraries, hardware, systems, etc."},"abstract":{"value":"Key-Value (KV) cache compression is usually evaluated by how much cache state is removed or distorted, but cache distortion alone does not determine whether the model output is preserved. Under deterministic greedy decoding, we study when compression preserves the dense-model token and when compressed generation first diverges. Our key observation is that output preservation depends not only on perturbation magnitude, but also on how the compression-induced change acts relative to the decision margin. Based on this observation, we derive an exact per-step characterization of output preservation through output crossing. We then introduce the Output Crossing Predictor (OCP), a first-order framework that predicts whether a candidate compression will cause output crossing before compressed logits are observed. OCP combines the local cache-read perturbation with the local sensitivity of the dense model. Repeated across decoding steps, the same prediction estimates the first divergence time. Experiments across multiple models and compression operators show that OCP predicts output crossing and first divergence, while the compression boundary becomes more conservative as generation length increases. These results shift KV-cache compression from measuring cache distortion to predicting output preservation and first divergence."},"_bibtex":{"value":"@inproceedings{\nanonymous2026predicting,\ntitle={Predicting Output Preservation and First Divergence under {KV}-Cache Compression},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=zKxUf16XnW},\nnote={under review}\n}"},"title":{"value":"Predicting Output Preservation and First Divergence under KV-Cache Compression"},"pdf":{"value":"/pdf/21e3861429abd8b83de481953a9a4b6d160dd79a.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791226994638,"tcdate":1789217754128,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission15160/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission15160/Authors"],"forum":"zKxUf16XnW","license":"CC BY 4.0","number":15160,"cdate":1789217754128,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Edit","ICLR.cc/2027/Conference/Submission15160/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing"],"mdate":1791226994638,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"zKxUf16XnW","version":2},{"content":{"summary":{"value":"This paper proposes a novel method, Dynamic-LLaVA, which dynamically sparsifies visual context during the prefill stage and vision-text context during the decoding stage of Visual Language Model inference. Specifically, two learnable predictors are designed to identify important visual and language tokens. Additionally, masked softmax is applied during training, and Gumbel-softmax with a Straight-Through Gradient Estimator is used to address gradient flow issues. Furthermore, a batch-parallel strategy is introduced to enable efficient sparsification during inference in the Dynamic-LLaVA framework. Empirical experiments demonstrate that this method can compress approximately 75% of visual context and save around 50% of GPU memory for the KV cache."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"1. Previous studies have shown that deeper layers exhibit higher attention sparsity. How is sparsification strength adjusted for different decoder layers?\n2. Running predictors introduces additional latency. However, the proportion of this latency relative to overall prefill and decoding time is not disclosed. Additionally, in standard LLaVA decoding, previous KV cache is fetched, and byproduct attention scores are accumulated to assess token importance, which is quick and efficient. However, for \"Online KV Cache Compression,\" running predictor incurs additional latency. And only the current token is used to predict all previous tokens, do you think solely performing prediction on a current token provide enough information? Have you analyzed the wrongly evicted tokens?\n3. Can this method be applied to language models that utilize Group Query Attention?\n4. Have you considered scenarios where long-text context is present during the prefill stage?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"This paper introduces a novel design for two token predictors, formulating visual and textual context compression as an end-to-end optimization problem. Techniques such as Gumbel-Softmax and the Straight-Through Gradient Estimator are employed to enhance training stability. In addition, parallel sparsification inference optimization would provide an effective solution for model serving."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. It is unclear whether the two learnable predictors are applied to each decoder layer. If they are, do they share the same parameters or are they independently parameterized? If not all layers, please clarify which specific decoder layers the predictors are applied to.\n2. Compressing generated text is primarily beneficial for long-generation tasks. Would you please provide specific statistics on the average output length for each task they evaluated? And discuss how the benefits of their method scale with increasing output length.\n3. Please quantify the additional training time and computational resources required for this method compared to direct KV cache compression approaches, as well as additional prediction overhead compared normal decoding. This would help readers better understand the tradeoffs.\n4. More recent related work about KV Cache compression method should be included, such as TidalDecode (https://arxiv.org/abs/2410.05076) focusing on decoding compression. \n5. A few mathematical representations are not appropriate. For example, in section 3.1, $\\mathcal{I}^I = $\\{ $1, 2, \\cdots, N_{l}^I $}, layer index $l$ should be set for $\\mathcal{I}^I $; And in section 3.3.1, all layers share the same binary masks $\\mathcal{M}^I and \\mathcal{M}^{OT}$? since there are no layer index $l$. \n6. The writing could be improved to make the narrative more concise and clear."}},"nonreaders":[],"tmdate":1732692268144,"tcdate":1730710355516,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission4573/Reviewer_UyMq"],"signatures":["ICLR.cc/2025/Conference/Submission4573/Reviewer_UyMq"],"forum":"hzVpZDrW73","number":3,"license":"CC BY 4.0","cdate":1730710355516,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission4573/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732692268144,"domain":"ICLR.cc/2025/Conference","replyto":"hzVpZDrW73","id":"XtZY02t9nu","forumContent":{"venue":{"value":"ICLR 2025 Poster"},"TLDR":{"value":"Dynamic-LLaVA is the first MLLM acceleration framework that simultaneously sparsifies both vision and language contexts while integrating inference efficiency optimization across different MLLM inference modes into a unified framework."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Efficient Multimodal Large Language Model"]},"supplementary_material":{"value":"/attachment/d345998fca0c4510cdce9578ff0013e82c380754.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Multimodal Large Language Models (MLLMs) have achieved remarkable success in vision understanding, reasoning, and interaction. However, the inference computation and memory increase progressively with the generation of output tokens during decoding, directly affecting the efficacy of MLLMs. Existing methods attempt to reduce the vision context redundancy to achieve efficient MLLMs. Unfortunately, the efficiency benefits of the vision context reduction in the prefill stage gradually diminish during the decoding stage. To address this problem, we proposed a dynamic vision-language context sparsification framework Dynamic-LLaVA, which dynamically reduces the redundancy of vision context in the prefill stage and decreases the memory and computation overhead of the generated language context during decoding. Dynamic-LLaVA designs a tailored sparsification inference scheme for different inference modes, i.e., prefill, decoding with and without KV cache, to achieve efficient inference of MLLMs. In practice, Dynamic-LLaVA can reduce computation consumption by $\\sim$75\\% in the prefill stage. Meanwhile, throughout the entire generation process of MLLMs, Dynamic-LLaVA reduces the $\\sim$50\\% computation consumption under decoding without KV cache, while saving $\\sim$50\\% GPU memory overhead when decoding with KV cache, due to the vision-language context sparsification. Extensive experiments also demonstrate that Dynamic-LLaVA achieves efficient inference for MLLMs with negligible understanding and generation ability degradation or even performance gains compared to the full-context inference baselines. Code is available at https://github.com/Osilly/dynamic_llava."},"_bibtex":{"value":"@inproceedings{\nhuang2025dynamicllava,\ntitle={Dynamic-{LL}a{VA}: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification},\nauthor={Wenxuan Huang and Zijie Zhai and Yunhang Shen and Shaosheng Cao and Fei Zhao and Xiangfeng Xu and Zheyu Ye and Shaohui Lin},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=hzVpZDrW73}\n}"},"title":{"value":"Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification"},"pdf":{"value":"/pdf/a07b1b7ec31c8e57a857c631b1c5a7181d1e509d.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"huang|dynamicllava_efficient_multimodal_large_language_models_via_dynamic_visionlanguage_context_sparsification"},"authorids":{"value":["~Wenxuan_Huang2","~Zijie_Zhai1","~Yunhang_Shen1","~Shaosheng_Cao2","~Fei_Zhao5","~Xiangfeng_Xu1","~Zheyu_Ye1","~Shaohui_Lin1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Wenxuan Huang","Zijie Zhai","Yunhang Shen","Shaosheng Cao","Fei Zhao","Xiangfeng Xu","Zheyu Ye","Shaohui Lin"]}},"version":2},{"content":{"summary":{"value":"This paper proposes FastGen, an adaptive key-value (KV) cache compression method to reduce the memory footprint and accelerate inference for large language models (LLMs). The key ideas are: 1) Profiling attention modules to discern their intrinsic structures, such as primarily attending to local contexts or special tokens. 2) Constructing the KV cache adaptively based on the recognized structure to compress less useful contexts. 3) The lightweight attention profiling guides the KV cache compression without expensive fine-tuning.\n\nThe experiments are conducted on LLaMa models with sizes from 7B to 65B parameters on diverse generative tasks. Results show FastGen effectively compresses the KV cache to 40-50% smaller with negligible quality loss. It also outperforms non-adaptive baselines."},"presentation":{"value":"3 good"},"contribution":{"value":"3 good"},"soundness":{"value":"3 good"},"strengths":{"value":"Strengths:\n-----------------\n+ Adaptively compressing KV cache better aligns with model-specific attributes without retraining.\n+ Comprehensive experiments verify FastGen works for diverse models and tasks. Up to 50% compression on 65B LLaMa with little quality loss is remarkable.\n+ Ablation studies provide good insight into the design choices. The profiling method and compression policies are well motivated."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"Weaknesses:\n-----------------\n- The compression policies are combined in a naive way. More advanced adaptive selection could be explored (see detailed in C1).\n- No experiment on encoder-decoder models. The efficacy on them is unclear (see detailed in C2). \n- More analysis on the overhead of profiling could be provided (see detailed in C3)."},"confidence":{"value":"3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked."},"questions":{"value":"Comments:\n-----------------\nC1:\tThe compression policies are combined in a simple naive way in FastGen, by just taking the union of multiple policies such as Cspecial + Cpunct + Cfrequent. This straightforward combination approach has several potential issues. First, is it possible that the union combination may introduce redundancy, as different policies could select overlapping important content, leading to suboptimal compression ratios? More intelligent strategies should consider the complementarity between modules to avoid duplicating the key contexts. Second, is it possible that existing policies may not be fully compatible? Some combinations could introduce conflicts and hurt generation quality. More systematic analysis should examine the compatibility between policies.\n\nC2:\tThe experiments in the paper are all conducted on the decoder-only LLaMa models, without validation on encoder-decoder models like BART and T5. These models are also widely used for generative tasks, so the efficacy of FastGen on them remains unclear. This is worth further investigation. \n\nC3:\tThe paper lacks sufficient analysis on the overhead and time cost of conducting attention profiling, which is important to judge the efficiency of FastGen in real deployment. Specifically, the time complexity of attention profiling needs analysis, and concrete profiler time under different model sizes should be provided or disscussed. Moreover, analyzing the extra memory or GPU memory required for the profiler and assessing its impact on deployment is necessary. In summary, quantitatively analyzing the resource overhead for profiling and demonstrating effective solutions to reduce it could strengthen the practicality of FastGen in real-world usage. Further experiments on optimized profiling and its cost-benefit trade-off with compression performance could provide more comprehensive insights into the efficacy of the approach."},"rating":{"value":"8: accept, good paper"},"code_of_conduct":{"value":"Yes"}},"nonreaders":[],"tmdate":1699636738293,"tcdate":1698662301479,"writers":["ICLR.cc/2024/Conference","ICLR.cc/2024/Conference/Submission6547/Reviewer_9wxD"],"signatures":["ICLR.cc/2024/Conference/Submission6547/Reviewer_9wxD"],"forum":"uNrFpDPMyo","number":3,"license":"CC BY 4.0","cdate":1698662301479,"readers":["everyone"],"invitations":["ICLR.cc/2024/Conference/Submission6547/-/Official_Review","ICLR.cc/2024/Conference/-/Edit"],"mdate":1699636738293,"domain":"ICLR.cc/2024/Conference","replyto":"uNrFpDPMyo","id":"aNqNdIZp5T","forumContent":{"TLDR":{"value":"We introduce adaptive KV cache compression, a plug-and-play method that reduces the memory footprint of generative inference for Large Language Models (LLMs) and accelerates its generation throughput."},"venue":{"value":"ICLR 2024 oral"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Large Language Model","Efficient Inference","Generative Inference","Key-Value Cache"]},"primary_area":{"value":"representation learning for computer vision, audio, language, and other modalities"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide."},"abstract":{"value":"In this study, we introduce adaptive KV cache compression, a plug-and-play method that reduces the memory footprint of generative inference for Large Language Models (LLMs). Different from the conventional KV cache that retains key and value vectors for all context tokens, we conduct targeted profiling to discern the intrinsic structure of attention modules. Based on the recognized structure, we then construct the KV cache in an adaptive manner: evicting long-range contexts on attention heads emphasizing local contexts, discarding non-special tokens on attention heads centered on special tokens, and only employing the standard KV cache for attention heads that broadly attend to all tokens. Moreover, with the lightweight attention profiling used to guide the construction of the adaptive KV cache, FastGen can be deployed without resource-intensive fine-tuning or re-training. In our experiments across various asks, FastGen demonstrates substantial reduction on GPU memory consumption with negligible generation quality loss. We will release our code and the compatible CUDA kernel for reproducibility."},"_bibtex":{"value":"@inproceedings{\nge2024model,\ntitle={Model Tells You What to Discard: Adaptive {KV} Cache Compression for {LLM}s},\nauthor={Suyu Ge and Yunan Zhang and Liyuan Liu and Minjia Zhang and Jiawei Han and Jianfeng Gao},\nbooktitle={The Twelfth International Conference on Learning Representations},\nyear={2024},\nurl={https://openreview.net/forum?id=uNrFpDPMyo}\n}"},"title":{"value":"Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs"},"pdf":{"value":"/pdf/757a55aa24be0345fe1687e09fa5ca448934e52f.pdf"},"venueid":{"value":"ICLR.cc/2024/Conference"},"paperhash":{"value":"ge|model_tells_you_what_to_discard_adaptive_kv_cache_compression_for_llms"},"authorids":{"value":["~Suyu_Ge1","~Yunan_Zhang1","~Liyuan_Liu3","~Minjia_Zhang1","~Jiawei_Han1","~Jianfeng_Gao1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors' identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Suyu Ge","Yunan Zhang","Liyuan Liu","Minjia Zhang","Jiawei Han","Jianfeng Gao"]}},"version":2},{"content":{"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["KV Cache Reuse","Position-Independent Cache","LLM Inference","Prefill"]},"primary_area":{"value":"infrastructure, software libraries, hardware, systems, etc."},"abstract":{"value":"Vision-language model (VLM) serving engines redundantly compute and store an image's decoder key-value (KV) cache for every request, even when multiple requests share the same image. Existing solutions fail to fully eliminate redundancy: prefix cache methods reuse KV only when text and positions align exactly, while position-independent approaches still require per-request recomputation, preventing a single physical cache from serving multiple requests. We present Rivet, which builds a request-independent vision KV cache that a serving engine computes once and shares across requests with zero recomputation. For VLMs with multi-dimensional rotary position embeddings (MRoPE), Rivet removes the dependence of vision KV on the request with (1) canonical MRoPE pinning, which replaces the position-dependent offset with a fixed canonical offset, producing identical post-RoPE keys across requests, and (2) sink-preserving isolation that retains the attention sink while blocking context-specific preceding text, making vision-token values request-independent. Together, these techniques produce a vision-KV object that is identical for every request, so one physical copy serves an entire batch through page-table sharing. When a long text precedes the image, Rivet shifts the text positions instead of the image and preserves the accuracy of full recomputation. On Qwen3-VL at 2B, 4B, and 8B scales, Rivet achieves accuracy on par with or even surpassing partial-recomputation baselines such as VLCache and CacheBlend across VQAv2, ChartQA, and TextVQA, while delivering up to 7.4$\\times$ time-to-first-token speedup over full recomputation and 1.4--2.3$\\times$ over previous methods."},"_bibtex":{"value":"@inproceedings{\nanonymous2026rivet,\ntitle={Rivet: Request-Independent Vision {KV} Cache for Zero-Recomputation {VLM} Serving},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=sHauXbtI9b},\nnote={under review}\n}"},"title":{"value":"Rivet: Request-Independent Vision KV Cache for Zero-Recomputation VLM Serving"},"pdf":{"value":"/pdf/9f56ba278a64f2de3a2cc8eea16468e7dea94640.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791226582756,"tcdate":1789135807625,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission14374/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission14374/Authors"],"forum":"sHauXbtI9b","license":"CC BY 4.0","number":14374,"cdate":1789135807625,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/Submission14374/-/Deletion","ICLR.cc/2027/Conference/Submission14374/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791226582756,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"sHauXbtI9b","version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2503.08879v1"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"wang|llms_know_what_to_drop_selfattention_guided_kv_cache_eviction_for_efficient_longcontext_inference"},"html":{"value":"https://doi.org/10.48550/arXiv.2503.08879"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2503-08879,\n publtype={informal},\n author={Guangtao Wang and Shubhangi Upasani and Chen Wu and Darshan Gandhi and Jonathan Li and Changran Hu and Bo Li and Urmish Thakker},\n title={LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference},\n year={2025},\n month={March},\n cdate={1740787200000},\n journal={CoRR},\n volume={abs/2503.08879},\n url={https://doi.org/10.48550/arXiv.2503.08879}\n}\n"},"abstract":{"value":"Efficient long-context inference is critical as large language models (LLMs) adopt context windows of ranging from 128K to 1M tokens. However, the growing key-value (KV) cache and the high computational complexity of attention create significant bottlenecks in memory usage and latency. In this paper, we find that attention in diverse long-context tasks exhibits sparsity, and LLMs implicitly \"know\" which tokens can be dropped or evicted at the head level after the pre-filling stage. Based on this insight, we propose Self-Attention Guided Eviction~(SAGE-KV), a simple and effective KV eviction cache method for long-context inference. After prefilling, our method performs a one-time top-k selection at both the token and head levels to compress the KV cache, enabling efficient inference with the reduced cache. Evaluations on LongBench and three long-context LLMs (Llama3.1-8B-Instruct-128k, Llama3-8B-Prolong-512k-Instruct, and Qwen2.5-7B-Instruct-128k) show that SAGE-KV maintains accuracy comparable to full attention while significantly improving efficiency. Specifically, SAGE-KV achieves 4x higher memory efficiency with improved accuracy over the static KV cache selection method StreamLLM, and 2x higher memory efficiency with better accuracy than the dynamic KV cache selection method Quest."},"title":{"value":"LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference"},"authors":{"value":[{"fullname":"Guangtao Wang","username":""},{"fullname":"Shubhangi Upasani","username":""},{"fullname":"Chen Wu","username":""},{"fullname":"Darshan Gandhi","username":""},{"fullname":"Jonathan Li","username":""},{"fullname":"Changran Hu","username":""},{"fullname":"Bo Li","username":"~Bo_Li61"},{"fullname":"Urmish Thakker","username":""}]}},"tmdate":1791896343810,"pdate":1767139200000,"externalIds":["dblp:journals/corr/abs-2503-08879"],"tcdate":1791896328629,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Bo_Li61"],"forum":"plP6roaxMe","license":"CC BY-SA 4.0","number":177706,"cdate":1740787200000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1791896343810,"domain":"OpenReview.net/Public_Article","id":"plP6roaxMe","version":2},{"content":{"TLDR":{"value":"LLMs is able to identify evictable tokens in long-context inference. We propose a self-attention guided KV compression method to reduce memory and latency while preserving accuracy"},"venue":{"value":"SLLM"},"keywords":{"value":["KV cache eviction","long-context LLM inference","attention sparsity","memory efficiency"]},"abstract":{"value":"Efficient long-context inference is critical as large language models (LLMs) adopt context windows of ranging from 128K to 1M tokens. However, the growing key-value (KV) cache and the high computational complexity of attention create significant bottlenecks in memory usage and latency. In this paper, we find that attention in diverse long-context tasks exhibits sparsity, and LLMs implicitly ``know\" which tokens can be dropped or evicted at the head level after the pre-filling stage. Based on this insight, we propose Self-Attention Guided Eviction~(SAGE-KV), a simple and effective KV eviction cache method for long-context inference. After prefilling, our method performs a one-time top-k selection at both the token and head levels to compress the KV cache, enabling efficient inference with the reduced cache. Evaluations on LongBench, InfiniteBench and three long-context LLMs (Llama3.1-8B-Instruct-128k, Llama3-8B-Prolong-512k-Instruct, and Qwen2.5-7B-Instruct-128k) show that SAGE-KV maintains accuracy comparable to full attention while significantly improving efficiency. Specifically, SAGE-KV achieves 4x higher memory efficiency with improved accuracy over the static KV cache selection method StreamLLM, and 2x higher memory efficiency with better accuracy than the dynamic KV cache selection method Quest."},"_bibtex":{"value":"@inproceedings{\nwang2025llms,\ntitle={{LLM}s Know What to Drop: Self-Attention Guided {KV} Cache Eviction for Efficient Long-Context Inference},\nauthor={Guangtao Wang and Shubhangi Upasani and Chen Wu and Darshan Gandhi and Jonathan Lingjie Li and Changran Hu and Bo Li and Urmish Thakker},\nbooktitle={Sparsity in LLMs (SLLM): Deep Dive into Mixture of Experts, Quantization, Hardware, and Inference},\nyear={2025},\nurl={https://openreview.net/forum?id=qg9dlCcNzr}\n}"},"title":{"value":"LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference"},"Anonymization":{"value":"This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs."},"pdf":{"value":"/pdf/7ea8e60c50b2eddc784c7c30f5c43e000ba45223.pdf"},"venueid":{"value":"ICLR.cc/2025/Workshop/SLLM"},"paperhash":{"value":"wang|llms_know_what_to_drop_selfattention_guided_kv_cache_eviction_for_efficient_longcontext_inference"},"authorids":{"value":["~Guangtao_Wang1","~Shubhangi_Upasani1","~Chen_Wu1","~Darshan_Gandhi1","~Jonathan_Lingjie_Li1","~Changran_Hu1","~Bo_Li43","~Urmish_Thakker1"]},"Track":{"value":"long paper (up to 4 pages)"},"authors":{"value":["Guangtao Wang","Shubhangi Upasani","Chen Wu","Darshan Gandhi","Jonathan Lingjie Li","Changran Hu","Bo Li","Urmish Thakker"]}},"tmdate":1744222019935,"pdate":1741181469852,"tcdate":1738427284375,"writers":["ICLR.cc/2025/Workshop/SLLM","ICLR.cc/2025/Workshop/SLLM/Submission11/Authors"],"signatures":["ICLR.cc/2025/Workshop/SLLM/Submission11/Authors"],"forum":"qg9dlCcNzr","license":"CC BY 4.0","number":11,"cdate":1738427284375,"readers":["everyone"],"invitations":["ICLR.cc/2025/Workshop/SLLM/-/Submission","ICLR.cc/2025/Workshop/SLLM/-/Post_Submission","ICLR.cc/2025/Workshop/SLLM/-/Edit","ICLR.cc/2025/Workshop/SLLM/Submission11/-/Camera_Ready"],"mdate":1744222019935,"odate":1741196734220,"domain":"ICLR.cc/2025/Workshop/SLLM","id":"qg9dlCcNzr","version":2},{"content":{"venue":{"value":"ACL Main Conference"},"venueid":{"value":"OpenReview.net/Archive"},"paperhash":{"value":"zhang|mixkvq_queryaware_mixedprecision_kv_cache_quantization_for_longcontext_reasoning"},"authorids":{"value":["~Tao_Zhang44","~Ziqian_Zeng1","~Hao_Peng7","~Huiping_Zhuang2","~Cen_Chen4"]},"abstract":{"value":"Long Chain-of-Thought (CoT) reasoning has significantly advanced the capabilities of Large Language Models (LLMs), but this progress is accompanied by substantial memory and latency overhead from the extensive Key-Value (KV) cache. Although KV cache quantization is a promising compression technique, existing low-bit quantization methods often exhibit severe performance degradation on complex reasoning tasks. Fixed-precision quantization struggles to handle outlier channels in the key cache, while current mixed-precision strategies fail to accurately identify components requiring high-precision representation. We find that an effective low-bit KV cache quantization strategy must consider two factors: a key channel's intrinsic quantization difficulty and its relevance to the query. Based on this insight, we propose MixKVQ, a novel plug-and-play method that introduces a lightweight, query-aware algorithm to identify and preserve critical key channels that need higher precision, while applying per-token quantization for value cache. Experiments on complex reasoning datasets demonstrate that our approach significantly outperforms existing low-bit methods, achieving performance comparable to a full-precision baseline at a substantially reduced memory footprint."},"title":{"value":"MixKVQ: Query-Aware Mixed-Precision KV Cache Quantization for Long-Context Reasoning"},"authors":{"value":["Tao Zhang","Ziqian Zeng","Hao Peng","Huiping Zhuang","Cen Chen"]}},"tmdate":1790237924864,"pdate":1775318400000,"tcdate":1790237874620,"writers":["~Tao_Zhang44","~Ziqian_Zeng1","~Hao_Peng7","~Huiping_Zhuang2","~Cen_Chen4"],"signatures":["~Tao_Zhang44"],"forum":"v5gTSema4J","license":"CC BY 4.0","number":54708,"cdate":1790237874620,"readers":["everyone"],"invitations":["OpenReview.net/Archive/-/Direct_Upload"],"mdate":1790237924864,"domain":"OpenReview.net/Archive","id":"v5gTSema4J","version":2},{"content":{"TLDR":{"value":"We propose ZSMerge, a zero-shot KV cache compression method that achieves 82% memory reduction and 3x throughput improvement for long-context LLMs without performance degradation or model retraining."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Large Language Models","KV Cache Compression","Memory Efficiency"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"The linear growth of key-value (KV) cache memory and quadratic computational complexity in attention mechanisms pose significant bottlenecks for large language models (LLMs) in long-context processing. While existing KV cache optimization methods address these challenges through token pruning or feature merging, they often incur irreversible information loss or require costly retraining. To this end, we propose ZSMerge, a dynamic KV cache compression framework designed for efficient cache management, featuring three key operations: (1) fine-grained memory allocation guided by multi-dimensional token importance metrics at head-level granularity, (2) a residual merging mechanism that preserves critical context through compensated attention scoring, and (3) a zero-shot adaptation mechanism compatible with diverse LLM architectures without requiring retraining. ZSMerge significantly enhances memory efficiency and inference speed. When applied to LLaMA2-7B, it demonstrates a 20:1 compression ratio for key-value cache retention (reducing memory footprint to 5% of baseline) while sustaining generation quality and achieving a 2.25× throughput improvement at extreme 54k-token contexts, eliminating out-of-memory failures. The code is available at https://anonymous.4open.science/r/ZSMerge-FC36."},"_bibtex":{"value":"@misc{\nliu2026zsmerge,\ntitle={{ZSM}erge: Zero-Shot {KV} Cache Compression for Memory-Efficient Long-Context {LLM}s},\nauthor={Xin LIU and Xudong Wang and Pei Liu and Guoming Tang},\nyear={2026},\nurl={https://openreview.net/forum?id=WJ5BIKfc5A}\n}"},"title":{"value":"ZSMerge: Zero-Shot KV Cache Compression for Memory-Efficient Long-Context LLMs"},"pdf":{"value":"/pdf/f7d895632021830af7230b939df7c05b1482b78c.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"liu|zsmerge_zeroshot_kv_cache_compression_for_memoryefficient_longcontext_llms"},"authorids":{"value":["~Xin_LIU63","~Xudong_Wang10","~Pei_Liu7","~Guoming_Tang1"]},"authors":{"value":["Xin LIU","Xudong Wang","Pei Liu","Guoming Tang"]}},"tmdate":1770805034737,"tcdate":1758252057985,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission15503/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission15503/Authors"],"forum":"WJ5BIKfc5A","license":"CC BY 4.0","number":15503,"cdate":1758252057985,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/-/Submission","ICLR.cc/2026/Conference/-/Post_Submission","ICLR.cc/2026/Conference/Submission15503/-/Full_Submission","ICLR.cc/2026/Conference/Submission15503/-/Rebuttal_Revision","ICLR.cc/2026/Conference/-/Edit"],"mdate":1770805034737,"odate":1759896705795,"domain":"ICLR.cc/2026/Conference","id":"WJ5BIKfc5A","version":2},{"content":{"venue":{"value":"ICLR 2027 Conference Desk Rejected Submission"},"keywords":{"value":["Autoregressive Video Generation","KV Cache"]},"primary_area":{"value":"generative models"},"abstract":{"value":"Autoregressive (AR) video generation has emerged as a promising paradigm for long-horizon video synthesis, where each frame is generated conditioned on previously generated tokens. To accelerate inference, the KV cache is used to avoid redundant recomputation across generation steps. Nevertheless, its growth with generation length introduces increasing memory and error accumulation, limiting the scalability of AR models to even longer sequences. Existing KV cache compression methods mitigate this issue by selectively retaining only video tokens deemed important. However, most existing methods assess token importance using short-horizon signals derived from the current or historical generation context, making these methods prone to overlooking tokens that appear unimportant at early steps but later become critical for future frames. In this work, we identify an important property of trained AR video models: although RoPE-modulated queries evolve across autoregressive steps, the underlying canonical pre-RoPE query distribution remains remarkably stable throughout the video generation process. This approximate stationarity implies that future query distributions are estimable from historical statistics, enabling principled future-aware cache decisions without any additional training. Building on this insight, we propose Future Forcing, a training-free future-aware KV cache policy for AR video generation. Specifically, Future Forcing first constructs a future query proxy from historical statistics, then scores KV cache tokens by their importance under this proxy, and finally merges redundant token pairs within the affine subspace induced by the future query. Extensive experiments show that Future Forcing improves long-horizon consistency under limited KV caches, achieving up to 1.49 improvement in subject consistency on VBench-Long for 60s generation over existing AR video KV cache policies."},"_bibtex":{"value":"@misc{\nanonymous2026future,\ntitle={Future Forcing: Future-aware Training-free {KV} Cache Policy for Autoregressive Video Generation},\nauthor={Anonymous},\nyear={2026},\nurl={https://openreview.net/forum?id=z6I3thBB6N}\n}"},"title":{"value":"Future Forcing: Future-aware Training-free KV Cache Policy for Autoregressive Video Generation"},"pdf":{"value":"/pdf/4f0b8e97a528c9164fa7622f0fe46698cb1e8fd1.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Desk_Rejected_Submission"}},"tmdate":1792254761988,"tcdate":1789311623713,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission16163/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission16163/Authors"],"forum":"z6I3thBB6N","license":"CC BY 4.0","number":16163,"cdate":1789311623713,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission16163/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit","ICLR.cc/2027/Conference/-/Desk_Rejected_Submission"],"mdate":1792254761988,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"z6I3thBB6N","version":2},{"content":{"venue":{"value":"CoRR 2026"},"pdf":{"value":"https://arxiv.org/pdf/2605.30083v1"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"luo|future_forcing_futureaware_trainingfree_kv_cache_policy_for_autoregressive_video_generation"},"html":{"value":"https://doi.org/10.48550/arXiv.2605.30083"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2605-30083,\n publtype={informal},\n author={Jiayi Luo and Qiyan Liu and Tengyang Wang and JunHao Liu and Jiayu Chen and Cong Wang and Hanxin Zhu and Chen Gao and Xiaobin Hu and Qingyun Sun and Zhibo Chen},\n title={Future Forcing: Future-aware Training-free KV Cache Policy for Autoregressive Video Generation},\n year={2026},\n month={May},\n cdate={1777593600000},\n journal={CoRR},\n volume={abs/2605.30083},\n url={https://doi.org/10.48550/arXiv.2605.30083}\n}\n"},"abstract":{"value":"Autoregressive (AR) video generation has emerged as a promising paradigm for long-horizon video synthesis, where each frame is generated conditioned on previously generated tokens. To accelerate inference, the KV cache is used to avoid redundant recomputation across generation steps. Nevertheless, its growth with generation length introduces increasing memory and error accumulation, limiting the scalability of AR models to even longer sequences. Existing KV cache compression methods mitigate this issue by selectively retaining only video tokens deemed important. However, most existing methods assess token importance using short-horizon signals derived from the current or historical generation context, making these methods prone to overlooking tokens that appear unimportant at early steps but later become critical for future frames. In this work, we identify an important property of trained AR video models: although RoPE-modulated queries evolve across autoregressive steps, the underlying canonical pre-RoPE query distribution remains remarkably stable throughout the video generation process. This approximate stationarity implies that future query distributions are estimable from historical statistics, enabling principled future-aware cache decisions without any additional training. Building on this insight, we propose Future Forcing, a training-free future-aware KV cache policy for AR video generation. Specifically, Future Forcing first constructs a future query proxy from historical statistics, then scores KV cache tokens by their importance under this proxy, and finally merges redundant token pairs within the affine subspace induced by the future query. Extensive experiments show that Future Forcing improves long-horizon consistency under limited KV caches, achieving up to 1.49 improvement in subject consistency on VBench-Long for 60s generation over existing AR video KV cache policies."},"title":{"value":"Future Forcing: Future-aware Training-free KV Cache Policy for Autoregressive Video Generation"},"authors":{"value":[{"fullname":"Jiayi Luo","username":"~Jiayi_Luo1"},{"fullname":"Qiyan Liu","username":""},{"fullname":"Tengyang Wang","username":""},{"fullname":"JunHao Liu","username":""},{"fullname":"Jiayu Chen","username":""},{"fullname":"Cong Wang","username":""},{"fullname":"Hanxin Zhu","username":""},{"fullname":"Chen Gao","username":""},{"fullname":"Xiaobin Hu","username":""},{"fullname":"Qingyun Sun","username":"~Qingyun_Sun2"},{"fullname":"Zhibo Chen","username":""}]}},"tmdate":1784621874884,"pdate":1798675200000,"externalIds":["dblp:journals/corr/abs-2605-30083"],"tcdate":1784621614373,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Jiayi_Luo1"],"forum":"B5m49VhXxu","license":"CC BY-SA 4.0","number":80739,"cdate":1777593600000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit","OpenReview.net/Public_Article/-/Authorship_Claim"],"mdate":1784621874884,"domain":"OpenReview.net/Public_Article","id":"B5m49VhXxu","version":2},{"content":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Lossless Compression","KV cache","LLM","Speculative Decoding"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Lossy KV cache compression is a well-explored subfield of machine learning efficiency, with improved latency being one of its major gains. However, lossy compression techniques can fumble from time to time, exhibiting various — and often catastrophic — failure patterns that are not only difficult to resolve but sometimes even hard to identify in the first place, making the direct deployment of models with compressed KV cache a risky endeavor. In this work, we explore a way to preserve lossless generation quality while still benefiting from the acceleration provided by attending only to a compressed KV cache. Specifically, we draw inspiration from the n-gram candidate pool decoding paradigm pioneered by Lookahead Decoding — a largely overlooked and underdeveloped way to achieve efficient yet lossless decoding — where we purposely allow the model to Fumble Around with compressed KV cache to generate multiple lossy ''n-gram guesses'' with just one forward pass, while Find Out via lossless verification in the same forward pass in truly parallel fashion. From a conceptual standpoint, our proposed framework is compatible with all typical static or dynamic KV cache compression methods from the token dropping realm, thus opening up a new avenue for the stagnant n-gram decoding paradigm. Practically, we show that — with careful system support — this framework presents many useful traits that similar draftless baselines (e.g., Self-Speculative Decoding) simply cannot achieve, such as requiring only one set of KV cache and being far less sensitive to model, task, and input-length scenarios. Our comprehensive empirical results show FAFO provides 1.20-2.71x latency speedup over the original model, while consistently outperforming other lossless + draftless solutions by a large margin."},"_bibtex":{"value":"@misc{\nle2026fafo,\ntitle={{FAFO}: Lossy {KV} Cache Compression for Lossless Inference Acceleration via Draftless Fumble Decoding},\nauthor={Hoang Anh Duy Le and Shaochen Zhong and Yifan Lu and Yingtong Dou and Jiayi Yuan and Yu-Neng Chuang and Xiran Fan and Guanchu Wang and Yuzhong Chen and Xia Hu},\nyear={2026},\nurl={https://openreview.net/forum?id=oSk9tP5Mgs}\n}"},"title":{"value":"FAFO: Lossy KV Cache Compression for Lossless Inference Acceleration via Draftless Fumble Decoding"},"pdf":{"value":"/pdf/491f077ab86c5e61bbd4e473081e2322aaeae17e.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"le|fafo_lossy_kv_cache_compression_for_lossless_inference_acceleration_via_draftless_fumble_decoding"},"authorids":{"value":["~Hoang_Anh_Duy_Le1","~Shaochen_Zhong1","~Yifan_Lu8","~Yingtong_Dou1","~Jiayi_Yuan1","~Yu-Neng_Chuang1","~Xiran_Fan1","~Guanchu_Wang1","~Yuzhong_Chen2","~Xia_Hu4"]},"authors":{"value":["Hoang Anh Duy Le","Shaochen Zhong","Yifan Lu","Yingtong Dou","Jiayi Yuan","Yu-Neng Chuang","Xiran Fan","Guanchu Wang","Yuzhong Chen","Xia Hu"]}},"tmdate":1770804996959,"tcdate":1758217219607,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission13383/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission13383/Authors"],"forum":"oSk9tP5Mgs","license":"CC BY 4.0","number":13383,"cdate":1758217219607,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/-/Submission","ICLR.cc/2026/Conference/-/Post_Submission","ICLR.cc/2026/Conference/Submission13383/-/Full_Submission","ICLR.cc/2026/Conference/Submission13383/-/Rebuttal_Revision","ICLR.cc/2026/Conference/-/Edit"],"mdate":1770804996959,"odate":1759896705795,"domain":"ICLR.cc/2026/Conference","id":"oSk9tP5Mgs","version":2},{"content":{"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["KV-Cache","Low-Rank Compression","Large Language Model"]},"supplementary_material":{"value":"/attachment/40e81a6b91524415dc57b5e0efa1e9f64f750a5a.zip"},"primary_area":{"value":"infrastructure, software libraries, hardware, systems, etc."},"abstract":{"value":"Long-context inference makes the key-value (KV) cache a major memory bottleneck for large language models. We propose BELT, a low-rank KV cache compression framework. Its key-side objective targets attention distributions rather than raw keys or pre-softmax logits, yielding a quadratic surrogate with a closed-form low-rank solution. A learned global rank allocator uses differentiable soft selection during training and hard selection at deployment to meet a user-specified component budget. Experiments demonstrate improved model quality under aggressive compression and increased decoding throughput at comparable accuracy. Evaluated across three LLMs and multiple benchmarks, BELT achieves up to 70\\% low-rank KV cache compression while outperforming the strongest baseline by up to 23.3 points on LongBench, and up to $17.8\\times$ overall KV cache reduction when combined with quantization. With Triton-based GPU kernels and a quantized cache, BELT delivers up to $2.7\\times$ speedup for the attention module and $3.3\\times$ end-to-end decoding throughput."},"_bibtex":{"value":"@inproceedings{\nanonymous2026belt,\ntitle={{BELT} : {BE}havior-Aligned Low-Rank {KV} Cache Compression with Learned Top-k Rank Allocation},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=PCAODQ61Qw},\nnote={under review}\n}"},"title":{"value":"BELT : BEhavior-Aligned Low-Rank KV Cache Compression with Learned Top-k Rank Allocation"},"pdf":{"value":"/pdf/02259c117b980a90009875c7b4490599bd57e38c.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791219292887,"tcdate":1787192975219,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission326/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission326/Authors"],"forum":"PCAODQ61Qw","license":"CC BY 4.0","number":326,"cdate":1787192975219,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission326/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791219292887,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"PCAODQ61Qw","version":2},{"content":{"venue":{"value":"ICML 2024 Poster"},"abstract":{"value":"Many computational factors limit broader deployment of large language models. In this paper, we focus on a memory bottleneck imposed by the key-value (KV) cache, a computational shortcut that requires storing previous KV pairs during decoding. While existing KV cache methods approach this problem by pruning or evicting large swaths of relatively less important KV pairs to dramatically reduce the memory footprint of the cache, they can have limited success in tasks that require recollecting a majority of previous tokens. To alleviate this issue, we propose LESS, a simple integration of a (nearly free) constant sized cache with eviction-based cache methods, such that all tokens can be queried at later decoding steps. Its ability to retain information throughout time shows merit on a variety of tasks where we demonstrate LESS can help reduce the performance gap from caching everything, sometimes even matching it, all while being efficient. Relevant code can be found at https://github.com/hdong920/LESS."},"_bibtex":{"value":"@inproceedings{\ndong2024get,\ntitle={Get More with {LESS}: Synthesizing Recurrence with {KV} Cache Compression for Efficient {LLM} Inference},\nauthor={Harry Dong and Xinyu Yang and Zhenyu Zhang and Zhangyang Wang and Yuejie Chi and Beidi Chen},\nbooktitle={Forty-first International Conference on Machine Learning},\nyear={2024},\nurl={https://openreview.net/forum?id=uhHDhVKFMW}\n}"},"title":{"value":"Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference"},"pdf":{"value":"/pdf/040e999f871b691e215c0db6d9ee1f77062598b7.pdf"},"venueid":{"value":"ICML.cc/2024/Conference"},"paperhash":{"value":"dong|get_more_with_less_synthesizing_recurrence_with_kv_cache_compression_for_efficient_llm_inference"},"authorids":{"value":["~Harry_Dong1","~Xinyu_Yang4","~Zhenyu_Zhang4","~Zhangyang_Wang1","~Yuejie_Chi1","~Beidi_Chen1"]},"authors":{"value":["Harry Dong","Xinyu Yang","Zhenyu Zhang","Zhangyang Wang","Yuejie Chi","Beidi Chen"]}},"tmdate":1719287280456,"pdate":1714610460441,"tcdate":1706855672628,"writers":["ICML.cc/2024/Conference","ICML.cc/2024/Conference/Submission8085/Authors"],"signatures":["ICML.cc/2024/Conference/Submission8085/Authors"],"forum":"uhHDhVKFMW","license":"CC BY 4.0","number":8085,"cdate":1706855672628,"readers":["everyone"],"invitations":["ICML.cc/2024/Conference/-/Submission","ICML.cc/2024/Conference/-/Post_Submission","ICML.cc/2024/Conference/-/Edit","ICML.cc/2024/Conference/Submission8085/-/Camera_Ready_Revision"],"mdate":1719287280456,"odate":1717693087575,"domain":"ICML.cc/2024/Conference","id":"uhHDhVKFMW","version":2},{"content":{"summary":{"value":"Large scale chain of though RL runs with full cache make it memory and compute intensive. Simple solution is drop earlier context once cache limit hit or sliding window cache but authors show this comes at cost reasoning peformance. Further they propose to learn a global query vector $q_g$ from these evicted tokens KVs which can function like a memory. This $q_g$ is updated with a simple self-attention like update rule and then injected into a lightweight Lora Adapter via a small weight update $ \\Delta W$. The model parameters now through these remember the evicted tokens. This adds minimal computational cost but allows scaling up RL by fixing cache size. They experiment with DAPO recipe on top of qwen-2.4 4/7B models and DeepSeek-R1-Distill-Llama-8B and show reasonable improvements over slidding window cache baseline."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"- What do the authors think is the impact of the cache eviction strategy on the q_g update would be. Eg bulk eviction or one at a time.\n- Contribution [1] : Is that a novel contribution or already established in the community. Authors should clarify\n- What do authors think about mamba hybrid models like nano-v2-12b https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-12B-v2 as they are optimized for inference. Its a hybrid model so cache still grows up but not to the same level. Also they will still able to model the long reasoning context much better.\n- Comments on writing\n - In line 196 Table 1 is mentioned but we don’t know what lora_c is and its not explained so confusing to read\n - line 220 is not coherent\n - global query is not well introduced as being the latent that learns the evicted context."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"- The idea of encoding the “to be evicted token” into latent vectors which are them embedded into LoRA adapter is very novel and clever. This makes the Lora adapter like a learned dynamic memory without blowing up KV cache linearly\n- Well formulated and simple idea with clear experiments\n- Good ablations and analysis"},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- Results and experimental setup is not convincing enough.\n- Unclear if it will apply to full finetuned models compared to the LoRA only setting in this paper. Also unclear if this will translate over to larger models but I understand that will be outside the scope of this study.\n- max sequence length of 3072 is not enough for reasoning RL runs and doesn’t give me confidence in the results especially given the authors tout this as benefitial for long reasoning RL. Only at longer context lengths the benefits or downside of this appraoch can be demonstrated.\n- AIME25 still has variance at pass@16 so the slight improvment in those benchmarks is not conclusive\n- Its also unclear why this approach should be benefitial over full-cache setup especially given the short max 6k context experiments. At larger contexts I can see how longer context can hurt sometimes. More thorough investigation is needed.\n- $q_g$ is just one fixed vector for the whole evicted sequence. How much capacity can it even have? As we scale up the reasoning RL which can go upto 64k sequence length in frontier RL runs, I am not convinced this $q_g$ can hold that long context well. Ofc it can be better than no context of evicted tokens in sliding window cache but unclear how much we will lose compared to full cache."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1764358395400,"tcdate":1761980644779,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission4473/Reviewer_MbcZ"],"signatures":["ICLR.cc/2026/Conference/Submission4473/Reviewer_MbcZ"],"forum":"q4iJxp47CT","number":2,"license":"CC BY 4.0","cdate":1761980644779,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission4473/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1764358395400,"domain":"ICLR.cc/2026/Conference","replyto":"q4iJxp47CT","id":"4t8rlMA2y1","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["efficient reasoning LLM","KV cache","test-time learning"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Large reasoning models (LRMs) excel on complex problems but face a critical barrier to efficiency: reinforcement learning (RL) training requires long rollouts for outcome-based rewards, where autoregressive decoding dominates time and memory usage. While sliding-window cache strategies can bound memory, they disrupt long-context reasoning and degrade performance. We introduce Progressive Thought Encoding, a parameter-efficient fine-tuning method that enables LRMs to reason effectively under fixed-size caches. By progressively encoding intermediate reasoning into compact representations, our approach eliminates the need to backpropagate through full-cache rollouts, thereby reducing training-time memory usage, while maintaining constant memory during inference. Experiments on three models, including Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct, and DeepSeek-R1-Distill-Llama-8B, across six widely used challenging mathematical benchmarks show consistent gains: our method achieves +19.3\\% improvement over LoRA and +29.9\\% over the baseline on average, with up to +23.4 absolute gains on AIME2024/2025 under tight cache budgets. These results demonstrate that Progressive Thought Encoding not only improves reasoning accuracy but also makes RL training of LRMs substantially more efficient and scalable under real-world memory constraints."},"_bibtex":{"value":"@inproceedings{\nzhang2026training,\ntitle={Training Large Reasoning Models Efficiently via Progressive Thought Encoding},\nauthor={Zeliang Zhang and Xiaodong Liu and Hao Cheng and Hao Sun and Chenliang Xu and Jianfeng Gao},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=q4iJxp47CT}\n}"},"title":{"value":"Training Large Reasoning Models Efficiently via Progressive Thought Encoding"},"pdf":{"value":"/pdf/472daff6ce690a46c5547f472b9e61b2e663a46f.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"zhang|training_large_reasoning_models_efficiently_via_progressive_thought_encoding"},"authorids":{"value":["~Zeliang_Zhang1","~Xiaodong_Liu1","~Hao_Cheng4","~Hao_Sun6","~Chenliang_Xu1","~Jianfeng_Gao1"]},"authors":{"value":["Zeliang Zhang","Xiaodong Liu","Hao Cheng","Hao Sun","Chenliang Xu","Jianfeng Gao"]}},"version":2},{"content":{"TLDR":{"value":"We introduce a training-free KV cache consolidation policy guided by attention response distortion to sustain autoregressive long-video generation quality under a fixed cache budget."},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["Autoregressive Video Generation","Long Video Generation","KV Cache Compression","Training-Free Inference"]},"primary_area":{"value":"generative models"},"abstract":{"value":"In autoregressive video generation, retaining all historical key–value (KV) states increases memory usage and per-step attention costs as videos grow longer. A fixed cache budget bounds these costs but raises the challenge of preserving historical content that supports consistent, high-quality generation. Existing methods control cache size through history selection or consolidation, yet deciding what to retain and how to limit the loss of useful content remains a shared challenge. Our analysis shows that spatial queries depend on different historical content and that higher current attention coverage does not necessarily yield better long-video quality. We introduce FrameFold, a training-free KV cache consolidation method that compares the content retrieved by the same recent historical queries before and after merging to guide both representative construction and merge selection. Exploiting temporal redundancy between adjacent historical entries, FrameFold merges their KV states into shared representatives while retaining complete spatial grids. It fits value-merging coefficients in closed form by matching conditional attention responses, then selects the merge with the lowest remaining response distortion. Under matched KV cache budgets, experiments on Self Forcing and Causal Forcing at 30 and 60 seconds and on LongLive at 120 seconds demonstrate improved consistency and visual quality. Continuous eight-minute Self Forcing rollouts further show less quality degradation than the baselines."},"_bibtex":{"value":"@inproceedings{\nanonymous2026framefold,\ntitle={FrameFold: Response-Guided {KV} Cache Consolidation for Autoregressive Long Video Generation},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=VnRf54qkjE},\nnote={under review}\n}"},"title":{"value":"FrameFold: Response-Guided KV Cache Consolidation for Autoregressive Long Video Generation"},"pdf":{"value":"/pdf/4124b78ca37aae523b9ad063a9319fd951ecb84c.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791219927674,"tcdate":1787554794111,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission3900/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission3900/Authors"],"forum":"VnRf54qkjE","license":"CC BY 4.0","number":3900,"cdate":1787554794111,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Edit","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission3900/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing"],"mdate":1791219927674,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"VnRf54qkjE","version":2},{"content":{"venue":{"value":"EACL (Student Research Workshop) 2026"},"pdf":{"value":"https://aclanthology.org/2026.eacl-srw.11.pdf"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"zhang|thesis_proposal_efficient_kv_cache_reuse_for_multidocument_retrievalaugmented_generation"},"html":{"value":"https://doi.org/10.18653/v1/2026.eacl-srw.11"},"_bibtex":{"value":"@inproceedings{DBLP:conf/eacl/ZhangI26,\n author={Zhipeng Zhang and Dmitry Ilvovsky},\n title={Thesis Proposal: Efficient KV Cache Reuse for Multi-Document Retrieval-Augmented Generation},\n year={2026},\n cdate={1767225600000},\n pages={160-169},\n url={https://doi.org/10.18653/v1/2026.eacl-srw.11},\n booktitle={EACL (Student Research Workshop)},\n crossref={conf/eacl/2026-4}\n}\n"},"abstract":{"value":"Retrieval-Augmented Generation (RAG) systems face efficiency bottlenecks in prefill due to attention mechanism, and traditional KV cache only accelerates decoding. In this context, reusing document-level KV cache computed for retrieved documents in previous sessions during the prefill stage appears to be a natural way to amortize computation, but it raises serious correctness challenges due to position and context misalignment across queries and sessions. This research proposes a multi-document KV cache reuse framework for multi-document RAG workloads across queries and sessions to resolve position misalignment and context misalignment, preserving accuracy while eliminating document-specific quadratic complexity in prefill. Theoretical analysis will establish conditions under which multi-document KV cache reuse remains stable and close to full recomputation, providing principled guarantees for both efficiency and accuracy. These results will enable deployment in existing RAG pipelines without architectural changes or model retraining. Crucially, to ensure robustness in real-world deployments, validation will extend beyond standard benchmarks to include noise-robustness tests and domain-specific workloads (e.g., legal). The research aims to empirically confirm these guarantees and demonstrate that substantial prefill speedups can be achieved without materially degrading task-level performance."},"title":{"value":"Thesis Proposal: Efficient KV Cache Reuse for Multi-Document Retrieval-Augmented Generation"},"authors":{"value":[{"fullname":"Zhipeng Zhang","username":"~Zhipeng_Zhang2"},{"fullname":"Dmitry Ilvovsky","username":""}]}},"tmdate":1784097487906,"pdate":1798675200000,"externalIds":["dblp:conf/eacl/ZhangI26"],"tcdate":1784097469267,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Zhipeng_Zhang2"],"forum":"IxAcksgml2","license":"CC BY-SA 4.0","number":55958,"cdate":1767225600000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1784097487906,"domain":"OpenReview.net/Public_Article","id":"IxAcksgml2","version":2},{"content":{"venue":{"value":"NeurIPS 2025 poster"},"TLDR":{"value":"We propose a projection-based scoring function in KV cache eviction for LLM acceleration."},"keywords":{"value":["Large Language Models","KV Cache Eviction","LLM Inference Acceleration"]},"primary_area":{"value":"deep_learning"},"abstract":{"value":"Key-Value (KV) cache eviction---which retains the KV pairs of the most important tokens while discarding less important ones---is a critical technique for optimizing both memory usage and inference latency in large language models (LLMs).\nHowever, existing approaches often rely on simple heuristics---such as attention weights---to measure token importance, overlooking the spatial relationships between token value states in the vector space.\nThis often leads to suboptimal token selections and thus performance degradation.\nTo tackle this problem, we propose a novel method, namely **AnDPro** (**An**chor **D**irection **Pro**jection), which introduces a projection-based scoring function to more accurately measure token importance.\nSpecifically, AnDPro operates in the space of value vectors and leverages the projections of these vectors onto an *``Anchor Direction''*---the direction of the pre-eviction output---to measure token importance and guide more accurate token selection.\nExperiments on $16$ datasets from the LongBench benchmark demonstrate that AnDPro can maintain $96.07\\\\%$ of the full cache accuracy using only $3.44\\\\%$ KV cache budget, reducing KV cache budget size by $46.0\\\\%$ without compromising quality compared to previous state-of-the-arts."},"_bibtex":{"value":"@inproceedings{\ngeng2025accurate,\ntitle={Accurate {KV} Cache Eviction via Anchor Direction Projection for Efficient {LLM} Inference},\nauthor={Zijie Geng and Jie Wang and Ziqi Liu and Feng Ju and Yiming Li and Xing Li and Mingxuan Yuan and Jianye HAO and Defu Lian and Enhong Chen and Feng Wu},\nbooktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},\nyear={2025},\nurl={https://openreview.net/forum?id=Tdl89SZItB}\n}"},"title":{"value":"Accurate KV Cache Eviction via Anchor Direction Projection for Efficient LLM Inference"},"pdf":{"value":"/pdf/27e5165f6a6821bbd2a496ddaa625c6b9de5e57e.pdf"},"venueid":{"value":"NeurIPS.cc/2025/Conference"},"paperhash":{"value":"geng|accurate_kv_cache_eviction_via_anchor_direction_projection_for_efficient_llm_inference"},"authorids":{"value":["~Zijie_Geng1","~Jie_Wang1","~Ziqi_Liu9","~Feng_Ju5","~Yiming_Li21","~Xing_Li6","~Mingxuan_Yuan1","~Jianye_HAO1","~Defu_Lian1","~Enhong_Chen1","~Feng_Wu1"]},"authors":{"value":["Zijie Geng","Jie Wang","Ziqi Liu","Feng Ju","Yiming Li","Xing Li","Mingxuan Yuan","Jianye HAO","Defu Lian","Enhong Chen","Feng Wu"]}},"tmdate":1783627271537,"pdate":1758217557262,"tcdate":1747033258907,"writers":["NeurIPS.cc/2025/Conference","NeurIPS.cc/2025/Conference/Submission27280/Authors"],"signatures":["NeurIPS.cc/2025/Conference/Submission27280/Authors"],"forum":"Tdl89SZItB","license":"CC BY 4.0","number":27280,"cdate":1747033258907,"readers":["everyone"],"invitations":["NeurIPS.cc/2025/Conference/-/Submission","NeurIPS.cc/2025/Conference/-/Post_Submission","NeurIPS.cc/2025/Conference/Submission27280/-/Full_Submission","NeurIPS.cc/2025/Conference/-/Edit","NeurIPS.cc/2025/Conference/Submission27280/-/Camera_Ready_Revision"],"mdate":1783627271537,"odate":1761705092820,"domain":"NeurIPS.cc/2025/Conference","id":"Tdl89SZItB","version":2},{"content":{"TLDR":{"value":"Our work, Mustafar, unlocks 70% sparsity in KV cache pruning by leveraging unstructured sparsity pattern, supported by a custom attention kernel, and boosts the inference efficiency of LLMs."},"venue":{"value":"NeurIPS 2025 poster"},"keywords":{"value":["KV cache compression","Unstructured Pruning","Efficient LLM Inference"]},"supplementary_material":{"value":"/attachment/1a06fd780e4dbf16a537ebe1316d3f2ede271c64.zip"},"primary_area":{"value":"infrastructure"},"abstract":{"value":"We demonstrate that unstructured sparsity significantly improves KV cache compression for LLMs, enabling sparsity levels up to 70\\% without compromising accuracy or requiring fine-tuning. We conduct a systematic exploration of pruning strategies and find per-token magnitude-based pruning as highly effective for both Key and Value caches under unstructured sparsity, surpassing prior structured pruning schemes. The Key cache benefits from prominent outlier elements, while the Value cache surprisingly benefits from a simple magnitude-based pruning despite its uniform distribution. KV cache size is the major bottleneck in decode performance due to high memory overhead for large context lengths. To address this, we use a bitmap-based sparse format and a custom attention kernel capable of compressing and directly computing over compressed caches pruned to arbitrary sparsity patterns, significantly accelerating memory-bound operations in decode computations and thereby compensating for the overhead of runtime pruning and compression. Our custom attention kernel coupled with the bitmap-based format delivers substantial compression of KV cache up to 45\\% of dense inference and thereby enables longer context lengths and increased tokens/sec throughput of up to 2.23$\\times$ compared to dense inference. Our pruning mechanism and sparse attention kernel is available at https://github.com/dhjoo98/mustafar."},"_bibtex":{"value":"@inproceedings{\njoo2025mustafar,\ntitle={{MUSTAFAR}: Promoting Unstructured Sparsity for {KV} Cache Pruning in {LLM} Inference},\nauthor={Donghyeon Joo and Helya Hosseini and Ramyad Hadidi and Bahar Asgari},\nbooktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},\nyear={2025},\nurl={https://openreview.net/forum?id=C69741fMFX}\n}"},"title":{"value":"MUSTAFAR: Promoting Unstructured Sparsity for KV Cache Pruning in LLM Inference"},"pdf":{"value":"/pdf/3f12569e75bc44a8825d64bce1b98b485b326f52.pdf"},"venueid":{"value":"NeurIPS.cc/2025/Conference"},"paperhash":{"value":"joo|mustafar_promoting_unstructured_sparsity_for_kv_cache_pruning_in_llm_inference"},"authorids":{"value":["~Donghyeon_Joo1","~Helya_Hosseini1","~Ramyad_Hadidi1","~Bahar_Asgari1"]},"authors":{"value":["Donghyeon Joo","Helya Hosseini","Ramyad Hadidi","Bahar Asgari"]}},"tmdate":1783627378464,"pdate":1758216961466,"tcdate":1746814866260,"writers":["NeurIPS.cc/2025/Conference","NeurIPS.cc/2025/Conference/Submission13305/Authors"],"signatures":["NeurIPS.cc/2025/Conference/Submission13305/Authors"],"forum":"C69741fMFX","license":"CC BY 4.0","number":13305,"cdate":1746814866260,"readers":["everyone"],"invitations":["NeurIPS.cc/2025/Conference/-/Submission","NeurIPS.cc/2025/Conference/-/Post_Submission","NeurIPS.cc/2025/Conference/Submission13305/-/Full_Submission","NeurIPS.cc/2025/Conference/Submission13305/-/Supplementary_Material","NeurIPS.cc/2025/Conference/-/Edit","NeurIPS.cc/2025/Conference/Submission13305/-/Camera_Ready_Revision"],"mdate":1783627378464,"odate":1761704890315,"domain":"NeurIPS.cc/2025/Conference","id":"C69741fMFX","version":2},{"content":{"summary":{"value":"It is hard to satisfy the latency requirement for Transformer based sequential recommendation models. CollaborativeKV analyzes the collaborative signals in KV caches and divides it into inter-user and intra-user components by SVD. The proposed method achieves better effectiveness and efficiency than state-of-the-art baselines on benchmark video recommendation datasets."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"--Though the cache efficiency improvement is obvious, the effectiveness improvement of CollaborativeKV is relatively small.\n\n--According to the choice of benchmark datasets, it is better to change the title to clarify that the proposed method is designed for video recommendation."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":3},"strengths":{"value":"--CollaborativeKV improve the efficiency of KV cache and at the same time the recommendation accuracy without tradeoff.\n\n--Practically, it solves the urgent latency problem by low-rank factorization for Transformer based sequential recommendation models.\n\n--The massive reduction in storage overhead has been verified on real-world datasets in Table 1."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"-The proposed method highly relies on the user similarities, which might not be effective for applications with non-overlapping user interests.\n\n-The recommendation accuracy in terms of GAUC is increased by 0.005 or so with the increase of the global KV pool size to 10000 according to Figure5(a). This performance improvement cannot be ignored compared with its improvement in Table 1. So how to derive the conclusion in the last passage of Page 8: …a relatively small global KV pool is SUFFICIENT to capture shared KV information…."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762919053329,"tcdate":1762094022218,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission6778/Reviewer_U1XG"],"signatures":["ICLR.cc/2026/Conference/Submission6778/Reviewer_U1XG"],"forum":"NCecQKw1Ni","number":3,"license":"CC BY 4.0","cdate":1762094022218,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission6778/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762919053329,"domain":"ICLR.cc/2026/Conference","replyto":"NCecQKw1Ni","id":"zeSv4fuOys","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["KV cache","Sequential recommendation"]},"primary_area":{"value":"other topics in machine learning (i.e., none of the above)"},"abstract":{"value":"Sequential recommendation models are widely used in applications, yet they face stringent latency requirements. \nMainstream models leverage the Transformer attention mechanism to improve performance, but its computational complexity grows with the sequence length, leading to a latency challenge for long sequences.\nConsequently, KV cache technology has recently been explored in sequential recommendation systems to reduce inference latency.\nHowever, KV cache introduces substantial storage overhead in sequential recommendation systems, which often have a large user base with potentially very long user history sequences.\nIn this work, we observe that KV sequences across different users exhibit significant similarities, indicating the existence of collaborative signals in KV. \nFurthermore, we analyze the KV using singular value decomposition (SVD) and find that the information in KV can be divided into two parts: the majority of the information is shareable across users, while a small portion is user-specific.\nMotivated by this, we propose CollectiveKV, a cross-user KV sharing mechanism.\nIt captures the information shared across users through a learnable global KV pool. During inference, each user retrieves high-dimensional shared KV from the pool and concatenates them with low-dimensional user-specific KV to obtain the final KV.\nExperiments on five sequential recommendation models and three datasets show that our method can compress the KV cache to only 0.8\\% of its original size, while maintaining or even enhancing model performance."},"_bibtex":{"value":"@inproceedings{\nli2026collectivekv,\ntitle={Collective{KV}: Decoupling and Sharing Collaborative Information in Sequential Recommendation},\nauthor={Jingyu Li and Zhaocheng Du and Qianhui Zhu and kaiyuan Li and Zhicheng Zhang and Song-Li Wu and Chaolang Li and Pengwen Dai},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=NCecQKw1Ni}\n}"},"title":{"value":"CollectiveKV: Decoupling and Sharing Collaborative Information in Sequential Recommendation"},"pdf":{"value":"/pdf/9041d4254a1972589c4669f79e3673c72afcce61.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"li|collectivekv_decoupling_and_sharing_collaborative_information_in_sequential_recommendation"},"authorids":{"value":["~Jingyu_Li7","~Zhaocheng_Du1","~Qianhui_Zhu1","~kaiyuan_Li1","~Zhicheng_Zhang5","~Song-Li_Wu1","~Chaolang_Li1","~Pengwen_Dai1"]},"authors":{"value":["Jingyu Li","Zhaocheng Du","Qianhui Zhu","kaiyuan Li","Zhicheng Zhang","Song-Li Wu","Chaolang Li","Pengwen Dai"]}},"version":2},{"content":{"TLDR":{"value":"a coarse-to-fine KV cache pruning framework that progressively prunes redundant KV tokens while preserving fast and efficient computation"},"venue":{"value":"ICLR 2027 Conference Desk Rejected Submission"},"keywords":{"value":["KV Cache Pruning","Unified Multimodal Model","Image Editing"]},"primary_area":{"value":"generative models"},"abstract":{"value":"Unified multimodal models have demonstrated strong capabilities in visual generation and editing, but long multimodal KV caches incur substantial memory and attention costs during generation. We propose a coarse-to-fine KV cache compression framework that progressively prunes redundant KV tokens while controlling the cost of importance estimation. The coarse stage exploits the local similarity of visual queries and their similar preferences over cached tokens. We sample one representative query from each local group to reduce the QK computation required for importance estimation. For stronger compression, the fine stage revisits the shortened cache at a later denoising step using updated, sampled queries. It ranks feature channels by their mean absolute query activations and uses the highest-scoring half for importance estimation, limiting the additional refinement cost. Retained keys and values remain full-dimensional. Experiments on image editing demonstrate favorable quality--efficiency trade-offs across compression levels, with substantial KV cache reductions and competitive editing quality."},"_bibtex":{"value":"@misc{\nanonymous2026queryguided,\ntitle={Query-Guided Coarse-to-Fine {KV} Cache {\\textbackslash}{\\textbackslash} Compression for Unified Multimodal Generation},\nauthor={Anonymous},\nyear={2026},\nurl={https://openreview.net/forum?id=ujpyf0UJs7}\n}"},"title":{"value":"Query-Guided Coarse-to-Fine KV Cache \\\\ Compression for Unified Multimodal Generation"},"pdf":{"value":"/pdf/6025b8949cb9e7735af718711c1676375a0288ef.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Desk_Rejected_Submission"}},"tmdate":1792087710396,"tcdate":1789616289291,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission28070/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission28070/Authors"],"forum":"ujpyf0UJs7","license":"CC BY 4.0","number":28070,"cdate":1789616289291,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission28070/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit","ICLR.cc/2027/Conference/-/Desk_Rejected_Submission"],"mdate":1792087710396,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"ujpyf0UJs7","version":2},{"content":{"summary":{"value":"The paper presents a KV cache optimization method with channel wise pruning for reducing the inference cost under long context scenario. The method is based on the previously proposed efficient context compression method ( H2O, KIVI, and SNAP). Specifically, the paper utilized a query driven method to involve the query for key channel selection."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"See weakness section."},"rating":{"value":5},"details_of_ethics_concerns":{"value":"N/A"},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"The proposed method is simple and straight forward, with direct structured pruning on the key channel.\nThe evaluation shows the pruned results achieves comparable results with the non-pruned baselines"},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"Lack of experiments compared to directly applying to vanilla models.\nWhy the proposed method is basing on the pruned method, instead of directly on the vanilla model? The observations come from the fact of that there are large outliers alongside the channel. This not an unique attribute on the compressed model (i.e. H2O). Yet, experimenting on the compressed model introduces confounding variables which affect the analysis. The current manuscript lacks experiments directly apply the method on the Vanilla model.\n\nIn the meantime, the current manuscript lacks the details about how the K cache selected K cache is removed. This is important in order to consider the real memory benefits. Is the K cache fully removed? Or full of the KV cache are always stored in memory? If the selected channel of the K cache is removed, when new queries come, how to upgrade to the new selected K cache size? If the whole K cache are always stored in memory, how is the memory efficiency comes from?\n\nIt is also worth to indicating whether the compression is done starting at the prefilling phase, or the prefill phase is concrete. The experiments of many the previous approach (i.e. H2O) did not compress during prefill phase. Detailing it will make the understanding of the experimental results more comprehensive.\n\nTypos:\nSec 3.1 line 149 KV cach\nLine 175 k is bold?"}},"nonreaders":[],"tmdate":1731427423473,"tcdate":1730760715340,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission1419/Reviewer_ELRT"],"signatures":["ICLR.cc/2025/Conference/Submission1419/Reviewer_ELRT"],"forum":"n0OtGl6VGb","number":5,"license":"CC BY 4.0","cdate":1730760715340,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission1419/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427423473,"domain":"ICLR.cc/2025/Conference","replyto":"n0OtGl6VGb","id":"cW0lsMLQeT","forumContent":{"venue":{"value":"ICLR 2025 Spotlight"},"TLDR":{"value":"We propose a novel query-dependent KV cache channel pruning method to reduce the memory usage of LLMs during inference."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Large Language Models; KV Cache Compression; KV Cache Pruning"]},"supplementary_material":{"value":"/attachment/32191712fe204d1245d069202f64a6c1981d65f8.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Large Language Models (LLMs) have revolutionized the field of natural language processing, achieving unprecedented performance across a variety of applications. \nHowever, their increased computational and memory demands present significant challenges, especially when handling long sequences.\nThis paper focuses on the long-context scenario, addressing the inefficiencies in KV cache memory consumption during inference. \nUnlike existing approaches that optimize the memory based on the sequence length, we identify substantial redundancy in the channel dimension of the KV cache, as indicated by an uneven magnitude distribution and a low-rank structure in the attention weights.\nIn response, we propose ThinK, a novel query-dependent KV cache pruning method designed to minimize attention weight loss while selectively pruning the least significant channels. Our approach not only maintains or enhances model accuracy but also achieves a reduction in KV cache memory costs by over 20\\% compared with vanilla KV cache eviction and quantization methods. For instance, ThinK integrated with KIVI can achieve $2.8\\times$ peak memory reduction while maintaining nearly the same quality, enabling a batch size increase from 4$\\times$ (with KIVI alone) to 5$\\times$ when using a single GPU. Extensive evaluations on the LLaMA and Mistral models across various long-sequence datasets verified the efficiency of ThinK. Our code has been made available at https://github.com/SalesforceAIResearch/ThinK."},"_bibtex":{"value":"@inproceedings{\nxu2025think,\ntitle={ThinK: Thinner Key Cache by Query-Driven Pruning},\nauthor={Yuhui Xu and Zhanming Jie and Hanze Dong and Lei Wang and Xudong Lu and Aojun Zhou and Amrita Saha and Caiming Xiong and Doyen Sahoo},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=n0OtGl6VGb}\n}"},"title":{"value":"ThinK: Thinner Key Cache by Query-Driven Pruning"},"pdf":{"value":"/pdf/7f5379da15a795ee6f5f3260d6b837fa1771b920.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"xu|think_thinner_key_cache_by_querydriven_pruning"},"authorids":{"value":["~Yuhui_Xu2","~Zhanming_Jie2","~Hanze_Dong1","~Lei_Wang28","~Xudong_Lu1","~Aojun_Zhou2","~Amrita_Saha2","~Caiming_Xiong1","~Doyen_Sahoo1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Yuhui Xu","Zhanming Jie","Hanze Dong","Lei Wang","Xudong Lu","Aojun Zhou","Amrita Saha","Caiming Xiong","Doyen Sahoo"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2407.08454v2"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"wang|model_tells_you_where_to_merge_adaptive_kv_cache_merging_for_llms_on_longcontext_tasks"},"authorids":{"value":["~Zheng_Wang38","https://dblp.org/search/pid/api?q=author:Boxiao_Jin:","https://dblp.org/search/pid/api?q=author:Zhongzhi_Yu:","https://dblp.org/search/pid/api?q=author:Minjia_Zhang:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2407.08454"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2407-08454,\n publtype={informal},\n author={Zheng Wang and Boxiao Jin and Zhongzhi Yu and Minjia Zhang},\n title={Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks},\n year={2024},\n cdate={1704067200000},\n journal={CoRR},\n volume={abs/2407.08454},\n url={https://doi.org/10.48550/arXiv.2407.08454}\n}\n"},"abstract":{"value":"How to efficiently serve Large Language Models (LLMs) has become a pressing issue because of their huge computational cost in their autoregressive generation process. To mitigate computational costs, LLMs often employ the KV Cache technique to improve the generation speed. While improving the computational efficiency, the storage requirements of the KV cache are substantial, particularly in long-context scenarios, leading to significant memory consumption. Existing KV cache eviction methods often degrade the performance of LLMs in long-context scenarios due to the information loss introduced by eviction. In this paper, we propose a novel KV cache merging approach, called KVMerger, to achieve adaptive KV cache compression for long-context tasks without significant performance degradation under constrained memory budgets. Our approach is inspired by the intriguing observation that key states exhibit high similarity at the token level within a single sequence. To facilitate merging, we develop an effective yet straightforward merging set identification algorithm to identify suitable KV states for merging. Our merging set identification algorithm stimulates the second observation that KV cache sparsity, from similarity perspective, is independent of the dataset and remains persistent at the model level. Subsequently, we propose a Gaussian kernel weighted merging algorithm to selectively merge all states within each merging set. We conduct extensive experiments to demonstrate the effectiveness of KVMerger for long-context tasks under constrained memory budgets, applying it to models including Llama2-7B-chat and Llama2-13B-chat. Using the LongBench and ZeroScroll benchmarks, we compare our method with other KV cache compression techniques, including H2O and CaM, showing that our method achieves superior performance across tasks with both 50% and 35% KV cache budgets."},"title":{"value":"Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks"},"authors":{"value":["Zheng Wang","Boxiao Jin","Zhongzhi Yu","Minjia Zhang"]}},"tmdate":1760542918844,"pdate":1704067200000,"externalIds":["dblp:journals/corr/abs-2407-08454"],"tcdate":1760542909491,"writers":["~"],"signatures":["~Zheng_Wang38"],"forum":"DHWQzc5V0d","license":"CC BY-SA 4.0","number":645188,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1760542918844,"domain":"DBLP.org","id":"DHWQzc5V0d","version":2},{"content":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["KV cache compression","LLM","Gaussian Kernel weighted merging"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"How to efficiently serve Large Language Models (LLMs) has become a pressing issue because of their huge computational cost in their autoregressive generation process. To mitigate computational costs, LLMs often employ the KV Cache technique to improve the generation speed. While improving the computational efficiency, the storage requirements of the KV cache are substantial, particularly in long-context scenarios, leading to significant memory consumption. Existing KV cache eviction methods often degrade the performance of LLMs in long-context scenarios due to the information loss introduced by eviction. In this paper, we propose a novel KV cache merging approach, called KVMerger, to achieve adaptive KV cache compression for long-context tasks without significant performance degradation under constrained memory budgets. Our approach is inspired by the intriguing observation that key states exhibit high similarity at the token level within a single sequence. To facilitate merging, we develop an effective yet straightforward merging set identification algorithm to identify suitable KV states for merging. Our merging set identification algorithm stimulates the second observation that KV cache sparsity, from similarity perspective, is independent of the dataset and remains persistent at the model level. Subsequently, we propose a Gaussian kernel weighted merging algorithm to selectively merge all states within each merging set. We conduct extensive experiments to demonstrate the effectiveness of KVMerger for long-context tasks under constrained memory budgets, applying it to models including Llama2-7B/13B-chat and Mistral-7B-instruct across various tasks. We also compare our method with other KV cache compression algorithms, including H2O and CaM, showing that our method achieves superior performance across tasks with different KV cache budgets."},"_bibtex":{"value":"@misc{\nwang2025model,\ntitle={Model Tells You Where to Merge: Adaptive {KV} Cache Merging for {LLM}s on Long-Context Tasks},\nauthor={Zheng Wang and Boxiao Jin and Yuming Chang and Zhongzhi Yu and Minjia Zhang},\nyear={2025},\nurl={https://openreview.net/forum?id=Q5VlpYRxGF}\n}"},"title":{"value":"Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks"},"pdf":{"value":"/pdf/0dcebcc2f9ff67d358168b011e130f9826097cb7.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"wang|model_tells_you_where_to_merge_adaptive_kv_cache_merging_for_llms_on_longcontext_tasks"},"authorids":{"value":["~Zheng_Wang38","~Boxiao_Jin1","~Yuming_Chang2","~Zhongzhi_Yu1","~Minjia_Zhang1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Zheng Wang","Boxiao Jin","Yuming Chang","Zhongzhi Yu","Minjia Zhang"]}},"tmdate":1738735858747,"tcdate":1727482059013,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission12608/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission12608/Authors"],"forum":"Q5VlpYRxGF","license":"CC BY 4.0","number":12608,"cdate":1727482059013,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/-/Submission","ICLR.cc/2025/Conference/-/Post_Submission","ICLR.cc/2025/Conference/Submission12608/-/Full_Submission","ICLR.cc/2025/Conference/Submission12608/-/Rebuttal_Revision","ICLR.cc/2025/Conference/-/Edit"],"mdate":1738735858747,"odate":1728008565725,"domain":"ICLR.cc/2025/Conference","id":"Q5VlpYRxGF","version":2},{"content":{"venue":{"value":"EMNLP (Findings) 2025"},"pdf":{"value":"https://aclanthology.org/2025.findings-emnlp.429.pdf"},"venueid":{"value":"dblp.org/conf/EMNLP/2025"},"paperhash":{"value":"zhang|more_tokens_lower_precision_towards_the_optimal_tokenprecision_tradeoff_in_kv_cache_compression"},"authorids":{"value":["","","","","","","~Lifeng_Shang1","",""]},"html":{"value":"https://aclanthology.org/2025.findings-emnlp.429/"},"_bibtex":{"value":"@inproceedings{DBLP:conf/emnlp/ZhangZSWKLSLL25,\n author={Jiebin Zhang and Dawei Zhu and Yifan Song and Wenhao Wu and Chuqiao Kuang and Xiaoguang Li and Lifeng Shang and Qun Liu and Sujian Li},\n title={More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression},\n year={2025},\n cdate={1735689600000},\n pages={8092-8105},\n url={https://aclanthology.org/2025.findings-emnlp.429/},\n booktitle={EMNLP (Findings)},\n crossref={conf/emnlp/2025f}\n}\n"},"abstract":{"value":"As large language models (LLMs) process increasing context windows, the memory usage of KV cache has become a critical bottleneck during inference. The mainstream KV compression methods, including KV pruning and KV quantization, primarily focus on either token or precision dimensions separately. However, these works have left the trade-off between these two orthogonal dimensions largely unexplored. In this paper, we leverage the Information Bottleneck principle to formulate KV cache compression within a unified theoretical framework. We demonstrate that a carefully managed token-precision trade-off can achieve an optimal point within the Information Bottleneck compared to standalone KV pruning or KV quantization. Experiments reveal that storing more tokens in the KV cache at lower precision—a strategy we term quantized pruning—can significantly enhance the long-context performance of LLMs. An in-depth analysis of this token-precision trade-off across key aspects shows that quantized pruning achieves substantial improvements in retrieval-related tasks and consistently performs well across varying input lengths. Furthermore, quantized pruning exhibits notable stability and effectiveness across different KV pruning methods, quantization strategies, and model scales. These findings offer valuable insights into optimizing KV cache compression through balanced token-precision trade-off strategies. Our code isavailable at https://github.com/zhzihao/QPruningKV."},"title":{"value":"More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression"},"authors":{"value":["Jiebin Zhang","Dawei Zhu","Yifan Song","Wenhao Wu","Chuqiao Kuang","Xiaoguang Li","Lifeng Shang","Qun Liu","Sujian Li"]}},"tmdate":1773712240317,"pdate":1767139200000,"externalIds":["dblp:conf/emnlp/ZhangZSWKLSLL25"],"tcdate":1773712216663,"writers":["~"],"signatures":["~Lifeng_Shang1"],"forum":"zLDnK1Qv2s","license":"CC BY-SA 4.0","number":853056,"cdate":1735689600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1773712240317,"domain":"DBLP.org","id":"zLDnK1Qv2s","version":2},{"content":{"submission_length":{"value":"Long submission (more than 12 pages of main content)"},"venue":{"value":"Accepted by TMLR"},"abstract":{"value":"Large Language Models (LLMs) have revolutionized a wide range of domains such as natural language processing, computer vision, and multi-modal tasks due to their ability to comprehend context and perform logical reasoning. However, the computational and memory demands of LLMs, particularly during inference, pose significant challenges when scaling them to real-world, long-context, and real-time applications. Key-Value (KV) cache management has emerged as a critical optimization technique for accelerating LLM inference by reducing redundant computations and improving memory utilization. This survey provides a comprehensive overview of KV cache management strategies for LLM acceleration, categorizing them into token-level, model-level, and system-level optimizations. \nToken-level strategies include KV cache selection, budget allocation, merging, quantization, and low-rank decomposition, while model-level optimizations focus on architectural innovations and attention mechanisms to enhance KV reuse. System-level approaches address memory management, scheduling, and hardware-aware designs to improve efficiency across diverse computing environments. \nAdditionally, the survey provides an overview of both text and multimodal datasets and benchmarks used to evaluate these strategies. By presenting detailed taxonomies and comparative analyses, this work aims to offer useful insights for researchers and practitioners to support the development of efficient and scalable KV cache management techniques, contributing to the practical deployment of LLMs in real-world applications."},"_bibtex":{"value":"@article{\nli2025a,\ntitle={A Survey on Large Language Model Acceleration based on {KV} Cache Management},\nauthor={Haoyang LI and Yiming Li and Anxin Tian and Tianhao Tang and Zhanchao Xu and Xuejia Chen and Nicole HU and Wei Dong and Li Qing and Lei Chen},\njournal={Transactions on Machine Learning Research},\nissn={2835-8856},\nyear={2025},\nurl={https://openreview.net/forum?id=z3JZzu9EA3},\nnote={}\n}"},"title":{"value":"A Survey on Large Language Model Acceleration based on KV Cache Management"},"changes_since_last_submission":{"value":"Following the constructive comments from the editor and reviewers, we corrected all typos, merged similar sections, adjusted the citation format, and reduced the manuscript by 10 pages of content."},"pdf":{"value":"/pdf/aa7be78974b4700c0c6fa859f7df7c62e3fd7c47.pdf"},"venueid":{"value":"TMLR"},"paperhash":{"value":"li|a_survey_on_large_language_model_acceleration_based_on_kv_cache_management"},"authorids":{"value":["~Haoyang_LI3","~Yiming_Li6","~Anxin_Tian1","~Tianhao_Tang1","~Zhanchao_Xu1","~Xuejia_Chen1","~Nicole_HU1","~Wei_Dong5","~Li_Qing6","~Lei_Chen7"]},"assigned_action_editor":{"value":"~Simone_Scardapane1"},"authors":{"value":["Haoyang LI","Yiming Li","Anxin Tian","Tianhao Tang","Zhanchao Xu","Xuejia Chen","Nicole HU","Wei Dong","Li Qing","Lei Chen"]}},"tmdate":1747639664093,"pdate":1747639664044,"tcdate":1739251075658,"writers":["TMLR"],"signatures":["TMLR/Paper4181/Authors"],"forum":"z3JZzu9EA3","license":"CC BY 4.0","number":4181,"cdate":1739251075658,"readers":["everyone"],"invitations":["TMLR/-/Submission","TMLR/-/Edit","TMLR/-/Under_Review","TMLR/Paper4181/-/Revision","TMLR/Paper4181/-/Camera_Ready_Revision","TMLR/-/Accepted"],"mdate":1747639664093,"odate":1739523728629,"domain":"TMLR","id":"z3JZzu9EA3","version":2},{"content":{"venue":{"value":"LLA 2026 Poster"},"pdf":{"value":"/pdf/5c6ed290be90eea7b75202124c5bffecac714a4c.pdf"},"keywords":{"value":["LLM","Inference","Efficiency","Agent"]},"venueid":{"value":"ICLR.cc/2026/Workshop/LLA"},"paperhash":{"value":"li|continuum_efficient_and_robust_multiturn_llm_agent_scheduling_with_kv_cache_timetolive"},"authorids":{"value":["~Hanchen_Li2","~Qiuyang_Mang1","~Runyuan_He1","~Qizheng_Zhang1","~Huanzhi_Mao1","~Xiaokun_Chen1","~Hangrui_Zhou1","~Alvin_Cheung2","~Joseph_E._Gonzalez1","~Ion_Stoica1"]},"abstract":{"value":"KV cache management is essential for efficient LLM inference. To maximize utilization, existing inference engines evict finished requests' KV cache if new requests are waiting. \nThis policy breaks for agentic workloads, which interleave LLM calls with tools, introducing pauses that prevent effective KV reuse across turns. \nSince some tool calls have much shorter durations than human response multi-turn chatbot, it would be promising to retain the KV cache in during these tools.\nHowever, there are many challenges.\nFirst, we need to consider both the potential cost of recomputation or reloading (if CPU offloading enabled) and the increasing queueing delays after eviction from GPU.\nSecond, due to the internal variance of tool call durations, we need the method to remain robust under limited predictability of tool call durations.\n\nWe present Continuum, a serving system to optimize job completion time for multi-turn agent workloads \nby introducing time-to-live mechanism for KV cache retaining.\nFor LLM request that generates a tool call,\n Continuum selectively pins the KV cache in GPU memory with a time-to-live value determined by considering both the reload cost and ordering preserve benefit of retaining KV cache. \nMoreover, when the TTL expires, the KV cache can be automatically evicted to free up GPU memory, providing robust performance under edge cases.\n When combined with program-level first-come-first-serve, Continuum preserves multi-turn continuity, and reduces delay for complex agentic workflows.\nOur evaluation on real-world agentic workloads (SWE-Bench and BFCL) with Llama-3.1 8B/70B shows that Continuum significantly improves the average job completion times and its improvement scales with turn number increase."},"_bibtex":{"value":"@inproceedings{\nli2026continuum,\ntitle={Continuum: Efficient and Robust Multi-Turn {LLM} Agent Scheduling with {KV} Cache Time-to-Live},\nauthor={Hanchen Li and Qiuyang Mang and Runyuan He and Qizheng Zhang and Huanzhi Mao and Xiaokun Chen and Hangrui Zhou and Alvin Cheung and Joseph E. Gonzalez and Ion Stoica},\nbooktitle={ICLR 2026 Workshop on Lifelong Agents: Learning, Aligning, Evolving},\nyear={2026},\nurl={https://openreview.net/forum?id=sVzK0LC9pn}\n}"},"title":{"value":"Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live"},"authors":{"value":["Hanchen Li","Qiuyang Mang","Runyuan He","Qizheng Zhang","Huanzhi Mao","Xiaokun Chen","Hangrui Zhou","Alvin Cheung","Joseph E. Gonzalez","Ion Stoica"]}},"tmdate":1775837027527,"pdate":1772422565938,"tcdate":1770431817708,"writers":["ICLR.cc/2026/Workshop/LLA","ICLR.cc/2026/Workshop/LLA/Submission69/Authors"],"signatures":["ICLR.cc/2026/Workshop/LLA/Submission69/Authors"],"forum":"sVzK0LC9pn","license":"CC BY 4.0","number":69,"cdate":1770431817708,"readers":["everyone"],"invitations":["ICLR.cc/2026/Workshop/LLA/-/Submission","ICLR.cc/2026/Workshop/LLA/-/Post_Submission","ICLR.cc/2026/Workshop/LLA/-/Edit","ICLR.cc/2026/Workshop/LLA/Submission69/-/Revision"],"mdate":1775837027527,"odate":1772734880803,"domain":"ICLR.cc/2026/Workshop/LLA","id":"sVzK0LC9pn","version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"http://arxiv.org/pdf/2506.04642v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"joshi|tada_trainingfree_recipe_for_decoding_with_adaptive_kv_cache_compression_and_meancentering"},"authorids":{"value":["~Vinay_Joshi1","~Pratik_Prabhanjan_Brahma1","https://dblp.org/search/pid/api?q=author:Zicheng_Liu:","https://dblp.org/search/pid/api?q=author:Emad_Barsoum:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2506.04642"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2506-04642,\n publtype={informal},\n author={Vinay Joshi and Pratik Prabhanjan Brahma and Zicheng Liu and Emad Barsoum},\n title={TaDA: Training-free recipe for Decoding with Adaptive KV Cache Compression and Mean-centering},\n year={2025},\n month={June},\n cdate={1748736000000},\n journal={CoRR},\n volume={abs/2506.04642},\n url={https://doi.org/10.48550/arXiv.2506.04642}\n}\n"},"abstract":{"value":"The key-value (KV) cache in transformer models is a critical component for efficient decoding or inference, yet its memory demands scale poorly with sequence length, posing a major challenge for scalable deployment of large language models. Among several approaches to KV cache compression, quantization of key and value activations has been widely explored. Most KV cache quantization methods still need to manage sparse and noncontiguous outliers separately. To address this, we introduce TaDA, a training-free recipe for KV cache compression with quantization precision that adapts to error sensitivity across layers and a mean centering to eliminate separate outlier handling. Our approach yields substantial accuracy improvements for multiple models supporting various context lengths. Moreover, our approach does not need to separately manage outlier elements -- a persistent hurdle in most traditional quantization methods. Experiments on standard benchmarks demonstrate that our technique reduces KV cache memory footprint to 27% of the original 16-bit baseline while achieving comparable accuracy. Our method paves the way for scalable and high-performance reasoning in language models by potentially enabling inference for longer context length models, reasoning models, and longer chain of thoughts."},"title":{"value":"TaDA: Training-free recipe for Decoding with Adaptive KV Cache Compression and Mean-centering"},"authors":{"value":["Vinay Joshi","Pratik Prabhanjan Brahma","Zicheng Liu","Emad Barsoum"]}},"tmdate":1759680263522,"pdate":1735689600000,"externalIds":["dblp:journals/corr/abs-2506-04642"],"tcdate":1759402422671,"writers":["~"],"signatures":["~Pratik_Prabhanjan_Brahma2"],"forum":"NUOreMKCN7","license":"CC BY-SA 4.0","number":633721,"cdate":1748736000000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1759680263522,"domain":"DBLP.org","id":"NUOreMKCN7","version":2},{"content":{"summary":{"value":"This paper proposes MARCHÉ, a training-free acceleration method for masked autoregressive (MAR) image generation. The key idea is to reuse KV representations that remain stable across decoding steps by maintaining cached tokens and selectively refreshing active ones via attention signals and online softmax. The method reports up to ~1.7× speedup on MAR models with minor quality changes and requires no model modification."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"1. KV Memory Layout Question\nHave you explored pre-allocating a single contiguous KV buffer and directly writing both cached tokens and newly active tokens into it during loading, instead of handling them in separate buffers and fusing the output? In principle, constructing a contiguous KV block upfront could eliminate the concatenate step and enable one attention kernel call on a regularized sequence length, which may yield higher efficiency.\n\n2. Question on the “generation order” claim\nI am not a domain expert, but from reading LazyMAR, my understanding is that it keeps the predefined MAR decoding schedule (i.e., which tokens are decoded at step t) and only dynamically decides which tokens can reuse features within that step to avoid recomputation. In other words, LazyMAR seems to reuse features across layers but does not change which tokens are revealed at each decoding step.\nGiven this, I am not fully understanding the statement that LazyMAR “departs from MAR’s predefined generation order.” Could you clarify in what sense the generation order is altered? If LazyMAR truly modifies the token decoding order, it would be helpful to reference the specific mechanism or evidence."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"- Training-free & plug-and-play: No architectural change, no retraining, practical for deployment.\n- Clear and practical idea: Efficient KV reuse tailored to masked autoregressive generation."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Lack of evaluation across different MAR schedules\n - The KV-reuse assumption is tested under a single MAR step schedule.\n - It is unclear whether the method generalizes to different decoding steps.\n\n2. KV-cache strategy generality remains uncertain\n - Only evaluated on MAR.\n - It is unknown how well the approach applies to other iterative masked Autoregressive generative frameworks.\n\n3. Comparison to LazyMAR needs clarification\n - LazyMAR is also a training-free MAR acceleration method. However, the paper does not clearly demonstrate superior performance over LazyMAR (e.g., matched-speed or matched-quality comparisons). A direct quality-speed Pareto comparison would be necessary to substantiate claims of advantage."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762938338200,"tcdate":1762055087553,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission20908/Reviewer_TpJn"],"signatures":["ICLR.cc/2026/Conference/Submission20908/Reviewer_TpJn"],"forum":"1noRScr1uL","number":4,"license":"CC BY 4.0","cdate":1762055087553,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission20908/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762938338200,"domain":"ICLR.cc/2026/Conference","replyto":"1noRScr1uL","id":"QstkOjTfUY","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Efficient AI","Image generation","Large Language Models"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Masked autoregressive (MAR) models unify the strengths of masked and autoregressive generation by predicting tokens in a fixed order using bidirectional attention for image generation. While effective, MAR models suffer from significant computational overhead, as they recompute attention and feed-forward representations for all tokens at every decoding step, despite most tokens remaining semantically stable across steps. We propose a training-free generation framework MARché to address this inefficiency through two key components: cache-aware attention and selective KV refresh. Cache-aware attention partitions tokens into active and cached sets, enabling separate computation paths that allow efficient reuse of previously computed key/value projections without compromising full-context modeling. But a cached token cannot be used indefinitely without recomputation due to the changing contextual information over multiple steps. MARché recognizes this challenge and applies a technique called selective KV refresh. Selective KV refresh identifies contextually relevant tokens based on attention scores from newly generated tokens and updates only those tokens that require recomputation, while preserving image generation quality. MARché significantly reduces redundant computation in MAR without modifying the underlying architecture. Empirically, MARché achieves up to 1.7x speedup with negligible impact on image quality, offering a scalable and broadly applicable solution for efficient masked transformer generation. The code is available at https://anonymous.4open.science/r/MARche-26F0."},"_bibtex":{"value":"@misc{\njiang2026march,\ntitle={{MAR}ch\\'e: Fast Masked Autoregressive Image Generation with Cache-Aware Attention},\nauthor={Chaoyi Jiang and Sungwoo Kim and Lei Gao and Hossein Entezari Zarch and Won Woo Ro and Murali Annavaram},\nyear={2026},\nurl={https://openreview.net/forum?id=1noRScr1uL}\n}"},"title":{"value":"MARché: Fast Masked Autoregressive Image Generation with Cache-Aware Attention"},"pdf":{"value":"/pdf/b6d881b5b3e1c47f4536215bb40f79c3db648bd5.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"jiang|marché_fast_masked_autoregressive_image_generation_with_cacheaware_attention"},"authorids":{"value":["~Chaoyi_Jiang1","~Sungwoo_Kim2","~Lei_Gao3","~Hossein_Entezari_Zarch1","~Won_Woo_Ro1","~Murali_Annavaram1"]},"authors":{"value":["Chaoyi Jiang","Sungwoo Kim","Lei Gao","Hossein Entezari Zarch","Won Woo Ro","Murali Annavaram"]}},"version":2},{"content":{"TLDR":{"value":"We propose Progressive Mixed-Precision KV Cache Quantization (PM-KVQ) for long-CoT LLMs."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["KV Cache Quantization"]},"supplementary_material":{"value":"/attachment/4fb968aff2c70e60ae1ea69bd0dddaca3a596fa6.zip"},"primary_area":{"value":"infrastructure, software libraries, hardware, systems, etc."},"abstract":{"value":"Recently, significant progress has been made in developing reasoning-capable Large Language Models (LLMs) through long Chain-of-Thought (CoT) techniques.\nHowever, this long-CoT reasoning process imposes substantial memory overhead due to the large Key-Value (KV) Cache memory overhead.\nPost-training KV Cache quantization has emerged as a promising compression technique and has been extensively studied in short-context scenarios.\nHowever, directly applying existing methods to long-CoT LLMs causes significant performance degradation due to the following two reasons: \n(1) Large cumulative error: Existing methods fail to adequately leverage available memory, and they directly quantize the KV Cache during each decoding step, leading to large cumulative quantization error.\n(2) Short-context calibration: Due to Rotary Positional Embedding (RoPE), the use of short-context data during calibration fails to account for the distribution of less frequent channels in the Key Cache, resulting in performance loss.\nWe propose Progressive Mixed-Precision KV Cache Quantization (PM-KVQ) for long-CoT LLMs to address the above issues in two folds:\n(1) To reduce cumulative error, we design a progressive quantization strategy to gradually lower the bit-width of KV Cache in each block. Then, we propose block-wise memory allocation to assign a higher bit-width to more sensitive transformer blocks. \n(2) To increase the calibration length without additional overhead, we propose a new calibration strategy with positional interpolation that leverages short calibration data with positional interpolation to approximate the data distribution of long-context data.\nExtensive experiments on 7B–70B long-CoT LLMs show that PM-KVQ improves reasoning benchmark performance by up to 8% over SOTA baselines under the same memory budget and achieves 2.73–5.18$\\times$ throughput over the original 16-bit LLMs.\nOur code is available at https://github.com/thu-nics/PM-KVQ."},"_bibtex":{"value":"@inproceedings{\nliu2026pmkvq,\ntitle={{PM}-{KVQ}: Progressive Mixed-precision {KV} Cache Quantization for Long-CoT {LLM}s},\nauthor={Tengxuan Liu and Shiyao Li and Jiayi Yang and Tianchen Zhao and Feng Zhou and Xiaohui Song and Guohao Dai and Shengen Yan and Huazhong Yang and Yu Wang},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=Vem6FQvRvq}\n}"},"title":{"value":"PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMs"},"pdf":{"value":"/pdf/a8e9a1e5bfaa2d5ceb64478f4b1c36a89621ece3.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"liu|pmkvq_progressive_mixedprecision_kv_cache_quantization_for_longcot_llms"},"authorids":{"value":["~Tengxuan_Liu1","~Shiyao_Li2","~Jiayi_Yang5","~Tianchen_Zhao2","~Feng_Zhou10","~Xiaohui_Song1","~Guohao_Dai4","~Shengen_Yan1","~Huazhong_Yang2","~Yu_Wang3"]},"authors":{"value":["Tengxuan Liu","Shiyao Li","Jiayi Yang","Tianchen Zhao","Feng Zhou","Xiaohui Song","Guohao Dai","Shengen Yan","Huazhong Yang","Yu Wang"]}},"tmdate":1775877047719,"pdate":1769435981333,"tcdate":1758126276029,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission9531/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission9531/Authors"],"forum":"Vem6FQvRvq","license":"CC BY 4.0","number":9531,"cdate":1758126276029,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/-/Submission","ICLR.cc/2026/Conference/-/Post_Submission","ICLR.cc/2026/Conference/Submission9531/-/Full_Submission","ICLR.cc/2026/Conference/Submission9531/-/Rebuttal_Revision","ICLR.cc/2026/Conference/-/Edit","ICLR.cc/2026/Conference/Submission9531/-/Camera_Ready_Revision"],"mdate":1775877047719,"odate":1759896705795,"domain":"ICLR.cc/2026/Conference","id":"Vem6FQvRvq","version":2},{"content":{"venue":{"value":"ICLR 2026 Poster"},"TLDR":{"value":"MixKV enhances baseline KV cache compression for LVLMs by balancing importance and diversity, delivering gains under extreme compression without sacrificing inference efficiency."},"keywords":{"value":["Large Vision-language Models","Inference Efficiency","Large Language Models"]},"supplementary_material":{"value":"/attachment/75a7c7a8aa080bf73de81dc1159b54f70f260c31.zip"},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"Recent large vision-language models (LVLMs) demonstrate remarkable capabilities in processing extended multi-modal sequences, yet the resulting key-value (KV) cache expansion creates a critical memory bottleneck that fundamentally limits deployment scalability.\nWhile existing KV cache compression methods focus on retaining high-importance KV pairs to minimize storage, they often overlook the modality-specific semantic redundancy patterns that emerge distinctively in multi-modal KV caches.\nIn this work, we first analyze how, beyond simple importance, the KV cache in LVLMs exhibits varying levels of redundancy across attention heads. \nWe show that relying solely on importance can only cover a subset of the full KV cache information distribution, leading to potential loss of semantic coverage.\nTo address this, we propose MixKV, a novel method that mixes importance with diversity for optimized KV cache compression in LVLMs. MixKV adapts to head-wise semantic redundancy, selectively balancing diversity and importance when compressing KV pairs. Extensive experiments demonstrate that MixKV consistently enhances existing methods across multiple LVLMs. Under extreme compression (budget=64), MixKV improves baseline methods by an average of 5.1% across five multi-modal understanding benchmarks and achieves remarkable gains of 8.0% and 9.0% for SnapKV and AdaKV on GUI grounding tasks, all while maintaining comparable inference efficiency. Furthermore, MixKV extends seamlessly to LLMs with comparable performance gains."},"_bibtex":{"value":"@inproceedings{\nliu2026mixing,\ntitle={Mixing Importance with Diversity: Joint Optimization for {KV} Cache Compression in Large Vision-Language Models},\nauthor={Xuyang Liu and Xiyan Gui and Yuchao Zhang and Linfeng Zhang},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=B2iqbCQviR}\n}"},"title":{"value":"Mixing Importance with Diversity: Joint Optimization for KV Cache Compression in Large Vision-Language Models"},"pdf":{"value":"/pdf/49319084a96d450e92604d966be2993bebdec090.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"liu|mixing_importance_with_diversity_joint_optimization_for_kv_cache_compression_in_large_visionlanguage_models"},"authorids":{"value":["~Xuyang_Liu1","~Xiyan_Gui1","~Yuchao_Zhang6","~Linfeng_Zhang2"]},"authors":{"value":["Xuyang Liu","Xiyan Gui","Yuchao Zhang","Linfeng Zhang"]}},"tmdate":1775876963455,"pdate":1769435794884,"tcdate":1757846252452,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission5086/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission5086/Authors"],"forum":"B2iqbCQviR","license":"CC BY 4.0","number":5086,"cdate":1757846252452,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/-/Submission","ICLR.cc/2026/Conference/-/Post_Submission","ICLR.cc/2026/Conference/Submission5086/-/Full_Submission","ICLR.cc/2026/Conference/Submission5086/-/Rebuttal_Revision","ICLR.cc/2026/Conference/-/Edit","ICLR.cc/2026/Conference/Submission5086/-/Camera_Ready_Revision"],"mdate":1775876963455,"odate":1759896705795,"domain":"ICLR.cc/2026/Conference","id":"B2iqbCQviR","version":2},{"content":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Large language models","KV-Cache compression","Budget"]},"primary_area":{"value":"generative models"},"abstract":{"value":"Large language models (LLMs) inference relies heavily on KV-caches to accelerate autoregressive decoding, but the resulting memory footprint grows rapidly with sequence length, posing significant efficiency challenges.\nCurrent KV-cache compression methods suffer from a Procrustes' bed problem: they force diverse workloads into fixed compression ratios, leading to suboptimal resource allocation and inference performance. \nTo this end, we present GVote, an adaptive per-request KV-cache compression scheme that eliminates manual budget specification while achieving superior accuracy-efficiency trade-offs. \nGVote operates on the principle that the important keys are the aggregation of keys required by future queries. \nGvote predicts future query attention demands by Monte-Carlo style sampling potential queries and aggregating selected keys to determine the optimal cache budget without manual specification.\nExperimental evaluation demonstrates GVote's effectiveness across multiple benchmarks, including GSM8K, RULER and Longbench. \nCompared to baselines, GVote exhibits 2$\\times$ memory reduction while the accuracy maintains higher or comparable."},"_bibtex":{"value":"@misc{\ntang2026gvote,\ntitle={{GV}ote: Adaptive Per-request {KV}-Cache Compression without Manually Setting Budget},\nauthor={Chenxia Tang and Jianchun Liu and Hongli Xu and Jinyang Huang and Liusheng Huang},\nyear={2026},\nurl={https://openreview.net/forum?id=0yLdDZMutq}\n}"},"title":{"value":"GVote: Adaptive Per-request KV-Cache Compression without Manually Setting Budget"},"pdf":{"value":"/pdf/533fb78ad6425112140f9199de70e69be5be19c3.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"tang|gvote_adaptive_perrequest_kvcache_compression_without_manually_setting_budget"},"authorids":{"value":["~Chenxia_Tang1","~Jianchun_Liu1","~Hongli_Xu1","~Jinyang_Huang1","~Liusheng_Huang1"]},"authors":{"value":["Chenxia Tang","Jianchun Liu","Hongli Xu","Jinyang Huang","Liusheng Huang"]}},"tmdate":1770804793520,"tcdate":1756731766775,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission225/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission225/Authors"],"forum":"0yLdDZMutq","license":"CC BY 4.0","number":225,"cdate":1756731766775,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/-/Submission","ICLR.cc/2026/Conference/-/Post_Submission","ICLR.cc/2026/Conference/Submission225/-/Full_Submission","ICLR.cc/2026/Conference/-/Edit"],"mdate":1770804793520,"odate":1759896705795,"domain":"ICLR.cc/2026/Conference","id":"0yLdDZMutq","version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"http://arxiv.org/pdf/2503.08461v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"zhu|fastcache_optimizing_multimodal_llm_serving_through_lightweight_kvcache_compression_framework"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Jianian_Zhu:","https://dblp.org/search/pid/api?q=author:Hang_Wu:","https://dblp.org/search/pid/api?q=author:Haojie_Wang:","https://dblp.org/search/pid/api?q=author:Yinghui_Li:","https://dblp.org/search/pid/api?q=author:Biao_Hou:","~Ruixuan_Li1","~Jidong_Zhai1"]},"html":{"value":"https://doi.org/10.48550/arXiv.2503.08461"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2503-08461,\n publtype={informal},\n author={Jianian Zhu and Hang Wu and Haojie Wang and Yinghui Li and Biao Hou and Ruixuan Li and Jidong Zhai},\n title={FastCache: Optimizing Multimodal LLM Serving through Lightweight KV-Cache Compression Framework},\n year={2025},\n month={March},\n cdate={1740787200000},\n journal={CoRR},\n volume={abs/2503.08461},\n url={https://doi.org/10.48550/arXiv.2503.08461}\n}\n"},"abstract":{"value":"Multi-modal Large Language Models (MLLMs) serving systems commonly employ KV-cache compression to reduce memory footprint. However, existing compression methods introduce significant processing overhead and queuing delays, particularly in concurrent serving scenarios. We present \\texttt{FastCache}, a novel serving framework that effectively addresses these challenges through two key innovations: (1) a dynamic batching strategy that optimizes request scheduling across prefill, compression, and decode stages, and (2) an efficient KV-cache memory pool mechanism that eliminates memory fragmentation while maintaining high GPU utilization. Our comprehensive experiments on the GQA and MileBench datasets demonstrate that \\texttt{FastCache} achieves up to 19.3$\\times$ reduction in Time-To-First-Token (TTFT) and 12.1$\\times$ improvement in throughput compared to state-of-the-art baselines. The system maintains stable performance under high-concurrency scenarios (up to 40 req/s) while reducing average memory consumption by 20\\%. These results establish \\texttt{FastCache} as an efficient solution for real-world LLM serving systems with KV-cache compression."},"title":{"value":"FastCache: Optimizing Multimodal LLM Serving through Lightweight KV-Cache Compression Framework"},"authors":{"value":["Jianian Zhu","Hang Wu","Haojie Wang","Yinghui Li","Biao Hou","Ruixuan Li","Jidong Zhai"]}},"tmdate":1762325466320,"pdate":1735689600000,"tcdate":1752759348377,"writers":["~"],"signatures":["~Ruixuan_Li1"],"forum":"sunngRPNm8","license":"CC BY-SA 4.0","number":571256,"cdate":1740787200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1762325466320,"domain":"DBLP.org","id":"sunngRPNm8","version":2},{"content":{"summary":{"value":"The paper has made an observation: induction and echo heads are retrieval heads and others are local processing heads. They then keep complete KV Cache for retrieval heads and drop non-local KV for local processing heads. Following their hyper parameters, this leads to 70% asymptotic KV Compression."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"Please address the weaknesses above."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. The observation is neat. And experiments support the hypothesis quite well. \n2. The recipe to identify the heads is very lightweight (although is this a contribution or is it previously known from anthropic paper?)"},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Experiments\n\n A. why are some datasets from longbench missing like passage_retrieval etc\n\n B. Can you perform experiments on infinity benchmark for comparison.\n\n C. what is the exact KV compression in the longbench datasets? The 70% number is misleading since longbench generally has small context lengths and authors have used a context thold of 4000. Please add exact KV Cache compression numbers for all the methods to the table. \n\n2. Novelty: Can you please elaborate on the novelty of defining / identifying the echo and induction heads. what is already known and what is figured out?\n\n3. Writing errors: Please revise the manuscript for text/language errors. e.g. : \n \n A. \"There are been plenty ...\" (page 1) \n \n B. \" and proved that accuracy ..\" ( page 2) {you showed it empirically .. and also 70% is not what you show in the experiments.}.\n\n C. C_0 in equation 2. Do you mean L_h ?"}},"nonreaders":[],"tmdate":1732698264269,"tcdate":1730843185711,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission12841/Reviewer_hKDU"],"signatures":["ICLR.cc/2025/Conference/Submission12841/Reviewer_hKDU"],"forum":"tkiZQlL04w","number":5,"license":"CC BY 4.0","cdate":1730843185711,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission12841/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732698264269,"domain":"ICLR.cc/2025/Conference","replyto":"tkiZQlL04w","id":"zg4zSTqjb2","forumContent":{"TLDR":{"value":"A novel KV cache compression algorithm that achieves 3X reduction under general cases, compatible with FlashAttention"},"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["LLMs","KV cache compression","LLM inference acceleration"]},"supplementary_material":{"value":"/attachment/11a8b7d623606429419ce6128bbe4eb6a6b958bf.pdf"},"primary_area":{"value":"generative models"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The memory and computational demands of Key-Value (KV) cache present significant challenges for deploying long-context language models. Previous approaches attempt to mitigate this issue by selectively dropping tokens, which irreversibly erases critical information that might be needed for future queries. In this paper, we propose a novel compression technique for KV cache that preserves all token information. Our investigation reveals that: i) Most attention heads primarily focus on the local context; ii) Only a few heads, denoted as retrieval heads, can essentially pay attention to all input tokens. These key observations motivate us to use separate caching strategy for attention heads.Therefore, we propose RazorAttention, a training-free KV cache compression algorithm, which maintains a full cache for these crucial retrieval heads and discards the remote tokens in non-retrieval heads. Furthermore, we introduce a novel mechanism involving a “compensation token” to further recover the information in the dropped tokens. Extensive evaluations across a diverse set of large language models (LLMs) demonstrate that RazorAttention achieves a reduction in KV cache size by over 70% without noticeable impacts on performance. Additionally, RazorAttention is compatible with FlashAttention, rendering it an efficient and plug-and-play solution that enhances LLM inference efficiency without overhead or retraining of the original model."},"_bibtex":{"value":"@inproceedings{\ntang2025razorattention,\ntitle={RazorAttention: Efficient {KV} Cache Compression Through Retrieval Heads},\nauthor={Hanlin Tang and Yang Lin and Jing Lin and Qingsen Han and Danning Ke and Shikuan Hong and Yiwu Yao and Gongyi Wang},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=tkiZQlL04w}\n}"},"title":{"value":"RazorAttention: Efficient KV Cache Compression Through Retrieval Heads"},"pdf":{"value":"/pdf/808d99b57ee82af73b93fe34b2eeaf04c4171bfd.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"tang|razorattention_efficient_kv_cache_compression_through_retrieval_heads"},"authorids":{"value":["~Hanlin_Tang2","~Yang_Lin12","~Jing_Lin6","~Qingsen_Han1","~Danning_Ke1","~Shikuan_Hong1","~Yiwu_Yao1","~Gongyi_Wang1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Hanlin Tang","Yang Lin","Jing Lin","Qingsen Han","Danning Ke","Shikuan Hong","Yiwu Yao","Gongyi Wang"]}},"version":2},{"content":{"summary":{"value":"This paper investigates memory compression mechanisms in large language models (LLMs), focusing on achieving lossless or near-lossless memory efficiency. It systematically analyzes the relationship between attention redundancy, representation compression, and retrieval fidelity, and proposes a new compression architecture intended to retain semantic fidelity while reducing activation and KV cache memory."},"soundness":{"value":1},"confidence":{"value":5},"questions":{"value":"1. How exactly is “lossless” defined in your experiments? Is it equivalent to maintaining identical outputs, or bounded semantic deviation?\n2. Have you measured mutual information or entropy changes before and after compression?\n3. How does your method compare to dynamic memory eviction or entropy-based key selection methods?\n4. Does the compression framework support online adaptation — e.g., varying compression ratio during inference?\n5. Are the proposed memory structures compatible with GPU-efficient implementations (e.g., FlashAttention or PagedAttention)?"},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":1},"strengths":{"value":"The paper addresses an important issue in modern LLM systems — the scalability of context memory and cache compression. This is a crucial bottleneck for both open-source and proprietary systems. The authors provide a solid overview of various strategies (quantization, pruning, clustering, key-value cache compression, etc.), framing them under a unified memory-entropy perspective. This synthesis is useful for the community. The experiments, particularly those on retrieval fidelity and downstream reasoning, demonstrate that small amounts of information loss can lead to disproportionate accuracy degradation, supporting the motivation for “lossless” compression."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The “lossless” definition used in the paper lacks a rigorous mathematical formulation. The text refers vaguely to “semantic equivalence under compression,” but no formal mapping (e.g., bijection between original and compressed latent space preserving mutual information) is provided. This makes it hard to assess the conceptual contribution.\n2. While the paper cites several existing compression methods (e.g., KV cache quantization, low-rank adaptation, memory token merging), it is unclear what aspect is fundamentally new. The proposed framework seems to integrate known techniques rather than introduce a distinctly novel algorithm.\n3. The experiments appear to be conducted on small- to medium-scale models and limited benchmarks. It’s unclear how the proposed method scales to models beyond 7B parameters or generalizes to instruction-following and multimodal tasks. In addition, most results lack variance reporting and ablation analysis.\n4. The metrics primarily evaluate perplexity and accuracy but not information preservation. A more appropriate evaluation would involve mutual information, reconstruction error, or similarity metrics on attention distributions.\n5. The claim that “compression can improve reasoning stability by removing noise” is speculative and not well supported by experiments. Correlation is shown, but no causal analysis or controlled experiments are presented.\n6. The paper repeatedly claims that “redundant activations lead to inefficiency” and “compression improves generalization,” but it only shows correlations between compression ratio and performance metrics (e.g., perplexity).\nThere is no controlled experiment isolating causal factors — for instance, whether observed gains arise from reduced redundancy, implicit regularization, or simple noise filtering. Without causal validation, these claims remain speculative.\n7. Memory compression may cause unstable behavior, especially in autoregressive decoding where small perturbations can cascade.\nThe paper does not evaluate output stability under repeated sampling, sensitivity to compression noise, or robustness across different temperature settings. Without such analyses, it is unclear whether the approach is reliable for real-world use."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762919605543,"tcdate":1761822397662,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7492/Reviewer_cB7e"],"signatures":["ICLR.cc/2026/Conference/Submission7492/Reviewer_cB7e"],"forum":"nrBJ0Uvj7c","number":4,"license":"CC BY 4.0","cdate":1761822397662,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7492/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762919605543,"domain":"ICLR.cc/2026/Conference","replyto":"nrBJ0Uvj7c","id":"pzXTlDBVK7","forumContent":{"TLDR":{"value":"A widely applicable automatic pipeline combining spatio-temporal gradient checkpointing and spike compression that achieves up to 8× memory savings for SNN training while preserving BPTT-level accuracy and speed."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Spiking Neural Network","Training Memory Optimization","Gradient Checkpointing"]},"supplementary_material":{"value":"/attachment/8d63883ac849c6c7655cde77424096d09810af73.zip"},"primary_area":{"value":"applications to neuroscience & cognitive science"},"abstract":{"value":"Deep spiking neural networks (SNNs) hold immense promise for low-power event-driven computing, but their direct training via backpropagation through time (BPTT) incurs prohibitive memory cost, which limits their scalability. Existing memory-saving approaches, such as online learning, BPTT-to-BP, and reversible networks, compromise accuracy, training speed, or applicability. In this work, we propose a novel and broadly applicable pipeline for memory-efficient SNN training that preserves BPTT's accuracy. Our pipeline integrates layer-wise gradient checkpointing with lossless spike compression to eliminate internal state storage and reduce the memory cost of per-layer input spikes. We also introduce a multi-stage checkpoint adjustment strategy that adaptively refines checkpoint placement based on profiling results to further optimize memory usage and improve training speed. Wrapped in an optimization pass, the pipeline automatically restructures the computation flow before training with minimal user effort. Extensive experiments on diverse architectures and tasks demonstrate up to $8\\times$ memory efficiency gains with $\\le 20\\%$ speed reduction and no accuracy loss. Our method provides a practical solution for efficient and scalable SNN training. Code is available at https://github.com/AllenYolk/snn-gradient-checkpointing."},"_bibtex":{"value":"@inproceedings{\nhuang2026towards,\ntitle={Towards Lossless Memory-efficient Training of Spiking Neural Networks via Gradient Checkpointing and Spike Compression},\nauthor={Yifan Huang and Wei Fang and Zecheng Hao and Zhengyu Ma and Yonghong Tian},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=nrBJ0Uvj7c}\n}"},"title":{"value":"Towards Lossless Memory-efficient Training of Spiking Neural Networks via Gradient Checkpointing and Spike Compression"},"pdf":{"value":"/pdf/ec4d001c1c25eda6ef4c24cb9a8822499b38bea5.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"huang|towards_lossless_memoryefficient_training_of_spiking_neural_networks_via_gradient_checkpointing_and_spike_compression"},"authorids":{"value":["~Yifan_Huang7","~Wei_Fang2","~Zecheng_Hao1","~Zhengyu_Ma1","~Yonghong_Tian1"]},"authors":{"value":["Yifan Huang","Wei Fang","Zecheng Hao","Zhengyu Ma","Yonghong Tian"]}},"version":2},{"content":{"summary":{"value":"The paper introduces Rolling Forcing (RF), a streaming text-to-video diffusion method that reduces long-horizon error accumulation while keeping real-time throughput. Key ideas: (i) a rolling denoising window that jointly denoises several consecutive frames with bidirectional attention inside the window but emits one clean frame per forward pass; (ii) a global/temporal KV cache with dynamic RoPE so early frames act as a stable “attention-sink” anchor over minutes; and (iii) few-step DMD distillation on non-overlapping windows plus a mixed Self-Forcing (SF) objective for more natural motion. On VBench, RF improves quality and notably lowers quality drift vs. open AR baselines, with ablations for each component."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"Please see weaknesses"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"Effective drift control: Rolling window enables mutual refinement before a frame is finalized; ablations show large Quality_Drift gains. \n Long-term consistency trick: Global KV cache + dynamic RoPE keeps initial context usable without RoPE over-offset artifacts. \n Real-time demo & interactivity: Sub-second steady-state latency and prompt-switching during streaming."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"* Attention-sink rationale needs deeper evidence: “Attention sink” comes from LMs; here it’s adapted via KV caching + dynamic RoPE. The paper shows ablations, but provides limited analysis of why the sink phenomenon should emerge in video DiTs. As written, it risks reading as a concept transplant without sufficient mechanistic support. \n* Motion/scene diversity trade-off: Demos suggest mis-tuning can collapse into static outputs, RF may bias toward stillness to curb drift. More systematic tests compared to baselines on large motion and scene changes are needed, you may use like dynamic degree metric in Vbench. \n* Evaluation scope: Metrics center on 30 s clips (quality drift measured 0–5 s vs. last 5 s). Given the multi-minute claim, add standardized 2–5 min quantitative runs (and a small human preference study). \n* Scaling detail: Limited profiling across window/model sizes"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762917369384,"tcdate":1761708532419,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission4445/Reviewer_FrmR"],"signatures":["ICLR.cc/2026/Conference/Submission4445/Reviewer_FrmR"],"forum":"IAyzXjbfwo","number":1,"license":"CC BY 4.0","cdate":1761708532419,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission4445/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762917369384,"domain":"ICLR.cc/2026/Conference","replyto":"IAyzXjbfwo","id":"cuI0ni1Y0u","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"TLDR":{"value":"REAL-TIME streaming generation of MULTI-MINUTE videos!"},"keywords":{"value":["autoregressive video generation","long video generation","real-time video generation"]},"primary_area":{"value":"generative models"},"abstract":{"value":"Streaming video generation as one fundamental component in interactive world models and neural game engines aims to generate high-quality, low-latency, and temporally coherent long stream videos. However, most existing work suffers from severe error accumulation that often significantly degrades the generated stream videos over long horizons. We design Rolling Forcing, a novel video generation technique that enables streaming long videos with minimal error accumulation. Rolling Forcing comes with three novel designs. First, instead of iteratively sampling individual frames which accelerates error propagation, we design a joint denoising scheme that simultaneously denoises multiple frames with progressively increasing noise levels. This design relaxes the strict causality across adjacent frames, effectively suppressing error growth. Second, we introduce the attention sink mechanism into the long-horizon stream video generation task, which allows the model to keep key–value states of initial frames as a global context anchor and thereby enhances long-term global consistency. Third, we design an efficient training algorithm that enables few-step distillation over largely extended denoising windows. This algorithm operates on non-overlapping windows and mitigates exposure bias conditioned on self-generated histories. Extensive experiments show that Rolling Forcing enables real-time streaming generation of multi-minute videos on a single GPU, with substantially reduced error accumulation."},"_bibtex":{"value":"@inproceedings{\nliu2026rolling,\ntitle={Rolling Forcing: Autoregressive Long Video Diffusion in Real Time},\nauthor={Kunhao Liu and Wenbo Hu and Jiale Xu and Ying Shan and Shijian Lu},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=IAyzXjbfwo}\n}"},"title":{"value":"Rolling Forcing: Autoregressive Long Video Diffusion in Real Time"},"pdf":{"value":"/pdf/75046738baed876ab95731f249eaa20cb17c28fd.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"liu|rolling_forcing_autoregressive_long_video_diffusion_in_real_time"},"authorids":{"value":["~Kunhao_Liu1","~Wenbo_Hu2","~Jiale_Xu1","~Ying_Shan2","~Shijian_Lu1"]},"authors":{"value":["Kunhao Liu","Wenbo Hu","Jiale Xu","Ying Shan","Shijian Lu"]}},"version":2},{"content":{"TLDR":{"value":"CoCoKV improves KV cache compression by combining soft cross-head consensus with completeness-aware token selection, preserving reliable and comprehensive contextual evidence under tight cache budgets."},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["KV Cache Compression","LLM Inference","Consensus","Completeness"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Efficient long-context inference in Large Language Models (LLMs) is severely constrained by the memory footprint of the Key-Value (KV) cache. Recent prompt-stage methods reduce this overhead by estimating historical-token importance from observation-window attention and retaining only a compact subset of the KV cache. However, their reliance on head-specific token importance gives rise to two potential limitations: i) Head-wise Attention Partiality, where individual attention heads can capture specialized yet partial views of token importance, and ii) Token-wise Information Omission, where selecting highly attended tokens leads to semantic omission while overlooking other crucial information. To address these limitations, we present CoCoKV, a novel KV cache compression framework designed to retain both reliable and comprehensive evidence under constrained cache budgets. Specifically, CoCoKV introduces a soft cross-head consensus mechanism that uses collective signals to refine head-specific attention while preserving head distinctiveness. Building on the calibrated importance scores, a cross-token completeness selection strategy is applied to promote broader semantic coverage. Evaluations across Needle-in-a-Haystack and $16$ LongBench datasets on three LLMs demonstrate that CoCoKV outperforms competitive baselines, validating its effectiveness and robustness."},"_bibtex":{"value":"@inproceedings{\nanonymous2026cocokv,\ntitle={CoCo{KV}: Consensus- and Completeness-Aware {KV} Cache Compression for {LLM} Inference},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=RBoG5kVQXp},\nnote={under review}\n}"},"title":{"value":"CoCoKV: Consensus- and Completeness-Aware KV Cache Compression for LLM Inference"},"pdf":{"value":"/pdf/f9b69274be675c384e4fd35d34a5567b4cb79806.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791228261073,"tcdate":1789375735376,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission17513/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission17513/Authors"],"forum":"RBoG5kVQXp","license":"CC BY 4.0","number":17513,"cdate":1789375735376,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission17513/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791228261073,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"RBoG5kVQXp","version":2},{"content":{"comment":{"value":"Thank you for your insightful review and for raising the score to 5.\n\nAs we emphasized in our response to W1, our trainable KV-Cache eviction method adopts a different perspective compared to training-free approaches. Our method is designed to achieve adaptive KV-Cache eviction policies at multiple levels, including tasks, model layers, attention heads, and prompts. This level of adaptability is challenging for training-free methods, which are generally aimed at providing a more general-purpose strategy.\n\nWe sincerely appreciate your thoughtful and comprehensive feedback, which has greatly contributed to refining our work. Thank you again for your valuable comments!"},"title":{"value":"Sincere Appreciation and Emphasis on the Adaptability of Our Trainable KV-Cache Method"}},"tmdate":1732521096146,"tcdate":1732521096146,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission8951/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission8951/Authors"],"forum":"tvQNysCP7C","number":14,"license":"CC BY 4.0","cdate":1732521096146,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission8951/-/Official_Comment"],"mdate":1732521096146,"domain":"ICLR.cc/2025/Conference","replyto":"5X3RxvbBJF","id":"bKnyGFG1P9","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["KV-Cache eviction","Attention-Gate","LLM inference acceleration"]},"supplementary_material":{"value":"/attachment/147f28fa623cf49cbf5091a5ce45aec09d8eaa7a.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The KV-Cache technique has become the standard for the inference of large language models (LLMs).\nIt caches states of self-attention to avoid recomputation. \nYet, it is widely criticized that KV-Cache can become a bottleneck of the LLM inference system, especially when confronted with ultra-large models and long-context queries. \nA natural remedy is to discard the KV-Cache for less important tokens, with StreamingLLM as an example, but the used static eviction strategies cannot flexibly adapt to varying contexts. \nRemedies like H2O leverage accumulative attention scores to perform dynamic eviction but suffer from the attention bias issue in capturing contextual information. \nThis paper bridges this gap by devising a parameterized KV-Cache eviction mechanism, dubbed as *Attention-Gate*, which accepts the whole context as input and yields eviction flags for each token to realize *in-context* eviction. \nThe subsequent self-attention module proceeds according to the flags and only the KV states for the remaining tokens need to be cached. \nThe Attention-Gates can vary among different heads and layers and be trivially plugged into pre-trained LLMs, tuned by cost-effective continual pre-training or supervised fine-tuning objectives to acquire what to discard. \nThe computational and memory overhead introduced by Attention-Gates is minimal.\nOur method is validated across multiple tasks, demonstrating both efficiency and adaptability.\nAfter a highly efficient continual pre-training, it achieves higher average accuracy and evicts more tokens compared to traditional training-free methods. \nIn supervised fine-tuning, it not only evicts many tokens but also outperforms LoRA-finetuned LLMs on some datasets, such as RTE, where it improves accuracy by 13.9% while evicting 62.8% of tokens, showing that effective eviction of redundant tokens can even enhance performance."},"_bibtex":{"value":"@misc{\nzeng2025incontext,\ntitle={In-context {KV}-Cache Eviction for {LLM}s via Attention-Gate},\nauthor={Zihao Zeng and Bokai Lin and TianQi Hou and Hao Zhang and Zhijie Deng},\nyear={2025},\nurl={https://openreview.net/forum?id=tvQNysCP7C}\n}"},"title":{"value":"In-context KV-Cache Eviction for LLMs via Attention-Gate"},"pdf":{"value":"/pdf/28994ecde81dc28a4b4f92441562ed2f2f02c0bb.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"zeng|incontext_kvcache_eviction_for_llms_via_attentiongate"},"authorids":{"value":["~Zihao_Zeng1","~Bokai_Lin1","~TianQi_Hou1","~Hao_Zhang2","~Zhijie_Deng1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Zihao Zeng","Bokai Lin","TianQi Hou","Hao Zhang","Zhijie Deng"]}},"version":2},{"content":{"venue":{"value":"ICML 2026 regular"},"keywords":{"value":["Vision-Language Model","KV Cache Eviction","Inference Accleration"]},"supplementary_material":{"value":"/attachment/a327322c48af5b9bfa892bd9197af1578704a69b.zip"},"_bibtex":{"value":"@inproceedings{\nliu2026tgvkv,\ntitle={{TGV}-{KV}: Text-Grounded {KV} Eviction for Vision-Language Models},\nauthor={Jizhihui Liu and Ruizi Han and Miao Zhang and Rui Shao and Xuebo Liu and Weili Guan and Yaowei Wang},\nbooktitle={Forty-third International Conference on Machine Learning},\nyear={2026},\nurl={https://openreview.net/forum?id=suQ52Hqr9U}\n}"},"title":{"value":"TGV-KV: Text-Grounded KV Eviction for Vision-Language Models"},"paperhash":{"value":"liu|tgvkv_textgrounded_kv_eviction_for_visionlanguage_models"},"originally_submitted_PDF":{"value":"/pdf/a357c652f22ebb53dcbae6aec9b07e3ebcf152f3.pdf"},"TLDR":{"value":"We propose TGV-KV, a training-free multimodal KV cache eviction framework that fully utilize text guidance."},"primary_area":{"value":"deep_learning->foundation_models"},"abstract":{"value":"Vision-Language Models (VLMs) inherit the auto-regressive generation paradigm and cache the keys and values (KV) of all previous tokens to accelerate inference, resulting in memory consumption that scales linearly with context length. This issue is particularly pronounced in VLMs due to substantial redundancy in the visual modality. Although KV cache eviction approaches can effectively reduce inference memory, they often incur significant performance degradation in VLMs, as most are designed for language models and overlook the inherent gap between text and vision. By systematically analyzing the modality gap in VLMs in this work, we argue that the importance of visual information should be grounded in textual guidance and accordingly propose a **T**ext-**G**rounded KV Eviction method for **V**LMs (**TGV-KV**). TGV-KV comprises three submodules: *(1) Text-Vision Budgeting (TVB)* assigns budget to each layer based on the mutual information interaction. *(2) Text-Weighted Ranking (TWR)* assesses the priority of text and ranks vision importance based on weighted text-image attention. *(3) Text-Prioritised Retention (TPR)* policy strategically preserves text KV to avoid acute information loss. We evaluate TGV-KV across five models with different sizes and architectures, showing that TGV-KV preserves 99.2% full-KV accuracy on the VizWiz-VQA task with LLaVA-NeXT and boosts end-to-end throughput by 52.6% with an extreme retention budget of 5%. [Code Link](https://github.com/Danielement321/TGV-KV)."},"link_to_code":{"value":"https://github.com/Danielement321/TGV-KV"},"pdf":{"value":"/pdf/de0cb65c42a4cde068136e95c430142595623447.pdf"},"lay_summary":{"value":"Vision-language AI systems can answer questions about images and videos, but they often need a large amount of computer memory when generating responses. This is because they keep a “memory notebook” of many previous image and text pieces so they do not have to recompute them later. For images and videos, this notebook can become very large, even though many visual pieces are repetitive or less relevant to the user’s question.\n\nOur work introduces TGV-KV, a method that decides which parts of this notebook should be kept and which can be safely removed. The key idea is simple: the text question should guide what visual information matters. Instead of treating image and text pieces equally, TGV-KV protects important text information and keeps the visual details most related to it.\n\nAcross several vision-language models and tasks, TGV-KV greatly reduces memory use while preserving most of the original accuracy. This makes powerful multimodal AI systems faster and easier to deploy on limited hardware."},"venueid":{"value":"ICML.cc/2026/Conference"},"authorids":{"value":["~Jizhihui_Liu1","~Ruizi_Han1","~Miao_Zhang4","~Rui_Shao1","~Xuebo_Liu1","~Weili_Guan4","~Yaowei_Wang1"]},"authors":{"value":["Jizhihui Liu","Ruizi Han","Miao Zhang","Rui Shao","Xuebo Liu","Weili Guan","Yaowei Wang"]}},"tmdate":1790067257726,"pdate":1777576140152,"tcdate":1767873984141,"writers":["ICML.cc/2026/Conference","ICML.cc/2026/Conference/Submission1013/Authors"],"signatures":["ICML.cc/2026/Conference/Submission1013/Authors"],"forum":"suQ52Hqr9U","license":"CC BY 4.0","number":1013,"cdate":1767873984141,"readers":["everyone"],"invitations":["ICML.cc/2026/Conference/-/Submission","ICML.cc/2026/Conference/-/Post_Submission","ICML.cc/2026/Conference/Submission1013/-/Ready_For_Llm_Feedback","ICML.cc/2026/Conference/-/Edit","ICML.cc/2026/Conference/Submission1013/-/Full_Submission","ICML.cc/2026/Conference/Submission1013/-/Camera_Ready_Revision"],"mdate":1790067257726,"odate":1782341894664,"domain":"ICML.cc/2026/Conference","id":"suQ52Hqr9U","version":2},{"content":{"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["LLM-based Multi-agent Systems","KV Cache Quantization"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"LLM-based multi-agent systems solve complex tasks through problem decomposition, specialized tool use, and iterative verification. However, collaboration produces multiple growing key-value (KV) caches that can coexist in memory, making cache storage a bottleneck. KV-cache quantization reduces this footprint, but independently applying single-model policies ignores differences in agents' precision needs. This can over-compress sensitive computations while spending precision on more tolerant caches, potentially degrading collective reasoning. In this paper, we introduce **Co**ordinated **Mix**ed-Precision **KV** Cache Quantization (**CoMixKV**), a novel method that integrates online sensitivity estimation with joint bit-width allocation under a shared memory budget. To our knowledge, **CoMixKV** is the first mixed-precision KV-cache quantization framework to jointly allocate precision across simultaneously resident agent caches under such a budget. An online estimator derives each agent's sensitivity from a local attention-error bound using recent queries, keys, and values, without calibration data, trial quantization, or additional model passes. A shared controller combines these scores with remaining quantizable capacity and byte-level storage costs to minimize total estimated error. Allocations are updated as agents start or finish and fresh observations become available. Selected widths apply only to newly quantized blocks; recent states retain native precision and previously quantized blocks remain unchanged. Across four benchmarks and three LLM backbones at average 2-bit, 3-bit, and 4-bit KV-cache settings, **CoMixKV** achieves strong accuracy compared with the evaluated quantization baselines. At the average 3-bit setting on Gemma-4-31B-it, **CoMixKV** achieves 68.35\\% average accuracy, outperforming the strongest quantized baseline by 0.59\\%. On Qwen3-32B, **CoMixKV** further reduces peak KV-cache memory by 32.08\\% and achieves a 1.38$\\times$ inference speedup over FullKV."},"_bibtex":{"value":"@inproceedings{\nanonymous2026coordinated,\ntitle={Coordinated Mixed-Precision {KV} Cache Quantization for Efficient {LLM}-based Multi-agent Systems},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=x9qK9ehqNi},\nnote={under review}\n}"},"title":{"value":"Coordinated Mixed-Precision KV Cache Quantization for Efficient LLM-based Multi-agent Systems"},"pdf":{"value":"/pdf/052a827dced059360905162c2a620234a931c1a2.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791219299748,"tcdate":1787193539099,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission352/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission352/Authors"],"forum":"x9qK9ehqNi","license":"CC BY 4.0","number":352,"cdate":1787193539099,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission352/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791219299748,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"x9qK9ehqNi","version":2},{"content":{"TLDR":{"value":"Inference-time hyper-scaling uses key–value cache compression with Delayed Memory Sparsification (DMS) to boost Transformer LLM reasoning accuracy for equivalent compute or memory costs."},"venue":{"value":"NeurIPS 2025 poster"},"keywords":{"value":["reasoning","inference scaling","KV cache compression","token eviction","large language model"]},"supplementary_material":{"value":"/attachment/14c1db499e99aff4bc84c67185642cb125dd80cc.zip"},"primary_area":{"value":"deep_learning"},"abstract":{"value":"Inference-time scaling trades efficiency for increased reasoning accuracy by generating longer or more parallel sequences. However, in Transformer LLMs, generation cost is bottlenecked by the size of the key–value (KV) cache, rather than the number of generated tokens. Hence, we explore inference-time hyper-scaling: by compressing the KV cache, we can generate more tokens within the same compute budget and further improve the accuracy of scaled inference. The success of this approach, however, hinges on the ability of compression methods to preserve accuracy even at high compression ratios. To make hyper-scaling practical, we introduce Dynamic Memory Sparsification (DMS), a novel method for sparsifying KV caches that only requires 1K training steps to achieve 8× compression, while maintaining better accuracy than training-free sparse attention. Instead of prematurely discarding cached tokens, DMS delays token eviction, implicitly merging representations and preserving critical information. We demonstrate the effectiveness of inference-time hyper-scaling with DMS on multiple families of LLMs, showing that it boosts accuracy for comparable inference latency and memory load. For instance, we enhance Qwen-R1 32B by 9.1 points on AIME 24, 7.6 on GPQA, and 9.6 on LiveCodeBench on average for an equivalent number of memory reads."},"_bibtex":{"value":"@inproceedings{\nancucki2025inferencetime,\ntitle={Inference-Time Hyper-Scaling with {KV} Cache Compression},\nauthor={Adrian {\\L}a{\\'n}cucki and Konrad Staniszewski and Piotr Nawrot and Edoardo Ponti},\nbooktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},\nyear={2025},\nurl={https://openreview.net/forum?id=8ZiElzQxf1}\n}"},"title":{"value":"Inference-Time Hyper-Scaling with KV Cache Compression"},"pdf":{"value":"/pdf/bff876f643e028a8e0f2e4f8ae52a2b5e43de99b.pdf"},"venueid":{"value":"NeurIPS.cc/2025/Conference"},"paperhash":{"value":"acucki|inferencetime_hyperscaling_with_kv_cache_compression"},"authorids":{"value":["~Adrian_Łańcucki1","~Konrad_Staniszewski1","~Piotr_Nawrot1","~Edoardo_Ponti1"]},"authors":{"value":["Adrian Łańcucki","Konrad Staniszewski","Piotr Nawrot","Edoardo Ponti"]}},"tmdate":1783627347023,"pdate":1758217128189,"tcdate":1746895168267,"writers":["NeurIPS.cc/2025/Conference","NeurIPS.cc/2025/Conference/Submission17195/Authors"],"signatures":["NeurIPS.cc/2025/Conference/Submission17195/Authors"],"forum":"8ZiElzQxf1","license":"CC BY 4.0","number":17195,"cdate":1746895168267,"readers":["everyone"],"invitations":["NeurIPS.cc/2025/Conference/-/Submission","NeurIPS.cc/2025/Conference/-/Post_Submission","NeurIPS.cc/2025/Conference/Submission17195/-/Full_Submission","NeurIPS.cc/2025/Conference/Submission17195/-/Supplementary_Material","NeurIPS.cc/2025/Conference/-/Edit","NeurIPS.cc/2025/Conference/Submission17195/-/Camera_Ready_Revision"],"mdate":1783627347023,"odate":1761704947293,"domain":"NeurIPS.cc/2025/Conference","id":"8ZiElzQxf1","version":2},{"content":{"venue":{"value":"ICLR 2027 Conference Submission"},"TLDR":{"value":"Chunked KV-cache compression makes retrieval depend on a token's position relative to compression windows: accuracy in DeepSeek-V4 varies by up to 40 points across phases, and we trace this to phase-specialized heads."},"keywords":{"value":["KV Cache Compression","Mechanistic Interpretability","Large Language Models","Optimization Dynamics","Failure Mode Analysis"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Chunked KV-cache compression reduces the memory and attention costs of long-context inference by compressing windows of consecutive tokens into fewer cache entries at a fixed stride. Such compression also introduces a new positional coordinate: a token's phase, or its position relative to compression-window boundaries. We uncover a systematic asymmetry in models using such compression: the same information can be easy to retrieve at one phase and difficult at another. We call this periodic variation in retrieval performance phase-sensitivity. In the DeepSeek-V4 family, long context retrieval accuracy can differ by up to 40 percentage points across phases, revealing periodic weak spots that average benchmark scores can conceal.\n\nTo investigate this behavior, we pretrain a family of transformers from scratch across multiple KV-compression designs, reproducing phase-sensitivity across the tested variants. Mechanistic analysis using causal interventions in these models and DeepSeek-V4 reveals phase specialization: different attention components contribute asymmetrically to retrieving information at different source phases. We further analyze idealized retrieval models, showing how gradient flow dynamics may favor sharp phase specialization. Evaluating models with chunked KV-cache compression therefore requires measuring across compression phases: high average accuracy can coexist with systematic positional failures."},"_bibtex":{"value":"@inproceedings{\nanonymous2026periodic,\ntitle={Periodic Weak Spots: Phase Sensitivity from Chunked {KV}-Cache Compression},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=2dhoYgHiC2},\nnote={under review}\n}"},"title":{"value":"Periodic Weak Spots: Phase Sensitivity from Chunked KV-Cache Compression"},"pdf":{"value":"/pdf/b97235a60dd44a8f3790cee145e972c073360e92.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791230133103,"tcdate":1789537187952,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission23443/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission23443/Authors"],"forum":"2dhoYgHiC2","license":"CC BY 4.0","number":23443,"cdate":1789537187952,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/Submission23443/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791230133103,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"2dhoYgHiC2","version":2},{"content":{"venue":{"value":"CoRR 2026"},"pdf":{"value":"https://arxiv.org/pdf/2604.06694v1"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"wang|audiokv_kv_cache_eviction_in_efficient_large_audio_language_models"},"html":{"value":"https://doi.org/10.48550/arXiv.2604.06694"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2604-06694,\n publtype={informal},\n author={Yuxuan Wang and Peize He and Xiyan Gui and Xiaoqian Liu and Junhao He and Xuyang Liu and Zichen Wen and Xuming Hu and Linfeng Zhang},\n title={AudioKV: KV Cache Eviction in Efficient Large Audio Language Models},\n year={2026},\n month={April},\n cdate={1775001600000},\n journal={CoRR},\n volume={abs/2604.06694},\n url={https://doi.org/10.48550/arXiv.2604.06694}\n}\n"},"abstract":{"value":"Large Audio-Language Models (LALMs) have set new benchmarks in speech processing, yet their deployment is hindered by the memory footprint of the Key-Value (KV) cache during long-context inference. While general KV cache compression techniques excel in LLMs, they often fail in the audio domain by overlooking the intrinsic temporal continuity of acoustic signals. To bridge this gap, we propose AudioKV, a novel framework that robustly prioritizes audio-critical attention heads through a hardware-friendly semantic-acoustic alignment mechanism. Specifically, we identify these modality-specialized heads by analyzing attention scores in ASR tasks and dynamically allocate KV cache budgets preferentially to them. Furthermore, we introduce Spectral Score Smoothing (SSS), an FFT-based global filtering strategy designed to suppress high-frequency noise and recover smooth global trends from importance scores, ensuring more balanced token selection with unprecedented precision. Extensive evaluations across multiple LALMs, including Qwen and Gemma series, demonstrate that AudioKV significantly outperforms baselines while enhancing computational efficiency. Notably, at a 40% compression ratio, AudioKV maintains near-full accuracy on Qwen3-Omni-30B with only a 0.45% drop, whereas traditional methods suffer from catastrophic performance degradation and repetition. Our code will be released after acceptance."},"title":{"value":"AudioKV: KV Cache Eviction in Efficient Large Audio Language Models"},"authors":{"value":[{"fullname":"Yuxuan Wang","username":""},{"fullname":"Peize He","username":"~Peize_He1"},{"fullname":"Xiyan Gui","username":""},{"fullname":"Xiaoqian Liu","username":""},{"fullname":"Junhao He","username":""},{"fullname":"Xuyang Liu","username":""},{"fullname":"Zichen Wen","username":""},{"fullname":"Xuming Hu","username":""},{"fullname":"Linfeng Zhang","username":""}]}},"tmdate":1785851114520,"pdate":1798675200000,"externalIds":["dblp:journals/corr/abs-2604-06694"],"tcdate":1785851111318,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~He_Peize1"],"forum":"0diOZAZXP6","license":"CC BY-SA 4.0","number":115248,"cdate":1775001600000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1785851114520,"domain":"OpenReview.net/Public_Article","id":"0diOZAZXP6","version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2503.02812v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"godey|qfilters_leveraging_qk_geometry_for_efficient_kv_cache_compression"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Nathan_Godey:","https://dblp.org/search/pid/api?q=author:Alessio_Devoto:","https://dblp.org/search/pid/api?q=author:Yu_Zhao_0043:","https://dblp.org/search/pid/api?q=author:Simone_Scardapane:","https://dblp.org/search/pid/api?q=author:Pasquale_Minervini:","https://dblp.org/search/pid/api?q=author:Éric_de_la_Clergerie:","~Benoît_Sagot1"]},"html":{"value":"https://doi.org/10.48550/arXiv.2503.02812"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2503-02812,\n publtype={informal},\n author={Nathan Godey and Alessio Devoto and Yu Zhao and Simone Scardapane and Pasquale Minervini and Éric de la Clergerie and Benoît Sagot},\n title={Q-Filters: Leveraging QK Geometry for Efficient KV Cache Compression},\n year={2025},\n month={March},\n cdate={1740787200000},\n journal={CoRR},\n volume={abs/2503.02812},\n url={https://doi.org/10.48550/arXiv.2503.02812}\n}\n"},"abstract":{"value":"Autoregressive language models rely on a Key-Value (KV) Cache, which avoids re-computing past hidden states during generation, making it faster. As model sizes and context lengths grow, the KV Cache becomes a significant memory bottleneck, which calls for compression methods that limit its size during generation. In this paper, we discover surprising properties of Query (Q) and Key (K) vectors that allow us to efficiently approximate attention scores without computing the attention maps. We propose Q-Filters, a training-free KV Cache compression method that filters out less crucial Key-Value pairs based on a single context-agnostic projection. Contrarily to many alternatives, Q-Filters is compatible with FlashAttention, as it does not require direct access to attention weights. Experimental results in long-context settings demonstrate that Q-Filters is competitive with attention-based compression methods such as SnapKV in retrieval tasks while consistently outperforming efficient compression schemes such as Streaming-LLM in generation setups. Notably, Q-Filters achieves a 99% accuracy in the needle-in-a-haystack task with a x32 compression level while reducing the generation perplexity drop by up to 65% in text generation compared to Streaming-LLM."},"title":{"value":"Q-Filters: Leveraging QK Geometry for Efficient KV Cache Compression"},"authors":{"value":["Nathan Godey","Alessio Devoto","Yu Zhao","Simone Scardapane","Pasquale Minervini","Éric de la Clergerie","Benoît Sagot"]}},"tmdate":1768984394640,"pdate":1767139200000,"externalIds":["dblp:journals/corr/abs-2503-02812"],"tcdate":1768984362969,"writers":["~"],"signatures":["~Benoît_Sagot1"],"forum":"aulWPdjwzP","license":"CC BY-SA 4.0","number":781793,"cdate":1740787200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1768984394640,"domain":"DBLP.org","id":"aulWPdjwzP","version":2},{"content":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Game theory","Model inference","Large Language Models"]},"primary_area":{"value":"infrastructure, software libraries, hardware, systems, etc."},"abstract":{"value":"Large language models (LLMs) have achieved remarkable success in various aspects of human life. However, one of the major challenges in deploying these models is the substantial memory consumption required to store key-value pairs (KV), which imposes significant resource demands. Recent research has focused on KV cache budget allocation, with several approaches proposing head-level budget distribution by evaluating the importance of individual attention heads. These methods, however, assess the importance of heads independently, overlooking their cooperative contributions within the model, which may result in a deviation from their true impact on model performance. In light of this limitation, we propose CoKV, a novel method that models the cooperation between heads in model inference as a cooperative game. By attributing the contribution of each head within the model, CoKV can more effectively allocate the cache budget in KV cache techniques such as eviction and quantization. Extensive experiments demonstrate the effectiveness of CoKV on long-context benchmarks (e.g., LongBench, NIAH, and RULER) and mathematical reasoning benchmarks (e.g., GSM8K and MATH) across multiple model families, including Qwen, Llama, and Mistral. Code is provided in \\url{https://anonymous.4open.science/r/CoKV-40AC}."},"_bibtex":{"value":"@misc{\nsun2025cokv,\ntitle={Co{KV}: Optimizing {LLM} Inference with Game-Theoretic Adaptive {KV} Cache},\nauthor={Qiheng Sun and Hongwei Zhang and Haocheng Xia and Jiayao Zhang and Xiaokai Mao and Junxu Liu and Jinfei Liu and Haibo Hu and Kui Ren},\nyear={2025},\nurl={https://openreview.net/forum?id=1iuaBRErka}\n}"},"title":{"value":"CoKV: Optimizing LLM Inference with Game-Theoretic Adaptive KV Cache"},"pdf":{"value":"/pdf/ddefc83c5649cafb17e18ff43d88717c121cf746.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"sun|cokv_optimizing_llm_inference_with_gametheoretic_adaptive_kv_cache"},"authorids":{"value":["~Qiheng_Sun1","~Hongwei_Zhang12","~Haocheng_Xia1","~Jiayao_Zhang5","~Xiaokai_Mao1","~Junxu_Liu2","~Jinfei_Liu1","~Haibo_Hu2","~Kui_Ren4"]},"authors":{"value":["Qiheng Sun","Hongwei Zhang","Haocheng Xia","Jiayao Zhang","Xiaokai Mao","Junxu Liu","Jinfei Liu","Haibo Hu","Kui Ren"]}},"tmdate":1762963555131,"tcdate":1758004301822,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission6991/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission6991/Authors"],"forum":"1iuaBRErka","license":"CC BY 4.0","number":6991,"cdate":1758004301822,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/-/Submission","ICLR.cc/2026/Conference/-/Post_Submission","ICLR.cc/2026/Conference/Submission6991/-/Full_Submission","ICLR.cc/2026/Conference/-/Withdrawn_Submission"],"mdate":1762963555131,"odate":1759896705795,"domain":"ICLR.cc/2026/Conference","id":"1iuaBRErka","version":2},{"content":{"summary":{"value":"The paper introduces MARché, a training-free decoding scheme that speeds up masked autoregressive (MAR) image generation by avoiding redundant recomputation. It splits tokens each step into an active set (newly generated + a few contextually relevant “refresh” tokens) and a cached set whose key/value projections are reused via cache-aware attention; relevance is decided by attention from newly generated tokens (selective KV refresh). This preserves full-context modeling (using an online-softmax merge that’s equivalent to standard attention) while skipping unnecessary FFN work. On ImageNet 256×256, MARché reports up to ~1.7× latency speedup vs. MAR with only minor FID/IS changes, and requires no architecture changes or retraining."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"see above"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"Training-Free Speedup: MARché offers a significant speedup in masked autoregressive (MAR) image generation without requiring any retraining, making it highly efficient for real-time applications.\n\nEfficient Use of Cache: By reusing key/value projections from previous steps through cache-aware attention, the method avoids redundant calculations, which reduces computational load while preserving high-quality generation.\n\nNo Architecture Modifications: It doesn't require any changes to the underlying model architecture, making it easy to integrate into existing systems.\n\nPerformance Gains: The paper reports a 1.7× latency reduction with only minor decreases in image quality metrics like FID (Fréchet Inception Distance) and IS (Inception Score), showing that it delivers efficiency without significantly compromising performance.\n\nBroad Applicability: The approach can be applied to other autoregressive models without much modification, offering potential for wide adoption in various image generation tasks."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1 The acceleration gains from the proposed MARché method are modest. Despite achieving a 1.7× speedup, the improvement may not be significant enough for certain applications.\n\n2 The method was primarily tested on the ImageNet 256×256 dataset. Its performance and acceleration effect on larger-scale tasks, such as high-resolution images, video generation, or multi-modal data, remain unclear. If the method does not perform well in these scenarios, its general applicability could be limited.\n\n3 The method lacks significant novelty, as it primarily builds on existing techniques like cache management and selective KV refresh, and its scope is limited to optimizing a specific approach in the rapidly advancing field of autoregressive image generation."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1764360477669,"tcdate":1761777060654,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission20908/Reviewer_5nNx"],"signatures":["ICLR.cc/2026/Conference/Submission20908/Reviewer_5nNx"],"forum":"1noRScr1uL","number":3,"license":"CC BY 4.0","cdate":1761777060654,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission20908/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1764360477669,"domain":"ICLR.cc/2026/Conference","replyto":"1noRScr1uL","id":"gqBkxvWA3m","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Efficient AI","Image generation","Large Language Models"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Masked autoregressive (MAR) models unify the strengths of masked and autoregressive generation by predicting tokens in a fixed order using bidirectional attention for image generation. While effective, MAR models suffer from significant computational overhead, as they recompute attention and feed-forward representations for all tokens at every decoding step, despite most tokens remaining semantically stable across steps. We propose a training-free generation framework MARché to address this inefficiency through two key components: cache-aware attention and selective KV refresh. Cache-aware attention partitions tokens into active and cached sets, enabling separate computation paths that allow efficient reuse of previously computed key/value projections without compromising full-context modeling. But a cached token cannot be used indefinitely without recomputation due to the changing contextual information over multiple steps. MARché recognizes this challenge and applies a technique called selective KV refresh. Selective KV refresh identifies contextually relevant tokens based on attention scores from newly generated tokens and updates only those tokens that require recomputation, while preserving image generation quality. MARché significantly reduces redundant computation in MAR without modifying the underlying architecture. Empirically, MARché achieves up to 1.7x speedup with negligible impact on image quality, offering a scalable and broadly applicable solution for efficient masked transformer generation. The code is available at https://anonymous.4open.science/r/MARche-26F0."},"_bibtex":{"value":"@misc{\njiang2026march,\ntitle={{MAR}ch\\'e: Fast Masked Autoregressive Image Generation with Cache-Aware Attention},\nauthor={Chaoyi Jiang and Sungwoo Kim and Lei Gao and Hossein Entezari Zarch and Won Woo Ro and Murali Annavaram},\nyear={2026},\nurl={https://openreview.net/forum?id=1noRScr1uL}\n}"},"title":{"value":"MARché: Fast Masked Autoregressive Image Generation with Cache-Aware Attention"},"pdf":{"value":"/pdf/b6d881b5b3e1c47f4536215bb40f79c3db648bd5.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"jiang|marché_fast_masked_autoregressive_image_generation_with_cacheaware_attention"},"authorids":{"value":["~Chaoyi_Jiang1","~Sungwoo_Kim2","~Lei_Gao3","~Hossein_Entezari_Zarch1","~Won_Woo_Ro1","~Murali_Annavaram1"]},"authors":{"value":["Chaoyi Jiang","Sungwoo Kim","Lei Gao","Hossein Entezari Zarch","Won Woo Ro","Murali Annavaram"]}},"version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"http://arxiv.org/pdf/2502.17501v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"sun|cokv_optimizing_kv_cache_allocation_via_cooperative_game"},"authorids":{"value":["~Qiheng_Sun1","https://dblp.org/search/pid/api?q=author:Hongwei_Zhang:","https://dblp.org/search/pid/api?q=author:Haocheng_Xia:","https://dblp.org/search/pid/api?q=author:Jiayao_Zhang_0006:","https://dblp.org/search/pid/api?q=author:Jinfei_Liu:","https://dblp.org/search/pid/api?q=author:Kui_Ren_0001:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2502.17501"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2502-17501,\n publtype={informal},\n author={Qiheng Sun and Hongwei Zhang and Haocheng Xia and Jiayao Zhang and Jinfei Liu and Kui Ren},\n title={CoKV: Optimizing KV Cache Allocation via Cooperative Game},\n year={2025},\n month={February},\n cdate={1738368000000},\n journal={CoRR},\n volume={abs/2502.17501},\n url={https://doi.org/10.48550/arXiv.2502.17501}\n}\n"},"abstract":{"value":"Large language models (LLMs) have achieved remarkable success on various aspects of human life. However, one of the major challenges in deploying these models is the substantial memory consumption required to store key-value pairs (KV), which imposes significant resource demands. Recent research has focused on KV cache budget allocation, with several approaches proposing head-level budget distribution by evaluating the importance of individual attention heads. These methods, however, assess the importance of heads independently, overlooking their cooperative contributions within the model, which may result in a deviation from their true impact on model performance. In light of this limitation, we propose CoKV, a novel method that models the cooperation between heads in model inference as a cooperative game. By evaluating the contribution of each head within the cooperative game, CoKV can allocate the cache budget more effectively. Extensive experiments show that CoKV achieves state-of-the-art performance on the LongBench benchmark using LLama-3-8B-Instruct and Mistral-7B models."},"title":{"value":"CoKV: Optimizing KV Cache Allocation via Cooperative Game"},"authors":{"value":["Qiheng Sun","Hongwei Zhang","Haocheng Xia","Jiayao Zhang","Jinfei Liu","Kui Ren"]}},"tmdate":1744386570490,"pdate":1735689600000,"tcdate":1744386563797,"writers":["~"],"signatures":["~Qiheng_Sun1"],"forum":"IQ3ANONe7A","license":"CC BY-SA 4.0","number":393279,"cdate":1738368000000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1744386570490,"domain":"DBLP.org","id":"IQ3ANONe7A","version":2},{"content":{"research_area_keywords":{"value":"Efficient Methods for NLP"},"keywords":{"value":["Large Language Models","KV Cache Compression","Residual Quantization"]},"B_use_or_create_scientific_artifacts":{"value":"Yes"},"languages_studied":{"value":"english"},"D4_ethics_review_board_approval":{"value":"N/A"},"C4_parameters_for_packages":{"value":"Yes"},"B6_elaboration":{"value":"We use existing benchmarks (LongBench, RULER) evaluated in their standard configurations. No new datasets were created."},"B1_cite_creators_of_artifacts":{"value":"Yes"},"B2_discuss_the_license_for_artifacts":{"value":"N/A"},"B4_elaboration":{"value":"We use only existing public benchmarks and datasets. No new data was collected."},"B5_elaboration":{"value":"We do not introduce any new artifacts or datasets in this work."},"D_human_subjects_including_annotators":{"value":"No"},"A2_elaboration":{"value":"We discuss potential risks and ethical considerations in the Ethical Considerations section (Appendix G). Our method focuses purely on inference efficiency optimization and does not modify model behavior or training data, introducing no direct ethical concerns."},"C3_elaboration":{"value":"Results are reported from single runs with fixed random seeds to ensure reproducibility, which is standard practice for large-scale LLM evaluation due to computational cost."},"B2_elaboration":{"value":"We only use publicly available models and datasets. We do not distribute any new artifacts in this work."},"B3_elaboration":{"value":"All artifacts used (pretrained models, benchmarks, calibration data) are used solely for research purposes, consistent with their intended use."},"C2_elaboration":{"value":"Experimental setup and hyperparameters are reported in Section 4.1 and Appendix B."},"C4_elaboration":{"value":"Implementation and parameter settings for all methods are reported in Appendix B."},"E_ai_assistants_in_research_or_writing":{"value":"Yes"},"D1_elaboration":{"value":"No human subjects or annotators were used in this work."},"C1_elaboration":{"value":"Model sizes and computing infrastructure are reported in Appendix B."},"B1_elaboration":{"value":"Yes, all artifacts used are cited in Section 2 and Section 4.1."},"C2_experimental_setup_and_hyperparameters":{"value":"Yes"},"D2_elaboration":{"value":"No human subjects or annotators were used in this work."},"E1_elaboration":{"value":"AI assistant usage is disclosed in the AI Assistance Disclosure section (Appendix F)."},"C1_model_size_and_budget":{"value":"Yes"},"venue":{"value":"ACL ARR 2026 May Submission"},"D3_data_consent":{"value":"N/A"},"_bibtex":{"value":"@inproceedings{\nanonymous2026lprquant,\ntitle={{LPRQ}uant: Low-Rank and Predictive {KV} Cache Residual Quantization for {LLM}s},\nauthor={Anonymous},\nbooktitle={Submitted to ACL Rolling Review - May 2026},\nyear={2026},\nurl={https://openreview.net/forum?id=Df0MyQjf1u},\nnote={under review}\n}"},"title":{"value":"LPRQuant: Low-Rank and Predictive KV Cache Residual Quantization for LLMs"},"C3_descriptive_statistics":{"value":"No"},"contribution_types":{"value":["Approaches to low-resource settings"]},"A1_limitations_section":{"value":"This paper has a limitations section."},"B6_statistics_for_data":{"value":"N/A"},"B4_data_contains_personally_identifying_info_or_offensive_content":{"value":"N/A"},"B3_artifact_use_consistent_with_intended_use":{"value":"N/A"},"A2_potential_risks":{"value":"No"},"E1_information_about_use_of_ai_assistants":{"value":"Yes"},"abstract":{"value":"The key-value (KV) cache in large language models is a critical\nmemory bottleneck during inference. We propose LPRQuant, a\ntraining-free compression framework based on hierarchical residual\napproximation. LPRQuant decomposes KV cache compression into\ntwo stages: a structured first-level approximation followed by\nresidual quantization, so that the joint approximation error is\nmultiplicatively bounded and strictly smaller than either stage\nalone. Crucially, we treat the key and value caches asymmetrically\nbased on their distinct structural properties: the key cache,\nwhich exhibits strong low-rank structure, is compressed via PCA\nprojection plus residual quantization; the value cache is\ncompressed via cross-cache linear prediction from the\nreconstructed keys, followed by residual quantization of the\nprediction residual. Experiments on LongBench and RULER across\nthree model families (Llama, Mistral, Qwen) and scales (7B--32B)\nshow that LPRQuant consistently outperforms existing methods,\nachieving near-lossless quality at $\\sim$3-bit and retaining over\n98\\% of full-precision performance at $\\sim$2.5-bit."},"D1_instructions_given_to_participants":{"value":"N/A"},"paper_type":{"value":"Long"},"C_computational_experiments":{"value":"Yes"},"pdf":{"value":"/pdf/d55810ba21fa03d17ff1295e1686d9ab36bbd6f1.pdf"},"research_area":{"value":"Efficient Methods for NLP"},"B5_documentation_of_artifacts":{"value":"N/A"},"EMNLP_2026_AI_Reviewing_Experiment":{"value":"yes"},"D2_recruitment_and_payment":{"value":"N/A"},"venueid":{"value":"aclweb.org/ACL/ARR/2026/May/Submission"}},"tmdate":1790696961063,"tcdate":1779699402142,"writers":["aclweb.org/ACL/ARR/2026/May","aclweb.org/ACL/ARR/2026/May/Submission7195/Authors"],"signatures":["aclweb.org/ACL/ARR/2026/May/Submission7195/Authors"],"forum":"Df0MyQjf1u","license":"CC BY 4.0","number":7195,"cdate":1779699402142,"readers":["everyone"],"invitations":["aclweb.org/ACL/ARR/2026/May/-/Submission","aclweb.org/ACL/ARR/2026/May/-/Edit","aclweb.org/ACL/ARR/2026/May/-/Post_Submission","aclweb.org/ACL/ARR/2026/May/-/Preprint_Post_Submission"],"mdate":1790696961063,"odate":1780385598983,"domain":"aclweb.org/ACL/ARR/2026/May","id":"Df0MyQjf1u","version":2},{"content":{"venue":{"value":"ACL (1) 2026"},"pdf":{"value":"https://aclanthology.org/2026.acl-long.1250.pdf"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"zhao|question_tells_you_where_the_answer_is_intentionaware_longcontext_kv_cache_compression"},"html":{"value":"https://aclanthology.org/2026.acl-long.1250/"},"_bibtex":{"value":"@inproceedings{DBLP:conf/acl/ZhaoFZHZWWHLQ26,\n author={Liang Zhao and Xiaocheng Feng and Weihong Zhong and Lei Huang and Kun Zhu and Baoxin Wang and Dayong Wu and Guoping Hu and Ting Liu and Bing Qin},\n title={Question Tells You Where the Answer Is: Intention-aware Long-Context KV Cache Compression},\n year={2026},\n cdate={1767225600000},\n pages={27153-27169},\n url={https://aclanthology.org/2026.acl-long.1250/},\n booktitle={ACL (1)},\n crossref={conf/acl/2026-1}\n}\n"},"abstract":{"value":"The increasing context window greatly extends the capabilities of large language models, but on the other hand, it incurs an unaffordable memory overhead and computational latency due to the increasing Key-Value (KV) cache size. Recent KV cache compression methods manage to reduce the cache size by dropping irrelevant KVs. However, these methods often fail to identify crucial KVs for generation while excluding others accurately, resulting in severe information loss. To address this gap, we propose **IntentKV**, an intention-aware KV cache eviction method that identifies and retains crucial KVs according to the attention distribution of intention, which semantically reflects the user’s goal and determines which part of the context is relevant. The consistency between the semantics and attention distribution is further substantiated through meticulously designed experiments. On this basis, IntentKV first distinguishes intention tokens from the vanilla context tokens based on their attention distribution distances. Then, the block-wise cumulative attention is calculated via aggregating the intention token attention. Finally, blocks that acquire high cumulative attention are picked and stored in KV cache. We evaluate our method across diverse long-context tasks and models. Results demonstrate that IntentKV can effectively maintain the model performance while reducing the KV cache size from 128K to 2K, leading to a 6.3x increase in decoding speed and 7.8x enhancement in memory efficiency compared to the default setting."},"title":{"value":"Question Tells You Where the Answer Is: Intention-aware Long-Context KV Cache Compression"},"authors":{"value":[{"fullname":"Liang Zhao","username":""},{"fullname":"Xiaocheng Feng","username":"~Xiaocheng_Feng1"},{"fullname":"Weihong Zhong","username":""},{"fullname":"Lei Huang","username":""},{"fullname":"Kun Zhu","username":""},{"fullname":"Baoxin Wang","username":""},{"fullname":"Dayong Wu","username":""},{"fullname":"Guoping Hu","username":""},{"fullname":"Ting Liu","username":""},{"fullname":"Bing Qin","username":""}]}},"tmdate":1784641417963,"pdate":1798675200000,"externalIds":["dblp:conf/acl/ZhaoFZHZWWHLQ26"],"tcdate":1784641410660,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Xiaocheng_Feng1"],"forum":"GyOvmpzD01","license":"CC BY-SA 4.0","number":83805,"cdate":1767225600000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1784641417963,"domain":"OpenReview.net/Public_Article","id":"GyOvmpzD01","version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2511.02230v2"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"li|continuum_efficient_and_robust_multiturn_llm_agent_scheduling_with_kv_cache_timetolive"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Hanchen_Li:","https://dblp.org/search/pid/api?q=author:Qiuyang_Mang:","https://dblp.org/search/pid/api?q=author:Runyuan_He:","https://dblp.org/search/pid/api?q=author:Qizheng_Zhang:","~Huanzhi_Mao1","~Xiaokun_Chen1","~Alvin_Cheung2","https://dblp.org/search/pid/api?q=author:Joseph_Gonzalez_0001:","https://dblp.org/search/pid/api?q=author:Ion_Stoica:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2511.02230"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2511-02230,\n publtype={informal},\n author={Hanchen Li and Qiuyang Mang and Runyuan He and Qizheng Zhang and Huanzhi Mao and Xiaokun Chen and Alvin Cheung and Joseph Gonzalez and Ion Stoica},\n title={Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live},\n year={2025},\n month={November},\n cdate={1761955200000},\n journal={CoRR},\n volume={abs/2511.02230},\n url={https://doi.org/10.48550/arXiv.2511.02230}\n}\n"},"abstract":{"value":"KV cache management is essential for efficient LLM inference. To maximize utilization, existing inference engines evict finished requests' KV cache if new requests are waiting. This policy breaks for agentic workloads, which interleave LLM calls with tools, introducing pauses that prevent effective KV reuse across turns. Since some tool calls have much shorter durations than human response multi-turn chatbot, it would be promising to retain the KV cache in during these tools. However, there are many challenges. First, we need to consider both the potential cost of recomputation or reloading (if CPU offloading enabled) and the increasing queueing delays after eviction from GPU. Second, due to the internal variance of tool call durations, we need the method to remain robust under limited predictability of tool call durations. We present Continuum, a serving system to optimize job completion time for multi-turn agent workloads by introducing time-to-live mechanism for KV cache retaining. For LLM request that generates a tool call, Continuum selectively pins the KV cache in GPU memory with a time-to-live value determined by considering both the reload cost and ordering preserve benefit of retaining KV cache. Moreover, when the TTL expires, the KV cache can be automatically evicted to free up GPU memory, providing robust performance under edge cases. When combined with program-level first-come-first-serve, Continuum preserves multi-turn continuity, and reduces delay for complex agentic workflows. Our evaluation on real-world agentic workloads (SWE-Bench and BFCL) with Llama-3.1 8B/70B shows that Continuum significantly improves the average job completion times and its improvement scales with turn number increase. We release a preview version at: https://github.com/Hanchenli/vllm-continuum"},"title":{"value":"Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live"},"authors":{"value":["Hanchen Li","Qiuyang Mang","Runyuan He","Qizheng Zhang","Huanzhi Mao","Xiaokun Chen","Alvin Cheung","Joseph Gonzalez","Ion Stoica"]}},"tmdate":1791959551754,"pdate":1767139200000,"externalIds":["dblp:journals/corr/abs-2511-02230"],"tcdate":1769384242956,"writers":["~"],"signatures":["~Alvin_Cheung2"],"forum":"d0dGaLoo6W","license":"CC BY-SA 4.0","number":801168,"cdate":1761955200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1791959551754,"domain":"DBLP.org","id":"d0dGaLoo6W","version":2},{"content":{"summary":{"value":"This paper studies KV cache eviction in LLM inference from an output perturbation perspective. To this end, the authors devised a perturbation-constrained objective to retain token caches that preserve the output distribution. In particular, given a total budget b, the authors divide b into two parts: the first part directly evicts caches with lower attention weights; the second part considers the L1 norm of value cache rows to identify important entries. Experiments demonstrate the effectiveness of the proposed method."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"See weakness"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. The proposed method is plug-and-play and easy to use. \n2. The experimental results show clear improvements over baseline methods."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Some recent baselines are not compared [1, 2]. \n2. I believe the authors' definition of perturbation is too narrow—it is defined only based on the output at the current step, and the proposed method is built on this foundation. However, cache eviction is static. Although the perturbation may be reduced at the current step, the KV cache pruned based on this step's perturbation might become important again in subsequent steps. Therefore, how should the perturbation in future steps be measured? Also, how does the proposed method perform in multi-turn dialogue or ultra-long sequence scenarios? \n3. The proposed method shares strong relevance to Quest [3]. Quest approximates attention computation based on the maximum and minimum KV values, while the proposed method uses the L2 norm for approximation. However, the authors do not provide any discussion or comparison with Quest, even though Quest employs a dynamic selection approach. \n4. The comparison of inference speed is not sufficient. There should be results on acceleration for both 128K pre-filling and inference, similar to the presentation in DuoAttention [1].\n\n[1] DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads. In ICLR, 2025. \n[2] PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling. In Arxiv, 2024. \n[3] Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference. In ICML, 2024."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762927196728,"tcdate":1761920561674,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission17242/Reviewer_fAVR"],"signatures":["ICLR.cc/2026/Conference/Submission17242/Reviewer_fAVR"],"forum":"cZ74yWoKYr","number":4,"license":"CC BY 4.0","cdate":1761920561674,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission17242/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762927196728,"domain":"ICLR.cc/2026/Conference","replyto":"cZ74yWoKYr","id":"blFQEpHuCN","forumContent":{"TLDR":{"value":"Select critical KV cache based on constraining the output perturbation"},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Efficient AI","Large Language Model; LLM Inference"]},"supplementary_material":{"value":"/attachment/81b5ff7eada9eb49f17256853d311836a6f18b0e.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Large language models have revolutionized natural language processing but face significant challenges of high storage and runtime costs, due to the transformer architecture's reliance on self-attention, particularly the large Key-Value (KV) cache for long-sequence inference. \nRecent efforts to reduce KV cache size by pruning less critical entries based on attention weights remain empirical and lack formal grounding. This paper presents a formal study on identifying critical KV cache entries by analyzing attention output perturbation.\nOur analysis reveals that, beyond attention weights, the value states within KV entries and pretrained parameter matrices are also crucial. \nBased on this, we propose a perturbation-constrained selection algorithm that optimizes the worst-case output perturbation to identify critical entries. We demonstrate that our algorithm is a universal, plug-and-play enhancement that incurs negligible computational overhead. When integrated with three state-of-the-art cache eviction methods on three distinct LLMs, our algorithm significantly reduces the compression loss by more than half on average across 29 datasets from the Ruler and LongBench benchmarks. Further perturbation analysis, at both the head and layer levels, confirms the principles underlying our effectiveness. This work offers a new, formally grounded perspective to the cache eviction field, opening promising avenues for future research."},"_bibtex":{"value":"@misc{\nfeng2026identify,\ntitle={Identify Critical {KV} Cache in {LLM} Inference from an Output Perturbation Perspective},\nauthor={Yuan Feng and Junlin Lv and Haoyu Guo and Yukun Cao and Xike Xie and S Kevin Zhou},\nyear={2026},\nurl={https://openreview.net/forum?id=cZ74yWoKYr}\n}"},"title":{"value":"Identify Critical KV Cache in LLM Inference from an Output Perturbation Perspective"},"pdf":{"value":"/pdf/788723072db5a45b12c183846653df6dc9c7303d.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"feng|identify_critical_kv_cache_in_llm_inference_from_an_output_perturbation_perspective"},"authorids":{"value":["~Yuan_Feng10","~Junlin_Lv1","~Haoyu_Guo4","~Yukun_Cao1","~Xike_Xie1","~S_Kevin_Zhou1"]},"authors":{"value":["Yuan Feng","Junlin Lv","Haoyu Guo","Yukun Cao","Xike Xie","S Kevin Zhou"]}},"version":2},{"content":{"venue":{"value":"MLSys 2026"},"pdf":{"value":"/pdf/c029c638dc0ebc0ebd6fc266e4b9ab3577bd2702.pdf"},"topics":{"value":["Algorithms: Efficient algorithms for serving LLMs and generative models","Model Serving: System optimizations for model serving"]},"venueid":{"value":"MLSys.org/2026/Conference"},"paperhash":{"value":"lao|opkv_a_highthroughput_plugindriven_framework_for_recallable_sparsity_in_paged_kv_cache_systems"},"authorids":{"value":["laohuazheng@seu.edu.cn","~Xiaofeng_Li7","xurui21@seu.edu.cn","chen_long@seu.edu.cn","zhuxia@seu.edu.cn","ziq8860@gdut.edu.cn"]},"abstract":{"value":"Long-context large language model (LLM) inference faces severe KV cache inflation, making GPU memory a key bottleneck. Existing recallable sparsity methods mitigate memory pressure by offloading non-critical key–value (KV) pairs to CPU memory and recalling them on demand, they are intrusive to KV cache management in the existing inference frameworks and fail to cope with the linearly increasing recall overhead under high batches. To address these limitations, we propose OPKV, a high-throughput plugin-driven framework that seamlessly integrates recallable sparsity into paged KV cache systems and performs unified recall optimization. OPKV introduces a plugin interface that decouples sparsity logic from model and cache management, and applies object reaggregation and hot page hit algorithms to reduce the recall overhead based on the observation of spatial discreteness and temporal locality in critical KV selection. In addition, a local intra-iteration metadata manager is implemented to perform millisecond-level page retrieval and cache eviction. The experimental results show that OPKV helps the SoTA methods attain 1.3 - 1.8x higher decoding throughput under different batches."},"_bibtex":{"value":"@inproceedings{\nlao2026opkv,\ntitle={{OPKV}: A High-Throughput Plugin-Driven Framework for Recallable Sparsity in Paged {KV} Cache Systems},\nauthor={Huazheng Lao and Xiaofeng Li and Rui Xu and Long Chen and Xia Zhu and Jinquan Zhang},\nbooktitle={Ninth Conference on Machine Learning and Systems},\nyear={2026},\nurl={https://openreview.net/forum?id=EB5bgzv4qA}\n}"},"title":{"value":"OPKV: A High-Throughput Plugin-Driven Framework for Recallable Sparsity in Paged KV Cache Systems"},"authors":{"value":["Huazheng Lao","Xiaofeng Li","Rui Xu","Long Chen","Xia Zhu","Jinquan Zhang"]}},"tmdate":1779237679557,"pdate":1773943432932,"tcdate":1773911196319,"writers":["MLSys.org/2026/Conference","MLSys.org/2026/Conference/Submission131/Authors"],"signatures":["MLSys.org/2026/Conference/Submission131/Authors"],"forum":"EB5bgzv4qA","license":"CC BY 4.0","number":131,"cdate":1773911196319,"readers":["everyone"],"invitations":["MLSys.org/2026/Conference/-/Submission","MLSys.org/2026/Conference/-/Post_Submission","MLSys.org/2026/Conference/-/Edit","MLSys.org/2026/Conference/Submission131/-/Camera_Ready_Revision"],"mdate":1779237679557,"odate":1779237077341,"domain":"MLSys.org/2026/Conference","id":"EB5bgzv4qA","version":2},{"content":{"venue":{"value":"EACL 2026"},"venueid":{"value":"OpenReview.net/Archive"},"paperhash":{"value":"patwari|kv_pareto_systemslevel_optimization_of_kv_cache_and_model_compression_for_long_context_inference"},"authorids":{"value":["~Rajeev_Patwari1","~Devleena_Das1","~Sai_Gokhale1","~Elliott_Delaye1","~Ashish_Sirasao1"]},"html":{"value":"https://openreview.net/pdf?id=6srMhZ16P8"},"abstract":{"value":"Long-context Large Language Models (LLMs)\nface significant memory bottlenecks during in\u0002ference due to the linear growth of key-value\n(KV) cache with sequence length. While indi\u0002vidual optimization techniques like KV cache\nquantization, chunked prefill, and model weight\nquantization have shown promise, their joint\neffects and optimal configurations for edge de\u0002ployment remain underexplored. We introduce\nKV Pareto, a systems-level framework that sys\u0002tematically maps the trade-off frontier between\ntotal memory consumption and task accuracy\nacross these three complementary optimization\ntechniques. Our framework evaluates multi\u0002ple LLM architectures (Qwen, Llama, Mis\u0002tral) with varying KV quantization schemes\n(int2/4/8, mixed-precision), granularities (per\u0002token, per-tensor, per-block), and 4-bit weight\nquantization via AWQ. Our framework iden\u0002tifies model-specific Pareto-optimal configura\u0002tions that achieve 68-78% total memory reduc\u0002tion with minimal (1-3%) accuracy degradation\non long-context tasks. We additionally verify\nthe selected frontiers on additional benchmarks\nof Needle-in-a-Haystack, GSM8k and MMLU\nas well as extended context lengths of up to\n128k to demonstrate the practical need of joint\noptimization for efficient LLM inference."},"title":{"value":"KV Pareto: Systems-Level Optimization of KV Cache and Model Compression for Long Context Inference"},"authors":{"value":["Rajeev Patwari","Devleena Das","Sai Gokhale","Elliott Delaye","Ashish Sirasao"]}},"tmdate":1774644986056,"pdate":1774335600000,"tcdate":1774644728995,"writers":["~Rajeev_Patwari1","~Devleena_Das1","~Sai_Gokhale1","~Elliott_Delaye1","~Ashish_Sirasao1"],"signatures":["~Rajeev_Patwari1"],"forum":"GBp5elIb8v","license":"CC BY 4.0","number":46469,"cdate":1774644728995,"readers":["everyone"],"invitations":["OpenReview.net/Archive/-/Direct_Upload"],"mdate":1774644986056,"domain":"OpenReview.net/Archive","id":"GBp5elIb8v","version":2},{"content":{"data_release":{"value":"We authorize the release of our submission and author names to the public in the event of acceptance."},"venue":{"value":"AdaptFM Poster"},"email_sharing":{"value":"We authorize the sharing of all author emails with Program Chairs."},"pdf":{"value":"/pdf/4e21d68a683face2e80796fc2633bcc8a74e80f2.pdf"},"keywords":{"value":["KV Cache Compression","Efficient Decoding"]},"venueid":{"value":"ICML.cc/2026/Workshop/AdaptFM"},"paperhash":{"value":"baek|hybridkv_exploiting_headdominant_reconstruction_for_efficient_queryagnostic_kv_cache_compression"},"authorids":{"value":["~Changwoo_Baek1","~Kyeongbo_Kong1"]},"abstract":{"value":"Efficient key–value (KV) cache compression is crucial for large language models with long contexts. While context-reconstruction attention enables query-agnostic KV compression, its practical use is limited by large compression overhead, i.e., additional prefill-time computation required for reconstruction-based importance scoring beyond standard prefill. We show that reconstruction-based KV importance consistently concentrates on a subset of attention heads, largely independent of the input context. Based on this observation, we propose a hybrid KV cache compression method that combines context-independent head pre-pruning with token-level reconstruction-based pruning. By restricting expensive reconstruction scoring to selected heads, our method significantly reduces compression overhead. Experiments on long-context benchmarks demonstrate up to a 36\\% overhead reduction while largely preserving inference accuracy."},"_bibtex":{"value":"@inproceedings{\nbaek2026hybridkv,\ntitle={{HYBRIDKV}: Exploiting Head-Dominant Reconstruction for Efficient Query-Agnostic {KV} Cache Compression},\nauthor={Changwoo Baek and Kyeongbo Kong},\nbooktitle={AdaptFM: Resource-Adaptive Foundation Model Inference},\nyear={2026},\nurl={https://openreview.net/forum?id=sWF4OAXw7u}\n}"},"title":{"value":"HYBRIDKV: Exploiting Head-Dominant Reconstruction for Efficient Query-Agnostic KV Cache Compression"},"authors":{"value":["Changwoo Baek","Kyeongbo Kong"]}},"tmdate":1784438994071,"pdate":1780353776095,"tcdate":1777779805210,"writers":["ICML.cc/2026/Workshop/AdaptFM","ICML.cc/2026/Workshop/AdaptFM/Submission56/Authors"],"signatures":["ICML.cc/2026/Workshop/AdaptFM/Submission56/Authors"],"forum":"sWF4OAXw7u","license":"CC BY 4.0","number":56,"cdate":1777779805210,"readers":["everyone"],"invitations":["ICML.cc/2026/Workshop/AdaptFM/-/Submission","ICML.cc/2026/Workshop/AdaptFM/-/Submission_Change_Before_Reviewing","ICML.cc/2026/Workshop/AdaptFM/-/Submission_Release","ICML.cc/2026/Workshop/AdaptFM/Submission56/-/Camera_Ready_Revision","ICML.cc/2026/Workshop/AdaptFM/-/Edit"],"mdate":1784438994071,"odate":1780353776095,"domain":"ICML.cc/2026/Workshop/AdaptFM","id":"sWF4OAXw7u","version":2},{"content":{"venue":{"value":"ICLR 2027 Conference Submission"},"TLDR":{"value":"Repeated KV-cache pruning does not compound error: with surviving content held fixed, quality approaches a clean re-prefill, and cost is governed by how long junk lingers before eviction, not by how often the cache is pruned."},"keywords":{"value":["LLM; KV cache; agent context"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Long-running agents accumulate stale context from failed retrievals and superseded reasoning, occupying GPU memory and distorting attention. Removing it at the text layer forces a re-encode; at the KV-cache layer it is nearly free: excise rows, re-rotate survivor positions, continue. Whether that composite cache \\emph{behaves} like an honest re-prefill of the surviving tokens is untested, and production practice stays conservative. We test it directly, holding surviving content byte-identical and varying only operation count (Q1), window position (Q2), and eviction timing (Q3); relevance is oracle-given and the mechanism a reused RoPE rotation, so the operation itself is the only remaining variable.\n\nThe caution inverts: repeating the cheap operation makes the result \\emph{more} faithful, not less. Over 8~rounds at 92.9\\% pruning, composite-KV quality reaches 91.3\\% of a clean re-prefill and improves with round count, with no drift over 26+ real-trajectory rounds (Q1). Re-indexing costs only $+$0.8\\% and matters only when survivor positions exceed the model's trained window, confirmed by a pre-registered model swap (Q2). Timing is the genuine open variable (Q3): correctness slips only a little as junk lingers, a small, task-dependent cost that timely pruning keeps low. The worry that repeated pruning compounds error is unsupported; it is timing, not repetition, that costs."},"_bibtex":{"value":"@inproceedings{\nanonymous2026safety,\ntitle={Safety in Numbers: The {KV} Cache Was Never That Fragile to Repeated Pruning},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=QcY2SttxGj},\nnote={under review}\n}"},"title":{"value":"Safety in Numbers: The KV Cache Was Never That Fragile to Repeated Pruning"},"pdf":{"value":"/pdf/fd6a85444bd96effd3ad83ff049972f558a947e0.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791220513790,"tcdate":1787803687112,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission5249/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission5249/Authors"],"forum":"QcY2SttxGj","license":"CC BY 4.0","number":5249,"cdate":1787803687112,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission5249/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791220513790,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"QcY2SttxGj","version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"http://arxiv.org/pdf/2503.01586v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"zhou|elitekv_scalable_kv_cache_compression_via_rope_frequency_selection_and_joint_lowrank_projection"},"authorids":{"value":["~Yuhao_Zhou3","https://dblp.org/search/pid/api?q=author:Sirui_Song:","https://dblp.org/search/pid/api?q=author:Boyang_Liu:","https://dblp.org/search/pid/api?q=author:Zhiheng_Xi:","~Senjie_Jin1","https://dblp.org/search/pid/api?q=author:Xiaoran_Fan:","https://dblp.org/search/pid/api?q=author:Zhihao_Zhang_0002:","https://dblp.org/search/pid/api?q=author:Wei_Li:","https://dblp.org/search/pid/api?q=author:Xuanjing_Huang_0001:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2503.01586"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2503-01586,\n publtype={informal},\n author={Yuhao Zhou and Sirui Song and Boyang Liu and Zhiheng Xi and Senjie Jin and Xiaoran Fan and Zhihao Zhang and Wei Li and Xuanjing Huang},\n title={EliteKV: Scalable KV Cache Compression via RoPE Frequency Selection and Joint Low-Rank Projection},\n year={2025},\n month={March},\n cdate={1740787200000},\n journal={CoRR},\n volume={abs/2503.01586},\n url={https://doi.org/10.48550/arXiv.2503.01586}\n}\n"},"abstract":{"value":"Rotary Position Embedding (RoPE) enables each attention head to capture multi-frequency information along the sequence dimension and is widely applied in foundation models. However, the nonlinearity introduced by RoPE complicates optimization of the key state in the Key-Value (KV) cache for RoPE-based attention. Existing KV cache compression methods typically store key state before rotation and apply the transformation during decoding, introducing additional computational overhead. This paper introduces EliteKV, a flexible modification framework for RoPE-based models supporting variable KV cache compression ratios. EliteKV first identifies the intrinsic frequency preference of each head using RoPElite, selectively restoring linearity to certain dimensions of key within attention computation. Building on this, joint low-rank compression of key and value enables partial cache sharing. Experimental results show that with minimal uptraining on only $0.6\\%$ of the original training data, RoPE-based models achieve a $75\\%$ reduction in KV cache size while preserving performance within a negligible margin. Furthermore, EliteKV consistently performs well across models of different scales within the same family."},"title":{"value":"EliteKV: Scalable KV Cache Compression via RoPE Frequency Selection and Joint Low-Rank Projection"},"authors":{"value":["Yuhao Zhou","Sirui Song","Boyang Liu","Zhiheng Xi","Senjie Jin","Xiaoran Fan","Zhihao Zhang","Wei Li","Xuanjing Huang"]}},"tmdate":1773804920583,"pdate":1735689600000,"externalIds":["dblp:journals/corr/abs-2503-01586"],"tcdate":1758867248917,"writers":["~"],"signatures":["~Yuhao_Zhou3"],"forum":"hHo9BPyNMK","license":"CC BY-SA 4.0","number":630007,"cdate":1740787200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1773804920583,"domain":"DBLP.org","id":"hHo9BPyNMK","version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2503.10714v3"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"liu|zeromerge_parameterfree_kv_cache_compression_for_memoryefficient_longcontext_llms"},"html":{"value":"https://doi.org/10.48550/arXiv.2503.10714"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2503-10714,\n publtype={informal},\n author={Xin Liu and Pei Liu and Guoming Tang},\n title={ZeroMerge: Parameter-Free KV Cache Compression for Memory-Efficient Long-Context LLMs},\n year={2025},\n month={March},\n cdate={1740787200000},\n journal={CoRR},\n volume={abs/2503.10714},\n url={https://doi.org/10.48550/arXiv.2503.10714}\n}\n"},"abstract":{"value":"The linear growth of key-value (KV) cache memory and quadratic computational in attention mechanisms complexity pose significant bottlenecks for large language models (LLMs) in long-context processing. While existing KV cache optimization methods address these challenges through token pruning or feature merging, they often incur irreversible information loss or require costly parameter retraining. To this end, we propose ZSMerge, a dynamic KV cache compression framework designed for efficient cache management, featuring three key operations: (1) fine-grained memory allocation guided by multi-dimensional token importance metrics at head-level granularity, (2) a residual merging mechanism that preserves critical context through compensated attention scoring, and (3) a zero-shot adaptation mechanism compatible with diverse LLM architectures without requiring retraining. ZSMerge significantly enhances memory efficiency and inference speed with negligible performance degradation across LLMs. When applied to LLaMA2-7B, it demonstrates a 20:1 compression ratio for key-value cache retention (reducing memory footprint to 5\\% of baseline) while sustaining comparable generation quality, coupled with triple throughput gains at extreme 54k-token contexts that eliminate out-of-memory failures. The code is available at https://github.com/SusCom-Lab/ZSMerge."},"title":{"value":"ZeroMerge: Parameter-Free KV Cache Compression for Memory-Efficient Long-Context LLMs"},"authors":{"value":[{"fullname":"Xin Liu","username":""},{"fullname":"Pei Liu","username":"~Pei_Liu7"},{"fullname":"Guoming Tang","username":""}]}},"tmdate":1783953957134,"pdate":1767139200000,"externalIds":["dblp:journals/corr/abs-2503-10714"],"tcdate":1783953940756,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Pei_Liu7"],"forum":"NC96cjEwNy","license":"CC BY-SA 4.0","number":52761,"cdate":1740787200000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1783953957134,"domain":"OpenReview.net/Public_Article","id":"NC96cjEwNy","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"https://arxiv.org/pdf/2412.16187v3"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"liu|hashevict_a_preattention_kv_cache_eviction_strategy_using_localitysensitive_hashing"},"authorids":{"value":["","","","","","","~Furong_Huang1","",""]},"html":{"value":"https://doi.org/10.48550/arXiv.2412.16187"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2412-16187,\n publtype={informal},\n author={Minghui Liu and Tahseen Rabbani and Tony O'Halloran and Ananth Sankaralingam and Mary-Anne Hartley and Brian J. Gravelle and Furong Huang and Cornelia Fermüller and Yiannis Aloimonos},\n title={HashEvict: A Pre-Attention KV Cache Eviction Strategy using Locality-Sensitive Hashing},\n year={2024},\n cdate={1704067200000},\n journal={CoRR},\n volume={abs/2412.16187},\n url={https://doi.org/10.48550/arXiv.2412.16187}\n}\n"},"abstract":{"value":"Transformer-based large language models (LLMs) use the key-value (KV) cache to significantly accelerate inference by storing the key and value embeddings of past tokens. However, this cache consumes significant GPU memory. In this work, we introduce HashEvict, an algorithm that uses locality-sensitive hashing (LSH) to compress the KV cache. HashEvict quickly locates tokens in the cache that are cosine dissimilar to the current query token. This is achieved by computing the Hamming distance between binarized Gaussian projections of the current token query and cached token keys, with a projection length much smaller than the embedding dimension. We maintain a lightweight binary structure in GPU memory to facilitate these calculations. Unlike existing compression strategies that compute attention to determine token retention, HashEvict makes these decisions pre-attention, thereby reducing computational costs. Additionally, HashEvict is dynamic - at every decoding step, the key and value of the current token replace the embeddings of a token expected to produce the lowest attention score. We demonstrate that HashEvict can compress the KV cache by 30%-70% while maintaining high performance across reasoning, multiple-choice, long-context retrieval and summarization tasks."},"title":{"value":"HashEvict: A Pre-Attention KV Cache Eviction Strategy using Locality-Sensitive Hashing"},"authors":{"value":["Minghui Liu","Tahseen Rabbani","Tony O'Halloran","Ananth Sankaralingam","Mary-Anne Hartley","Brian J. Gravelle","Furong Huang","Cornelia Fermüller","Yiannis Aloimonos"]}},"tmdate":1772421379902,"pdate":1735603200000,"externalIds":["dblp:journals/corr/abs-2412-16187"],"tcdate":1772421339638,"writers":["~"],"signatures":["~Furong_Huang1"],"forum":"6GZHIjTUvt","license":"CC BY-SA 4.0","number":843602,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1772421379902,"domain":"DBLP.org","id":"6GZHIjTUvt","version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2511.11907v2"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"zhang|kvswap_diskaware_kv_cache_offloading_for_longcontext_ondevice_inference"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Huawei_Zhang:","~Chunwei_Xia1","https://dblp.org/search/pid/api?q=author:Zheng_Wang:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2511.11907"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2511-11907,\n publtype={informal},\n author={Huawei Zhang and Chunwei Xia and Zheng Wang},\n title={KVSwap: Disk-aware KV Cache Offloading for Long-Context On-device Inference},\n year={2025},\n month={November},\n cdate={1761955200000},\n journal={CoRR},\n volume={abs/2511.11907},\n url={https://doi.org/10.48550/arXiv.2511.11907}\n}\n"},"abstract":{"value":"Language models (LMs) underpin emerging mobile and embedded AI applications like meeting and video summarization and document analysis, which often require processing multiple long-context inputs. Running an LM locally on-device improves privacy, enables offline use, and reduces cost, but long-context inference quickly hits a \\emph{memory capacity wall} as the key-value (KV) cache grows linearly with context length and batch size. Existing KV-cache offloading schemes are designed to transfer cache data from GPU memory to CPU memory; however, they are not suitable for embedded and mobile systems, where the CPU and GPU (or NPU) typically share a unified memory and the non-volatile secondary storage (disk) offers limited I/O bandwidth. We present KVSwap, a software framework tailored for local devices that achieves high memory efficiency while effectively leveraging disk storage. KVSwap stores the full cache on disk, uses highly compact in-memory metadata to predict which entries to preload, overlaps computation with hardware-aware disk access, and orchestrates read patterns to match storage device characteristics. Our evaluation shows that across representative LMs and storage types, KVSwap delivers higher throughput under tight memory budgets while maintaining generation quality over existing KV cache offloading schemes."},"title":{"value":"KVSwap: Disk-aware KV Cache Offloading for Long-Context On-device Inference"},"authors":{"value":["Huawei Zhang","Chunwei Xia","Zheng Wang"]}},"tmdate":1771352796832,"pdate":1767139200000,"externalIds":["dblp:journals/corr/abs-2511-11907"],"tcdate":1771352789233,"writers":["~"],"signatures":["~Chunwei_Xia1"],"forum":"oiIx1mrgUB","license":"CC BY-SA 4.0","number":825165,"cdate":1761955200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1771352796832,"domain":"DBLP.org","id":"oiIx1mrgUB","version":2},{"content":{"venue":{"value":"ICML 2025 spotlightposter"},"keywords":{"value":["Long-Context LLMs","Inference","KV Cache Optimization"]},"_bibtex":{"value":"@inproceedings{\nsun2025shadowkv,\ntitle={Shadow{KV}: {KV} Cache in Shadows for High-Throughput Long-Context {LLM} Inference},\nauthor={Hanshi Sun and Li-Wen Chang and Wenlei Bao and Size Zheng and Ningxin Zheng and Xin Liu and Harry Dong and Yuejie Chi and Beidi Chen},\nbooktitle={Forty-second International Conference on Machine Learning},\nyear={2025},\nurl={https://openreview.net/forum?id=oa7MYAO6h6}\n}"},"title":{"value":"ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference"},"paperhash":{"value":"sun|shadowkv_kv_cache_in_shadows_for_highthroughput_longcontext_llm_inference"},"TLDR":{"value":"High-Throughput Long-Context LLM Inference System"},"primary_area":{"value":"deep_learning->large_language_models"},"abstract":{"value":"With the widespread deployment of long-context large language models (LLMs), there has been a growing demand for efficient support of high-throughput inference. However, as the key-value (KV) cache expands with the sequence length, the increasing memory footprint and the need to access it for decoding both result in low throughput when serving long-context LLMs. While various dynamic sparse attention methods have been proposed to accelerate inference while maintaining generation quality, they either fail to sufficiently reduce GPU memory usage or introduce significant decoding latency by offloading the KV cache to the CPU. We present ShadowKV, a high-throughput long-context LLM inference system that stores the low-rank key cache and offloads the value cache to reduce the memory footprint for larger batch sizes and longer sequences. To minimize decoding latency, ShadowKV employs an accurate KV selection strategy that reconstructs minimal sparse KV pairs on-the-fly. By evaluating ShadowKV on benchmarks like RULER, LongBench, and models such as Llama-3.1-8B and GLM-4-9B-1M, we demonstrate that it achieves up to 6$\\times$ larger batch sizes and 3.04$\\times$ higher throughput on an A100 GPU without sacrificing accuracy, even surpassing the performance achievable with infinite batch size under the assumption of infinite GPU memory."},"link_to_code":{"value":"https://github.com/ByteDance-Seed/ShadowKV"},"pdf":{"value":"/pdf/1c3d546a8774ef52eab7d0cceee17dc967b36eae.pdf"},"lay_summary":{"value":"Large language models that can understand very long texts require significant computer memory, which makes them slow and expensive to use, especially when many people are using them at once. Our research introduces ShadowKV, a new system designed to significantly speed up these models. ShadowKV works by smartly managing the model's memory: it keeps a small, compressed version of the most important data on the GPU and moves the less critical data to CPU. When the model needs information, ShadowKV quickly finds and retrieves only what's essential, avoiding delays and ensuring the model remains accurate. This allows us to handle many more requests and much longer texts simultaneously, making powerful long-context AI more efficient and accessible for broader use."},"venueid":{"value":"ICML.cc/2025/Conference"},"authorids":{"value":["~Hanshi_Sun1","~Li-Wen_Chang1","~Wenlei_Bao1","~Size_Zheng1","~Ningxin_Zheng1","~Xin_Liu51","~Harry_Dong1","~Yuejie_Chi1","~Beidi_Chen1"]},"authors":{"value":["Hanshi Sun","Li-Wen Chang","Wenlei Bao","Size Zheng","Ningxin Zheng","Xin Liu","Harry Dong","Yuejie Chi","Beidi Chen"]}},"tmdate":1753292938083,"pdate":1746105371369,"tcdate":1737563523765,"writers":["ICML.cc/2025/Conference","ICML.cc/2025/Conference/Submission6997/Authors"],"signatures":["ICML.cc/2025/Conference/Submission6997/Authors"],"forum":"oa7MYAO6h6","license":"CC BY 4.0","number":6997,"cdate":1737563523765,"readers":["everyone"],"invitations":["ICML.cc/2025/Conference/-/Submission","ICML.cc/2025/Conference/-/Post_Submission","ICML.cc/2025/Conference/Submission6997/-/Full_Submission","ICML.cc/2025/Conference/-/Edit","ICML.cc/2025/Conference/Submission6997/-/Camera_Ready_Revision"],"mdate":1753292938083,"odate":1750231190185,"domain":"ICML.cc/2025/Conference","id":"oa7MYAO6h6","version":2},{"content":{"venue":{"value":"AAAI 2025"},"pdf":{"value":"https://ojs.aaai.org/index.php/AAAI/article/download/34773/36928"},"venueid":{"value":"dblp.org/conf/AAAI/2025"},"paperhash":{"value":"zandieh|qjl_1bit_quantized_jl_transform_for_kv_cache_quantization_with_zero_overhead"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Amir_Zandieh:","~Majid_Daliri1","https://dblp.org/search/pid/api?q=author:Insu_Han:"]},"html":{"value":"https://doi.org/10.1609/aaai.v39i24.34773"},"_bibtex":{"value":"@inproceedings{DBLP:conf/aaai/ZandiehDH25,\n author={Amir Zandieh and Majid Daliri and Insu Han},\n title={QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overhead},\n year={2025},\n cdate={1735689600000},\n pages={25805-25813},\n url={https://doi.org/10.1609/aaai.v39i24.34773},\n booktitle={AAAI},\n crossref={conf/aaai/2025}\n}\n"},"abstract":{"value":"Serving LLMs requires substantial memory due to the storage requirements of Key-Value (KV) embeddings in the KV cache, which grows with sequence length. An effective approach to compress KV cache is quantization. However, traditional quantization methods face significant memory overhead due to the need to store quantization constants (at least a zero point and a scale) in full precision per data block. Depending on the block size, this overhead can add 1 or 2 bits per quantized number. We introduce QJL, a new quantization approach that consists of a Johnson-Lindenstrauss (JL) transform followed by sign-bit quantization. In contrast to existing methods, QJL eliminates memory overheads by removing the need for storing quantization constants. We propose an asymmetric estimator for the inner product of two vectors and demonstrate that applying QJL to one vector and a standard JL transform without quantization to the other provides an unbiased estimator with minimal distortion. We have developed an efficient implementation of the QJL sketch and its corresponding inner product estimator, incorporating a lightweight CUDA kernel for optimized computation. When applied across various LLMs and NLP tasks to quantize the KV cache to only 3 bits, QJL demonstrates a more than fivefold reduction in KV cache memory usage without compromising accuracy, all while achieving faster runtime."},"title":{"value":"QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overhead"},"authors":{"value":["Amir Zandieh","Majid Daliri","Insu Han"]}},"tmdate":1752255245997,"pdate":1735689600000,"tcdate":1752255237227,"writers":["~"],"signatures":["~Majid_Daliri1"],"forum":"y3YDhISnpJ","license":"CC BY-SA 4.0","number":566987,"cdate":1735689600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1752255245997,"domain":"DBLP.org","id":"y3YDhISnpJ","version":2},{"content":{"venue":{"value":"SLLM"},"keywords":{"value":["KV cache","Quantization","JL"]},"abstract":{"value":"Serving LLMs requires substantial memory due to the storage requirements of Key-Value (KV) embeddings in the KV cache, which grows with sequence length. \n An effective approach to compress KV cache is quantization.\n However, traditional quantization methods face significant memory overhead due to the need to store quantization constants (at least a zero point and a scale) in full precision per data block.\n Depending on the block size, this overhead can add 1 or 2 bits per quantized number.\n We introduce QJL, a new quantization approach that consists of a Johnson-Lindenstrauss (JL) transform followed by sign-bit quantization. \n In contrast to existing methods, QJL eliminates memory overheads by removing the need for storing quantization constants.\n We propose an asymmetric estimator for the inner product of two vectors and demonstrate that applying QJL to one vector and a standard JL transform without quantization to the other provides an unbiased estimator with minimal distortion.\n We have developed an efficient implementation of the QJL sketch and its corresponding inner product estimator, incorporating a lightweight CUDA kernel for optimized computation.\n When applied across various LLMs and NLP tasks to quantize the KV cache to only 3 bits, QJL demonstrates a more than fivefold reduction in KV cache memory usage without compromising accuracy, all while achieving faster runtime."},"_bibtex":{"value":"@inproceedings{\nzandieh2025qjl,\ntitle={{QJL}: 1-Bit Quantized {JL} Transform for {KV} Cache Quantization with Zero Overhead},\nauthor={Amir Zandieh and Majid Daliri and Insu Han},\nbooktitle={Sparsity in LLMs (SLLM): Deep Dive into Mixture of Experts, Quantization, Hardware, and Inference},\nyear={2025},\nurl={https://openreview.net/forum?id=dRilx3vAIH}\n}"},"title":{"value":"QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overhead"},"Anonymization":{"value":"This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs."},"pdf":{"value":"/pdf/b470267d0a4e09ab770de6b004939bc7c6114304.pdf"},"venueid":{"value":"ICLR.cc/2025/Workshop/SLLM"},"paperhash":{"value":"zandieh|qjl_1bit_quantized_jl_transform_for_kv_cache_quantization_with_zero_overhead"},"authorids":{"value":["~Amir_Zandieh2","~Majid_Daliri1","~Insu_Han1"]},"Track":{"value":"long paper (up to 4 pages)"},"authors":{"value":["Amir Zandieh","Majid Daliri","Insu Han"]}},"tmdate":1744174190550,"pdate":1741181471524,"tcdate":1738944198986,"writers":["ICLR.cc/2025/Workshop/SLLM","ICLR.cc/2025/Workshop/SLLM/Submission44/Authors"],"signatures":["ICLR.cc/2025/Workshop/SLLM/Submission44/Authors"],"forum":"dRilx3vAIH","license":"CC BY 4.0","number":44,"cdate":1738944198986,"readers":["everyone"],"invitations":["ICLR.cc/2025/Workshop/SLLM/-/Submission","ICLR.cc/2025/Workshop/SLLM/-/Post_Submission","ICLR.cc/2025/Workshop/SLLM/-/Edit","ICLR.cc/2025/Workshop/SLLM/Submission44/-/Camera_Ready"],"mdate":1744174190550,"odate":1741196735324,"domain":"ICLR.cc/2025/Workshop/SLLM","id":"dRilx3vAIH","version":2},{"content":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["KV-Cache","Quantization","JL Transform","Fast AutoRegressive Models"]},"supplementary_material":{"value":"/attachment/dc64dcfb3830d84d7b024fba926a99e964fdfb26.zip"},"primary_area":{"value":"optimization"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Serving LLMs requires substantial memory due to the storage requirements of Key-Value (KV) embeddings in the KV cache, which grows with sequence length. An effective approach to compress KV cache is quantization.However, traditional quantization methods face significant memory overhead due to the need to store quantization constants (at least a zero point and a scale) in full precision per data block. Depending on the block size, this overhead can add 1 or 2 bits per quantized number. We introduce QJL, a new quantization approach that consists of a Johnson-Lindenstrauss (JL) transform followed by sign-bit quantization. In contrast to existing methods, QJL eliminates memory overheads by removing the need for storing quantization constants. We propose an asymmetric estimator for the inner product of two vectors and demonstrate that applying QJL to one vector and a standard JL transform without quantization to the other provides an unbiased estimator with minimal distortion. We have developed an efficient implementation of the QJL sketch and its corresponding inner product estimator, incorporating a lightweight CUDA kernel for optimized computation. When applied across various LLMs and NLP tasks to quantize the KV cache to only 3 bits, QJL demonstrates a more than fivefold reduction in KV cache memory usage without compromising accuracy, all while achieving faster runtime."},"_bibtex":{"value":"@misc{\nzandieh2024qjl,\ntitle={{QJL}: 1-Bit Quantized {JL} Transform for {KV} Cache Quantization with Zero Overhead},\nauthor={Amir Zandieh and Majid Daliri and Insu Han},\nyear={2024},\nurl={https://openreview.net/forum?id=xHPVGmLXjd}\n}"},"title":{"value":"QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overhead"},"pdf":{"value":"/pdf/cdc1e01340268d77492b4572ac32cee9ab022030.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"zandieh|qjl_1bit_quantized_jl_transform_for_kv_cache_quantization_with_zero_overhead"},"authorids":{"value":["~Amir_Zandieh2","~Majid_Daliri1","~Insu_Han1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Amir Zandieh","Majid Daliri","Insu Han"]}},"tmdate":1732459634024,"tcdate":1727405572056,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission8696/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission8696/Authors"],"forum":"xHPVGmLXjd","license":"CC BY 4.0","number":8696,"cdate":1727405572056,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/-/Submission","ICLR.cc/2025/Conference/-/Post_Submission","ICLR.cc/2025/Conference/Submission8696/-/Full_Submission","ICLR.cc/2025/Conference/-/Withdrawn_Submission"],"mdate":1732459634024,"odate":1728008565725,"domain":"ICLR.cc/2025/Conference","id":"xHPVGmLXjd","version":2},{"content":{"summary":{"value":"The paper presents LogQuant, a KV-cache quantization technique based on the observation that the positions of high-attention spikes follow a log distribution."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"- The authors use the terms “quantization” and “compression” alternatively, however, these terms have different meanings.\n- I couldn’t find which quantization/compression technique is used in LogQuant. Furthermore, I found Figure 5 (and its caption) unclear."},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"- The paper points out an interesting observation regarding the distribution of the positions of high-attention spikes. \n- The paper targets the KV-cache quantization of LLM, which has a high importance in deploying LLM."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- The paper is mostly based on observation and proposes a heuristical solution. \n- The presentation of LogQuant (mostly section 3) requires revision to clarify the solution better."}},"nonreaders":[],"tmdate":1731428721146,"tcdate":1730283557023,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission6683/Reviewer_4LpZ"],"signatures":["ICLR.cc/2025/Conference/Submission6683/Reviewer_4LpZ"],"forum":"PHrqpxUczG","number":2,"license":"CC BY 4.0","cdate":1730283557023,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission6683/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428721146,"domain":"ICLR.cc/2025/Conference","replyto":"PHrqpxUczG","id":"H4EScQVRrl","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Large Language Models","KV Cache","Memory Efficiency","Quantization"]},"supplementary_material":{"value":"/attachment/7fa7d0836c5ed387858195e8315bfb01114d1307.zip"},"primary_area":{"value":"infrastructure, software libraries, hardware, systems, etc."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"We introduce LogQuant, a groundbreaking 2-bit quantization technique for KV Cache in large language model (LLM) inference, delivering substantial memory savings while preserving superior performance. Previous methods either assume that later tokens are more important or attempt to predict important tokens based on earlier attention patterns. Both approaches, however, can result in performance bottlenecks or frequent mispredictions.\n\nLogQuant takes a different approach. By applying a log-based filtering mechanism, it selectively compresses the KV Cache across the entire context, achieving better performance with the same or even reduced memory footprint compared to existing methods. In benchmark tests, it enhances throughput by 25\\% and boosts batch size by 60\\% without increasing memory consumption. For challenging tasks such as Math and Code Completion, LogQuant improves accuracy by 40\\% to 200\\% at the same compression ratio, outperforming comparable techniques. LogQuant integrates effortlessly with popular inference frameworks like Python’s \\texttt{transformers} library and will be made open-source upon publication."},"_bibtex":{"value":"@misc{\nhan2025logquant,\ntitle={LogQuant: Log-Distributed 2-Bit Quantization of {KV} Cache with Superior Accuracy Preservation},\nauthor={CHEN Han and Zicong Jiang and Zining Zhang and Bingsheng He and Luo Pingyi and Mian Lu and Yuqiang Chen},\nyear={2025},\nurl={https://openreview.net/forum?id=PHrqpxUczG}\n}"},"title":{"value":"LogQuant: Log-Distributed 2-Bit Quantization of KV Cache with Superior Accuracy Preservation"},"pdf":{"value":"/pdf/4cb7667121487b6516f5e70daa7a9343ba1692fb.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"han|logquant_logdistributed_2bit_quantization_of_kv_cache_with_superior_accuracy_preservation"},"authorids":{"value":["~CHEN_Han2","~Zicong_Jiang2","~Zining_Zhang1","~Bingsheng_He1","~Luo_Pingyi1","~Mian_Lu1","~Yuqiang_Chen1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["CHEN Han","Zicong Jiang","Zining Zhang","Bingsheng He","Luo Pingyi","Mian Lu","Yuqiang Chen"]}},"version":2},{"content":{"venue":{"value":"ACL ARR 2026 October Submission"},"research_area_keywords":{"value":"LLM efficiency; NLP in resource-constrained settings"},"keywords":{"value":["KV Cache Compression"]},"_bibtex":{"value":"@inproceedings{\nanonymous2026motskv,\ntitle={MoTs{KV}: Rethinking {KV} Cache Compression through Joint Memory and Access Optimization},\nauthor={Anonymous},\nbooktitle={Submitted to ACL Rolling Review - October 2026},\nyear={2026},\nurl={https://openreview.net/forum?id=JClQDiRbug},\nnote={under review}\n}"},"title":{"value":"MoTsKV: Rethinking KV Cache Compression through Joint Memory and Access Optimization"},"contribution_types":{"value":"Approaches to low-compute settings (efficiency)"},"abstract":{"value":"Long-context inference incurs substantial memory and bandwidth costs from the KV cache. We propose \\textbf{MoTsKV}, a training-free framework that jointly manages cache residency and decoding-time KV access. MoTsKV employs a hierarchical moment tree to identify informative historical KV states through query-guided selection, together with a closed-form allocation rule that balances permanent retention and sparse reading. By leveraging first- and second-order key statistics, the tree enables coarse-to-fine selection while preserving the original key-value representations. During decoding, a compact 2-bit key index enables query-adaptive routing, retrieving only selected full-precision KV states for attention computation. This two-stage design coordinates persistent KV selection with dynamic sparse access under a unified resource budget, without additional training or modification of the underlying model. Extensive evaluations on LongBench and RULER demonstrate competitive performance across models and KV budgets. On LongBench, MoTsKV outperforms RocketKV by 1.44 points on average at the smallest nominal budget, with particularly strong gains under aggressive compression. An optimized implementation further achieves substantial decode-only throughput improvements over FullKV and RocketKV in matched inference settings."},"paper_type":{"value":"Long"},"pdf":{"value":"/pdf/e0f603c2c8d8aa87720722b3e9b5ff99fece1a56.pdf"},"research_area":{"value":"Efficient Methods for NLP"},"venueid":{"value":"aclweb.org/ACL/ARR/2026/October/Submission"}},"tmdate":1792185044062,"tcdate":1791888864319,"writers":["aclweb.org/ACL/ARR/2026/October","aclweb.org/ACL/ARR/2026/October/Submission12042/Authors"],"signatures":["aclweb.org/ACL/ARR/2026/October/Submission12042/Authors"],"forum":"JClQDiRbug","license":"CC BY 4.0","number":12042,"cdate":1791888864319,"readers":["everyone"],"invitations":["aclweb.org/ACL/ARR/2026/October/-/Submission","aclweb.org/ACL/ARR/2026/October/-/Responsible_NLP_Checklist_Status","aclweb.org/ACL/ARR/2026/October/-/Edit","aclweb.org/ACL/ARR/2026/October/-/Preprint_Post_Submission"],"mdate":1792185044062,"odate":1791894541772,"domain":"aclweb.org/ACL/ARR/2026/October","id":"JClQDiRbug","version":2},{"content":{"summary":{"value":"The paper proposes FAFO (Fumble Around, Find Out), a speculative decoding method that integrates lossy KV-cache compression and verification within a single forward pass.\nInstead of relying on a separate draft model, FAFO constructs an n-gram (typically 2-gram) candidate pool that predicts likely next tokens from the compressed KV cache.\nVerification with the full KV cache occurs in parallel, enabled by a masked-attention design that allows both “fumble” (draft) and “find-out” (verify) computations in one pass."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"+ **Scalability under larger batches:**\nHave you evaluated FAFO with larger batch sizes or multi-sequence decoding? Since FAFO adds per-step computation and coordination, it would be helpful to understand whether the speedup persists as batch size increases or on GPUs with limited compute resources.\n\n+ **Computation vs. memory trade-off:**\nCan you quantify how much additional FLOPs FAFO introduces per decoding step compared to a standard n-gram (lookahead) or speculative decoding baseline? This would clarify when the approach transitions from memory-bound to compute-bound."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"+ **Interesting idea:** FAFO’s capability to perform n-gram drafting using a compressed KV cache and verify the n-gram drafts within the same forward pass is novel and technically elegant.\n\n+ **Clear presentation:** The paper clearly contrasts FAFO with prior speculative decoding methods (TriForce, Lookahead, Self-SD). Figure 2 effectively illustrates the mechanism, making the concept easy to grasp."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"+ **Limited workload consideration:** Experiments are limited to batch = 1. It remains unclear whether FAFO’s advantages persist under larger-batch or multi-sequence settings, where system-level bottlenecks may differ.\n\n+ **Scalability concern:** FAFO introduces extra computation per decoding step. This additional work could shift the decoding regime from memory-bound to compute-bound, potentially diminishing speedups on larger batches or lower-end GPUs."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762924023738,"tcdate":1761956533366,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission13383/Reviewer_Cruo"],"signatures":["ICLR.cc/2026/Conference/Submission13383/Reviewer_Cruo"],"forum":"oSk9tP5Mgs","number":3,"license":"CC BY 4.0","cdate":1761956533366,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission13383/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762924023738,"domain":"ICLR.cc/2026/Conference","replyto":"oSk9tP5Mgs","id":"YT5JouDAzc","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Lossless Compression","KV cache","LLM","Speculative Decoding"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Lossy KV cache compression is a well-explored subfield of machine learning efficiency, with improved latency being one of its major gains. However, lossy compression techniques can fumble from time to time, exhibiting various — and often catastrophic — failure patterns that are not only difficult to resolve but sometimes even hard to identify in the first place, making the direct deployment of models with compressed KV cache a risky endeavor. In this work, we explore a way to preserve lossless generation quality while still benefiting from the acceleration provided by attending only to a compressed KV cache. Specifically, we draw inspiration from the n-gram candidate pool decoding paradigm pioneered by Lookahead Decoding — a largely overlooked and underdeveloped way to achieve efficient yet lossless decoding — where we purposely allow the model to Fumble Around with compressed KV cache to generate multiple lossy ''n-gram guesses'' with just one forward pass, while Find Out via lossless verification in the same forward pass in truly parallel fashion. From a conceptual standpoint, our proposed framework is compatible with all typical static or dynamic KV cache compression methods from the token dropping realm, thus opening up a new avenue for the stagnant n-gram decoding paradigm. Practically, we show that — with careful system support — this framework presents many useful traits that similar draftless baselines (e.g., Self-Speculative Decoding) simply cannot achieve, such as requiring only one set of KV cache and being far less sensitive to model, task, and input-length scenarios. Our comprehensive empirical results show FAFO provides 1.20-2.71x latency speedup over the original model, while consistently outperforming other lossless + draftless solutions by a large margin."},"_bibtex":{"value":"@misc{\nle2026fafo,\ntitle={{FAFO}: Lossy {KV} Cache Compression for Lossless Inference Acceleration via Draftless Fumble Decoding},\nauthor={Hoang Anh Duy Le and Shaochen Zhong and Yifan Lu and Yingtong Dou and Jiayi Yuan and Yu-Neng Chuang and Xiran Fan and Guanchu Wang and Yuzhong Chen and Xia Hu},\nyear={2026},\nurl={https://openreview.net/forum?id=oSk9tP5Mgs}\n}"},"title":{"value":"FAFO: Lossy KV Cache Compression for Lossless Inference Acceleration via Draftless Fumble Decoding"},"pdf":{"value":"/pdf/491f077ab86c5e61bbd4e473081e2322aaeae17e.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"le|fafo_lossy_kv_cache_compression_for_lossless_inference_acceleration_via_draftless_fumble_decoding"},"authorids":{"value":["~Hoang_Anh_Duy_Le1","~Shaochen_Zhong1","~Yifan_Lu8","~Yingtong_Dou1","~Jiayi_Yuan1","~Yu-Neng_Chuang1","~Xiran_Fan1","~Guanchu_Wang1","~Yuzhong_Chen2","~Xia_Hu4"]},"authors":{"value":["Hoang Anh Duy Le","Shaochen Zhong","Yifan Lu","Yingtong Dou","Jiayi Yuan","Yu-Neng Chuang","Xiran Fan","Guanchu Wang","Yuzhong Chen","Xia Hu"]}},"version":2},{"content":{"venue":{"value":"ACL ARR 2026 October Submission"},"research_area_keywords":{"value":"Multimodality and Language Grounding to Vision, Robotics and Beyond"},"keywords":{"value":["KV-Cache Eviction","Ordinary Differential Equation","Vision–languagemodels"]},"_bibtex":{"value":"@inproceedings{\nanonymous2026rodeodeguided,\ntitle={{RODE}:{ODE}-Guided Token Potential for Visual {KV}-Cache Eviction},\nauthor={Anonymous},\nbooktitle={Submitted to ACL Rolling Review - October 2026},\nyear={2026},\nurl={https://openreview.net/forum?id=rdk50QMhun},\nnote={under review}\n}"},"title":{"value":"RODE:ODE-Guided Token Potential for Visual KV-Cache Eviction"},"contribution_types":{"value":"Theory"},"abstract":{"value":"Vision--language models (VLMs) retain key--value (KV) states for visual tokens during autoregressive generation, making visual-cache management important for inference. Existing cache selection commonly ranks tokens from prefill or current-query attention, although token relevance can vary across network depth and decode steps. We introduce \\rode{}, an attention-led visual KV eviction method that fits a constrained first-order ordinary differential equation (ODE) to each visual token's prefill-attention profile and uses its positive fitted depth-wise growth as a bounded potential in token ranking. A delayed-eviction policy re-scores low-ranked tokens over subsequent decode steps and batch-deletes candidates that remain below an adaptive rank threshold. We evaluate \\rode{} with LLaVA-1.5-7B on TextVQA, GQA, and frame-level captioning on StoryStream Rabbids across three visual-token retention ratios. Results show near-Full-Cache performance on the two question-answering benchmarks; among eviction methods, RODE gives the strongest BLEU-1 on two of the three retention ratios. Token-profile and decode-trace analyses characterize the ODE potential and the pending, cancellation, and batch-deletion behavior."},"paper_type":{"value":"Long"},"pdf":{"value":"/pdf/af989390112ea3266a3c85310ba5c58c2891fc84.pdf"},"research_area":{"value":"Multimodality and Language Grounding to Vision, Robotics and Beyond"},"venueid":{"value":"aclweb.org/ACL/ARR/2026/October/Submission"}},"tmdate":1792185048163,"tcdate":1791891735762,"writers":["aclweb.org/ACL/ARR/2026/October","aclweb.org/ACL/ARR/2026/October/Submission12896/Authors"],"signatures":["aclweb.org/ACL/ARR/2026/October/Submission12896/Authors"],"forum":"rdk50QMhun","license":"CC BY 4.0","number":12896,"cdate":1791891735762,"readers":["everyone"],"invitations":["aclweb.org/ACL/ARR/2026/October/-/Submission","aclweb.org/ACL/ARR/2026/October/-/Responsible_NLP_Checklist_Status","aclweb.org/ACL/ARR/2026/October/-/Edit","aclweb.org/ACL/ARR/2026/October/-/Preprint_Post_Submission"],"mdate":1792185048163,"odate":1791894541772,"domain":"aclweb.org/ACL/ARR/2026/October","id":"rdk50QMhun","version":2},{"content":{"venue":{"value":"CBW Poster"},"pdf":{"value":"/pdf/7b059f580a87c1e462f41d77e391784e22514397.pdf"},"keywords":{"value":["Compaction","KV cache optimization","Long Reasoning","Memory"]},"venueid":{"value":"colmweb.org/COLM/2026/Workshop/CBW"},"paperhash":{"value":"inhar|beyond_merging_and_dropping_adaptive_online_kv_cache_compaction"},"authorids":{"value":["~Izaaz_Inhar1","~Faeyza_Rishad_Ardi1","~Muhammad_Reza_Qorib1","~Alham_Fikri_Aji1","~Farid_Adilazuarda1"]},"abstract":{"value":"As Large Language Models (LLMs) increasingly rely on long Chain-of-Thought reasoning, linear Key-Value (KV) cache growth creates severe memory and latency bottlenecks. Existing learnable cache compressions are typically offline and enforce rigid, uniform compression ratios. To address this, we propose an adaptive online compaction method that combines token merging with learnable token dropping. By extending the STILL compactor with Gumbel-Sigmoid gating, our approach dynamically prunes unimportant latent token representations from a dense, merged initialization. This hybrid mechanism enables optimal, heterogeneous cache allocation across layers and heads while retaining reasoning capabilities. Trained sequentially on Open-R1 \\citep{openr1}, our compactor reaches stable 8x compression while keeping teacher--student divergence bounded across repeated sequential compaction. On Qwen3-1.7B, our method achieves 27.50\\% Avg@4 on AIME 26 and 19.69\\% on HMMT-Feb 26, outperforming vanilla STILL by 2.50 and 0.76 percentage points and DMS by 1.67 and 1.51 points, respectively. It reduces peak KV payload by approximately 83\\% relative to uncompressed inference and improves throughput by more than 22\\% over vanilla STILL."},"_bibtex":{"value":"@inproceedings{\ninhar2026beyond,\ntitle={Beyond Merging and Dropping: Adaptive Online {KV} Cache Compaction},\nauthor={Izaaz Inhar and Faeyza Rishad Ardi and Muhammad Reza Qorib and Alham Fikri Aji and Farid Adilazuarda},\nbooktitle={COLM Workshop on Context Beyond the Window: Persistent Knowledge in Language Models},\nyear={2026},\nurl={https://openreview.net/forum?id=rcR66RU5u1}\n}"},"title":{"value":"Beyond Merging and Dropping: Adaptive Online KV Cache Compaction"},"authors":{"value":["Izaaz Inhar","Faeyza Rishad Ardi","Muhammad Reza Qorib","Alham Fikri Aji","Farid Adilazuarda"]}},"tmdate":1788340629336,"pdate":1785868354479,"tcdate":1782132115374,"writers":["colmweb.org/COLM/2026/Workshop/CBW","colmweb.org/COLM/2026/Workshop/CBW/Submission31/Authors"],"signatures":["colmweb.org/COLM/2026/Workshop/CBW/Submission31/Authors"],"forum":"rcR66RU5u1","license":"CC BY 4.0","number":31,"cdate":1782132115374,"readers":["everyone"],"invitations":["colmweb.org/COLM/2026/Workshop/CBW/-/Submission","colmweb.org/COLM/2026/Workshop/CBW/-/Post_Submission","colmweb.org/COLM/2026/Workshop/CBW/-/Edit","colmweb.org/COLM/2026/Workshop/CBW/Submission31/-/Camera_Ready_Revision"],"mdate":1788340629336,"odate":1785868354479,"domain":"colmweb.org/COLM/2026/Workshop/CBW","id":"rcR66RU5u1","version":2},{"content":{"venue":{"value":"ICML 2026 regular"},"keywords":{"value":["KV cache","long-context LLM inference","approximate nearest neighbor search","drift robustness","GPU acceleration","vector search","quantization","memory offloading"]},"_bibtex":{"value":"@inproceedings{\nqi2026pariskv,\ntitle={Paris{KV}: Fast and Drift-Robust {KV}-Cache Retrieval for Long-Context {LLM}s},\nauthor={Yanlin Qi and Xinhang Chen and Huiqiang Jiang and Qitong Wang and Botao Peng and Themis Palpanas},\nbooktitle={Forty-third International Conference on Machine Learning},\nyear={2026},\nurl={https://openreview.net/forum?id=wxD4wTYQXt}\n}"},"title":{"value":"ParisKV: Fast and Drift-Robust KV-Cache Retrieval for Long-Context LLMs"},"paperhash":{"value":"qi|pariskv_fast_and_driftrobust_kvcache_retrieval_for_longcontext_llms"},"originally_submitted_PDF":{"value":"/pdf/a473c16bae56868e258198890c13a3c5e0c42e02.pdf"},"TLDR":{"value":"ParisKV enables fast and drift-robust long-context inference for LLMs via GPU-native, cluster-free KV-cache retrieval."},"primary_area":{"value":"deep_learning->large_language_models"},"abstract":{"value":"KV-cache retrieval is essential for long-context LLM inference, yet existing methods struggle with distribution drift and high latency at scale. We introduce **ParisKV**, a drift-robust, GPU-native KV-cache retrieval framework based on collision-based candidate selection, followed by a quantized inner-product reranking estimator. For million-token contexts, ParisKV supports CPU-offloaded KV caches via Unified Virtual Addressing (UVA), enabling on-demand top-$k$ fetching with minimal overhead. ParisKV matches or outperforms full attention quality on both **long-input** and **long-generation** benchmarks. It achieves state-of-the-art long-context decoding efficiency: it matches or exceeds full-attention speed even at batch size 1 for long contexts, delivers up to **2.8$\\times$** higher throughput within full attention’s runnable range, and scales to **million-token** contexts where full attention runs out of memory. At million-token scale, ParisKV reduces decode latency by **17$\\times$** and **44$\\times$** compared to MagicPIG and PQCache, respectively—two state-of-the-art KV-cache top-$k$ retrieval baselines; code is available at https://github.com/amy-77/ParisKV/tree/main."},"link_to_code":{"value":"https://github.com/amy-77/ParisKV/tree/main"},"pdf":{"value":"/pdf/7fcf23aec70e9810d2f4519fe0d43b5719fc2f26.pdf"},"lay_summary":{"value":"Large language models can read and generate very long texts, but doing so requires storing and searching through a large amount of past information, which makes inference slow and memory-intensive. ParisKV is a new system that helps language models efficiently find the most useful parts of this stored information when processing long contexts. Instead of checking everything, ParisKV quickly narrows down the important candidates and retrieves only the most relevant information. This makes long-context inference faster and more memory-efficient, while maintaining the model’s answer quality. Our results show that ParisKV can support much longer inputs and generations than standard approaches, and can be especially useful when full attention becomes too expensive or runs out of memory."},"venueid":{"value":"ICML.cc/2026/Conference"},"authorids":{"value":["~Yanlin_Qi1","~Xinhang_Chen1","~Huiqiang_Jiang2","~Qitong_Wang5","~Botao_Peng1","~Themis_Palpanas1"]},"authors":{"value":["Yanlin Qi","Xinhang Chen","Huiqiang Jiang","Qitong Wang","Botao Peng","Themis Palpanas"]}},"tmdate":1790067350258,"pdate":1777576418274,"tcdate":1769034236558,"writers":["ICML.cc/2026/Conference","ICML.cc/2026/Conference/Submission12946/Authors"],"signatures":["ICML.cc/2026/Conference/Submission12946/Authors"],"forum":"wxD4wTYQXt","license":"CC BY 4.0","number":12946,"cdate":1769034236558,"readers":["everyone"],"invitations":["ICML.cc/2026/Conference/-/Submission","ICML.cc/2026/Conference/-/Post_Submission","ICML.cc/2026/Conference/Submission12946/-/Full_Submission","ICML.cc/2026/Conference/-/Edit","ICML.cc/2026/Conference/Submission12946/-/Camera_Ready_Revision"],"mdate":1790067350258,"odate":1782341948214,"domain":"ICML.cc/2026/Conference","id":"wxD4wTYQXt","version":2},{"content":{"venue":{"value":"MLSys 2026"},"TLDR":{"value":"Efficient long-context LLM serving with temporal stability-aware hierarchical KV-cache management, achieving up to 1.55x higher throughput than vLLM while maintaining accuracy."},"pdf":{"value":"/pdf/d2dbce7862e70eea6d743e6981e4b42098aa74f2.pdf"},"keywords":{"value":["Efficient LLM Inference","Long-context LLM","KV cache offloading","Sparse Attention","Attention head stability"]},"topics":{"value":["Model Serving: System optimizations for model serving"]},"venueid":{"value":"MLSys.org/2026/Conference"},"paperhash":{"value":"takbir|flexicache_leveraging_temporal_stability_of_attention_heads_for_efficient_kv_cache_management"},"authorids":{"value":["ntakbir@uci.edu","hamidra@uci.edu","dutt@uci.edu","sangeetha.aj@uci.edu"]},"abstract":{"value":"Large Language Model (LLM) serving is increasingly constrained by the growing size of the key-value (KV) cache, which scales with both context length and generation length. Prior work shows that attention is dominated by a small subset of critical tokens, yet existing systems struggle to exploit this efficiently without degrading accuracy, especially in long generation. We make a key observation: the temporal stability of these critical tokens varies significantly across KV heads. Some heads consistently focus on the same tokens, while others shift frequently. Building on this insight, we introduce FlexiCache, a hierarchical KV-cache management system that leverages the temporal stability of KV heads to reduce GPU memory usage and computation overhead, while preserving model accuracy. FlexiCache classifies KV heads as stable or unstable: it retains all KV-cache pages from unstable heads in GPU memory, whereas for stable heads, it keeps only the top-K pages on the GPU and offloads the rest to host memory. By exploiting temporal stability, FlexiCache performs periodic reranking for stable heads to fetch newly promoted top pages. Implemented atop vLLM, FlexiCache reduces GPU memory footprint for long-context requests by up to 70%, improves offline serving throughput by 1.38–1.55×, and lowers online token latency by 1.6–2.1×, all while maintaining accuracy in long-context, long-generation scenarios."},"_bibtex":{"value":"@inproceedings{\ntakbir2026flexicache,\ntitle={FlexiCache: Leveraging Temporal Stability of Attention Heads for Efficient {KV} Cache Management},\nauthor={Nazmul Takbir and HamidReza Alikhani Koshkak and Nikil Dutt and Sangeetha Abdu Jyothi},\nbooktitle={Ninth Conference on Machine Learning and Systems},\nyear={2026},\nurl={https://openreview.net/forum?id=GgX6dPJx9M}\n}"},"title":{"value":"FlexiCache: Leveraging Temporal Stability of Attention Heads for Efficient KV Cache Management"},"authors":{"value":["Nazmul Takbir","HamidReza Alikhani Koshkak","Nikil Dutt","Sangeetha Abdu Jyothi"]}},"tmdate":1779237679512,"pdate":1773943432794,"tcdate":1773911193357,"writers":["MLSys.org/2026/Conference","MLSys.org/2026/Conference/Submission128/Authors"],"signatures":["MLSys.org/2026/Conference/Submission128/Authors"],"forum":"GgX6dPJx9M","license":"CC BY 4.0","number":128,"cdate":1773911193357,"readers":["everyone"],"invitations":["MLSys.org/2026/Conference/-/Submission","MLSys.org/2026/Conference/-/Post_Submission","MLSys.org/2026/Conference/-/Edit","MLSys.org/2026/Conference/Submission128/-/Camera_Ready_Revision"],"mdate":1779237679512,"odate":1779237077263,"domain":"MLSys.org/2026/Conference","id":"GgX6dPJx9M","version":2},{"content":{"review":{"value":"The paper addresses a really timely issue: how to alleviate the memory footprint of the KV cache during LLM inference. I find their strategy quite original: identifying five common attention structures, each of them allowing to save the K and V of specific tokens instead of the full sequence length. \nThe authors based their approach on two assumptions: 1) different layers and heads need different attention structures and 2) the attention structure identified during the prompt encoding is maintained during the token generation. I have some caveats about how the authors justify the validity of the assumptions. \nThe authors show results using GPT-4 as an evaluator, for model generations with full KV cache and their compressed KV cache (plus other non-adaptive methods in the literature). This is fine but I miss other more standard comparisons, like MMLU or WinoGrande.\n\nPros:\n- Instead of more naive approaches to compress the KV cache like sparse sampling or non-adaptive strategies, their method identifies the particular attention structure of a prompt (out of five possible structures) and applies it to compress the KV cache.\n- Contrary to other methods in the literature that apply the same structure to all heads and layers, their authors show the need to be more granular and apply different structures per head and layer.\n- Their method surpasses previous non-adaptive methods in the literature and matches the results using full KV caching, when using Llama 1 generation and comparing the generation quality with GPT-4.\n\nCons:\n- For the assumption of \"Head Distinctive Attention Structure\" in section 4.1, I'm missing more detail about 1) how the attention structure is identified in each head and 2) a comparison with more datasets that are not GSM8K (which is based on math word problems with fixed structure). I'm not sure if the claim of \"for the middle layers, the attention map focuses more on special tokens\" would hold for other datasets.\n- For the assumption of \"Profile Tends to be Consistent in One Sequence\" in section 4.2, I have two issues with their analysis: \n1) the authors only show two intermediate layers (33 and 23), and a few heads (0, 2, 3 and 5). Could the authors add a summarized view of all the layers and heads, as it is done in section 4.1? Maybe a metric like \"average derivative\" is enough for the visualization.\n2) the authors only generate 30 tokens, which I find not long enough compared to a normal 1024 generation length. In addition, some plots (see the ones for head 3) present a significant trend even after 30 tokens.\n- In line 257, it says that there's a \"~50% memory reduction in Llama-65B\", but in Table 1 I see that even with 44.9% reduction it is not reaching the 45% win rate. Are the authors rounding up that 44.9% to 50%? It seems quite a high jump in my opinion. Same with the numbers for the other model sizes. \n- I didn't find a mention about \"sequence length\" or number of generated tokens in the experiments of section 5.1. I believe it is important to mention it, specially given the assumption taken in section 4.2. Is the sequence length reported in line 254 (i.e. 512) the one used in the experiments in Figures 2 and 5? \n\nQuestions to clarify:\n- In the line 11 of the abstract, the authors mention \"FastGen\" for the first time. At this point, it is not clear what \"FastGen\" refers to.\n- In line 54, the citation is incorrect: the citation should refer to Llama 1, not Llama 2. Later in the experiments of section 5 this is correctly stated.\n- In line 94, the authors use the term \"Prompt encoding\". I've usually seen this part referred as \"prefill\" in the literature.\n- Typo in line 175: remove \"in Figure As\".\n- Typo in line 195: change \"visualized\" for \"visualize\"."},"confidence":{"value":4},"title":{"value":"The KV cache can be compressed by identifying the attention score structure of the prompt encoding"}},"nonreaders":[],"tmdate":1698687068018,"tcdate":1697937602280,"writers":["NeurIPS.cc/2023/Workshop/WANT","NeurIPS.cc/2023/Workshop/WANT/Submission18/Reviewer_n8Gu"],"signatures":["NeurIPS.cc/2023/Workshop/WANT/Submission18/Reviewer_n8Gu"],"forum":"e9D2STGwLJ","number":1,"license":"CC BY 4.0","cdate":1697937602280,"readers":["everyone"],"invitations":["NeurIPS.cc/2023/Workshop/WANT/Submission18/-/Official_Review","NeurIPS.cc/2023/Workshop/WANT/-/Edit"],"mdate":1698687068018,"domain":"NeurIPS.cc/2023/Workshop/WANT","replyto":"e9D2STGwLJ","id":"MvvtEf2txg","forumContent":{"TLDR":{"value":"We introduce adaptive KV cache compression, a plug-and-play method that reduces the memory footprint of generative inference for Large Language Models (LLMs) and accelerates its generation throughput."},"venue":{"value":"WANT@NeurIPS 2023 Poster"},"pdf":{"value":"/pdf/2c08ecfb3bea6504b81d9329b8bb5f1f5023f46c.pdf"},"keywords":{"value":["Large Language Model","Efficient Inference","Generative Inference","Key-Value Cache"]},"venueid":{"value":"NeurIPS.cc/2023/Workshop/WANT"},"paperhash":{"value":"ge|model_tells_you_what_to_discard_adaptive_kv_cache_compression_for_llms"},"authorids":{"value":["~Suyu_Ge1","~Yunan_Zhang1","~Liyuan_Liu3","~Minjia_Zhang1","~Jiawei_Han1","~Jianfeng_Gao1"]},"abstract":{"value":"In this study, we introduce adaptive KV cache compression, a plug-and-play method that reduces the memory footprint of generative inference for Large Language Models (LLMs). Different from the conventional KV cache that retains key and value vectors for all context tokens, we conduct targeted profiling to discern the intrinsic structure of attention modules. Based on the recognized structure, we then construct the KV cache in an adaptive manner: evicting long-range contexts on attention heads emphasizing local contexts, discarding non-special tokens on attention heads centered on special tokens, and only employing the standard KV cache for attention heads that broadly attend to all tokens. Moreover, with the lightweight attention profiling used to guide the construction of the adaptive KV cache, FastGen can be deployed without resource-intensive fine-tuning or re-training. In our experiments across various asks, FastGen demonstrates substantial reduction on GPU memory consumption with negligible generation quality loss. We will release our code and the compatible CUDA kernel for reproducibility."},"_bibtex":{"value":"@inproceedings{\nge2023model,\ntitle={Model Tells You What to Discard: Adaptive {KV} Cache Compression for {LLM}s},\nauthor={Suyu Ge and Yunan Zhang and Liyuan Liu and Minjia Zhang and Jiawei Han and Jianfeng Gao},\nbooktitle={Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@NeurIPS 2023)},\nyear={2023},\nurl={https://openreview.net/forum?id=e9D2STGwLJ}\n}"},"title":{"value":"Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs"},"authors":{"value":["Suyu Ge","Yunan Zhang","Liyuan Liu","Minjia Zhang","Jiawei Han","Jianfeng Gao"]}},"version":2},{"content":{"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["reinforcement learning with verifiable rewards","asynchronous rollout","KV cache quantization","efficient LLM training","reasoning models","mixed-precision inference"]},"supplementary_material":{"value":"/attachment/9c3c736f0d09ae477cc9cbb415a1708810a1c1ce.zip"},"primary_area":{"value":"infrastructure, software libraries, hardware, systems, etc."},"abstract":{"value":"Asynchronous rollout improves the efficiency of reinforcement learning with verifiable rewards by mitigating long-tail generation. However, sustaining higher concurrency compresses the same context-processing work into fewer decode steps, increasing the per-step cost of KV-cache processing. We characterize this \\emph{KV concentration} and show that KV-cache quantization provides substantially greater acceleration than weight-only or weight-activation quantization under the resulting workload. We further find that INT4 KV quantization can degrade learning despite importance correction. To address this, we propose \\emph{KV4-RL}, which combines calibration-free INT4 KV quantization with selective INT8 protection of attention sinks and recent context. This bounded protection reduces sampler mismatch while retaining efficient execution over the compressed cache. In Qwen3-8B-Base GRPO training, KV4-RL achieves $1.80\\times$ rollout and $1.27\\times$ full-iteration speedup over BF16 across training, with rollout speedup reaching $1.97\\times$ as responses lengthen. Under BF16 evaluation, final-policy mean accuracy across six math benchmarks remains within 0.5 percentage points of BF16-rollout training for both Qwen3-4B-Base and Qwen3-8B-Base."},"_bibtex":{"value":"@inproceedings{\nanonymous2026kvrl,\ntitle={{KV}4-{RL}: Unlocking Faster Asynchronous Rollouts for {RLVR} with {INT}4 {KV} Cache Quantization},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=LFTfgeBVb3},\nnote={under review}\n}"},"title":{"value":"KV4-RL: Unlocking Faster Asynchronous Rollouts for RLVR with INT4 KV Cache Quantization"},"pdf":{"value":"/pdf/0f8ed7520d328888e52e5a2b8889936201fd9b97.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791235227361,"tcdate":1789805137680,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission57216/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission57216/Authors"],"forum":"LFTfgeBVb3","license":"CC BY 4.0","number":57216,"cdate":1789805137680,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission57216/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791235227361,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"LFTfgeBVb3","version":2},{"content":{"venue":{"value":"ICML 2026 regular"},"keywords":{"value":["Quantization","Video Generation","KV Cache"]},"_bibtex":{"value":"@inproceedings{\nxi2026quant,\ntitle={Quant VideoGen: Auto-Regressive Long Video Generation via 2-Bit {KV}-Cache Quantization},\nauthor={Haocheng Xi and Shuo Yang and Yilong Zhao and Muyang Li and Han Cai and Xingyang Li and Yujun Lin and Zhuoyang Zhang and Jintao Zhang and Xiuyu Li and Zhiying Xu and Jun Wu and Chenfeng Xu and Ion Stoica and Song Han and Kurt Keutzer},\nbooktitle={Forty-third International Conference on Machine Learning},\nyear={2026},\nurl={https://openreview.net/forum?id=r9UJgVH1Fx}\n}"},"title":{"value":"Quant VideoGen: Auto-Regressive Long Video Generation via 2-Bit KV-Cache Quantization"},"paperhash":{"value":"xi|quant_videogen_autoregressive_long_video_generation_via_2bit_kvcache_quantization"},"originally_submitted_PDF":{"value":"/pdf/faeff417280646319be2445fd7b18ef84671d584.pdf"},"TLDR":{"value":"We propose a KV cache quantization method for efficient autoregressive video generation."},"primary_area":{"value":"deep_learning->generative_models_and_autoencoders"},"abstract":{"value":"Despite rapid progress in auto-regressive video diffusion, we identify an emerging system–algorithm bottleneck that limits both deployability and generation quality: KV-cache memory. In auto-regressive video generation models, the KV-cache grows with generation history and quickly dominates GPU memory (often ≥30 GB), preventing deployment on widely available hardware. More critically, memory-bounded KV budgets constrain the effective working memory, directly degrading long-horizon consistency in identity, layout, and motion.\nTo address this challenge, we present Quant VideoGen (QVG), a training-free KV-cache quantization framework for auto-regressive video diffusion models. QVG exploits video’s inherent spatiotemporal redundancy via Semantic-Aware Smoothing, producing low-magnitude, quantization-friendly residuals. Building on this, QVG introduces Progressive Residual Quantization, a coarse-to-fine multi-stage scheme that further reduces quantization error while enabling a smooth quality–memory trade-off.\nAcross LongCat-Video, HY-WorldPlay, and Self-Forcing, QVG establishes a new Pareto frontier between quality and memory efficiency, reducing KV memory by up to 7.0× with less than 4% end-to-end latency overhead, while delivering significantly better generation quality than existing baselines."},"link_to_code":{"value":"https://github.com/svg-project/Quant-VideoGen"},"pdf":{"value":"/pdf/4d8cbd9ba2bf9986c926e4d5303604e8beb2ccbc.pdf"},"lay_summary":{"value":"Modern video generation models are becoming increasingly powerful, but generating long videos remains difficult. Many of these models create video step by step, while keeping a memory of the frames and details they have already produced. As the video gets longer, this memory becomes very large and can take up most of the GPU, making the model hard to run on common hardware. When the model is forced to use a smaller memory budget, video quality can also suffer: characters may change appearance, scene layouts may drift, and motion may become less consistent over time.\n\nWe introduce Quant VideoGen, or QVG, a training-free method that makes this video memory much smaller without retraining the model. QVG takes advantage of the fact that neighboring video regions are often similar in space and time. It first reorganizes the stored information so that the remaining differences are easier to compress, then uses a multi-stage quantization method to reduce memory while preserving important visual details. Across several auto-regressive video diffusion models, QVG reduces KV-cache memory by up to 7.0× with less than 4% extra latency, while maintaining better video quality than existing memory-saving methods."},"venueid":{"value":"ICML.cc/2026/Conference"},"authorids":{"value":["~Haocheng_Xi1","~Shuo_Yang22","~Yilong_Zhao1","~Muyang_Li2","~Han_Cai1","~Xingyang_Li1","~Yujun_Lin1","~Zhuoyang_Zhang1","~Jintao_Zhang2","~Xiuyu_Li1","~Zhiying_Xu2","~Jun_Wu5","~Chenfeng_Xu1","~Ion_Stoica1","~Song_Han5","~Kurt_Keutzer1"]},"authors":{"value":["Haocheng Xi","Shuo Yang","Yilong Zhao","Muyang Li","Han Cai","Xingyang Li","Yujun Lin","Zhuoyang Zhang","Jintao Zhang","Xiuyu Li","Zhiying Xu","Jun Wu","Chenfeng Xu","Ion Stoica","Song Han","Kurt Keutzer"]}},"tmdate":1790067351312,"pdate":1777576423422,"tcdate":1769045520764,"writers":["ICML.cc/2026/Conference","ICML.cc/2026/Conference/Submission13187/Authors"],"signatures":["ICML.cc/2026/Conference/Submission13187/Authors"],"forum":"r9UJgVH1Fx","license":"CC BY 4.0","number":13187,"cdate":1769045520764,"readers":["everyone"],"invitations":["ICML.cc/2026/Conference/-/Submission","ICML.cc/2026/Conference/-/Post_Submission","ICML.cc/2026/Conference/Submission13187/-/Full_Submission","ICML.cc/2026/Conference/-/Edit","ICML.cc/2026/Conference/Submission13187/-/Camera_Ready_Revision"],"mdate":1790067351312,"odate":1782341948895,"domain":"ICML.cc/2026/Conference","id":"r9UJgVH1Fx","version":2},{"content":{"comment":{"value":"Hi, authors. While reading this paper, I found you should consider referencing the paper \"Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs\" (ICLR 2024, Oral), which presents a related approach to KV cache compression based on attention head-level granularity for large language models (LLMs). These two papers share similar motivations, making it relevant to highlight this prior work.\n\nEngineering-Related Question:\n\nIf each attention head retains a different number of tokens, how is attention computed? Typically, attention mechanisms assume that each head operates over the same number of tokens. I would appreciate clarification on how this issue is addressed in the current framework."},"title":{"value":"Missing Reference to Previous KV Cache Compression Work Based on Attention Head Level"}},"tmdate":1733209306016,"tcdate":1733209306016,"writers":["ICLR.cc/2025/Conference","~Xiang_Liu10"],"signatures":["~Xiang_Liu10"],"forum":"FJFVmeXusW","number":4,"license":"CC BY 4.0","cdate":1733209306016,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission8282/-/Public_Comment"],"mdate":1733209306016,"domain":"ICLR.cc/2025/Conference","replyto":"FJFVmeXusW","id":"7qHrqp1PBl","forumContent":{"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Key-Value cache","Contextual reasoning","Efficiency inference","Large-Lauguage Model"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Key-Value (KV) caching is a common technique to enhance the computational efficiency of Large Language Models (LLMs), but its memory overhead grows rapidly with input length. Prior work has shown that not all tokens are equally important for text generation, proposing layer-level KV cache compression to selectively retain key information. Recognizing the distinct roles of attention heads in generation, we propose HeadKV, a head-level KV cache compression method, and HeadKV-R2, which leverages a novel contextual reasoning ability estimation for compression. Our approach operates at the level of individual heads, estimating their importance for contextual QA tasks that require both retrieval and reasoning capabilities. Extensive experiments across diverse benchmarks (LongBench, LooGLE), model architectures (e.g., Llama-3-8B-Instruct, Mistral-7B-Instruct), and long-context abilities tests demonstrate that our head-level KV cache compression significantly outperforms strong baselines, particularly in low-resource settings (KV size = 64 & 128). Notably, our method retains just 1.5% of the KV cache while achieving 97% of the performance of the full KV cache on the contextual question answering benchmark."},"_bibtex":{"value":"@inproceedings{\nfu2025not,\ntitle={Not All Heads Matter: A Head-Level {KV} Cache Compression Method with Integrated Retrieval and Reasoning},\nauthor={Yu Fu and Zefan Cai and Abedelkadir Asi and Wayne Xiong and Yue Dong and Wen Xiao},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=FJFVmeXusW}\n}"},"title":{"value":"Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning"},"pdf":{"value":"/pdf/8a8cb82656124e983fa17aa3ca0a724128cc16c7.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"fu|not_all_heads_matter_a_headlevel_kv_cache_compression_method_with_integrated_retrieval_and_reasoning"},"authorids":{"value":["~Yu_Fu2","~Zefan_Cai1","~Abedelkadir_Asi1","~Wayne_Xiong1","~Yue_Dong2","~Wen_Xiao2"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Yu Fu","Zefan Cai","Abedelkadir Asi","Wayne Xiong","Yue Dong","Wen Xiao"]}},"version":2},{"content":{"research_area_keywords":{"value":"LLM Efficiency,pruning,code generation and understanding"},"keywords":{"value":["LLM Efficiency","pruning","code generation and understanding"]},"B_use_or_create_scientific_artifacts":{"value":"Yes"},"languages_studied":{"value":"English"},"D4_ethics_review_board_approval":{"value":"N/A"},"C4_parameters_for_packages":{"value":"N/A"},"B6_elaboration":{"value":"Section 5"},"B1_cite_creators_of_artifacts":{"value":"Yes"},"D_human_subjects_including_annotators":{"value":"No"},"B2_discuss_the_license_for_artifacts":{"value":"N/A"},"A2_elaboration":{"value":"Section 7"},"C2_elaboration":{"value":"Section 5"},"E_ai_assistants_in_research_or_writing":{"value":"No"},"C1_elaboration":{"value":"Section 5"},"C2_experimental_setup_and_hyperparameters":{"value":"Yes"},"B1_elaboration":{"value":"Section 2"},"C1_model_size_and_budget":{"value":"Yes"},"venue":{"value":"ACL ARR 2026 May Submission"},"D3_data_consent":{"value":"N/A"},"_bibtex":{"value":"@inproceedings{\nanonymous2026tablecache,\ntitle={TableCache: Primary Foreign Key Guided {KV} Cache Precomputation for Low Latency Text-to-{SQL}},\nauthor={Anonymous},\nbooktitle={Submitted to ACL Rolling Review - May 2026},\nyear={2026},\nurl={https://openreview.net/forum?id=B6JRSLM4W9},\nnote={under review}\n}"},"title":{"value":"TableCache: Primary Foreign Key Guided KV Cache Precomputation for Low Latency Text-to-SQL"},"C3_descriptive_statistics":{"value":"N/A"},"contribution_types":{"value":["NLP engineering experiment","Approaches to low-resource settings"]},"A1_limitations_section":{"value":"This paper has a limitations section."},"B6_statistics_for_data":{"value":"Yes"},"B4_data_contains_personally_identifying_info_or_offensive_content":{"value":"N/A"},"B3_artifact_use_consistent_with_intended_use":{"value":"N/A"},"A2_potential_risks":{"value":"Yes"},"E1_information_about_use_of_ai_assistants":{"value":"N/A"},"abstract":{"value":"In Text-to-SQL tasks, existing LLM-based methods often include extensive database schemas in prompts, leading to long context lengths and increased prefilling latency. While user queries typically focus on recurrent table sets—offering an opportunity for KV cache sharing across queries—current inference engines, such as SGLang and vLLM, generate redundant prefix cache copies when processing user queries with varying table orders. To address this inefficiency, we propose precomputing table representations as KV caches offline and querying the required ones online. A key aspect of our approach is the computation of table caches while preserving primary foreign key relationships between tables. Additionally, we construct a Table Trie structure to facilitate efficient KV cache lookups during inference. To enhance cache performance, we introduce a cache management module with a query-reranking strategy to improve cache hit rates and a computation-loading pipeline for parallelizing model inference and cache loading. Experimental results show that our proposed TableCache achieves up to a 3.62× speedup in Time to First Token (TTFT) with negligible performance degradation."},"D1_instructions_given_to_participants":{"value":"N/A"},"paper_type":{"value":"Long"},"C_computational_experiments":{"value":"Yes"},"pdf":{"value":"/pdf/e929e75ab0d4eb670642b9096ba4b5cf6f53e1b3.pdf"},"research_area":{"value":"NLP Applications"},"EMNLP_2026_AI_Reviewing_Experiment":{"value":"no"},"B5_documentation_of_artifacts":{"value":"N/A"},"D2_recruitment_and_payment":{"value":"N/A"},"venueid":{"value":"aclweb.org/ACL/ARR/2026/May/Submission"}},"tmdate":1790696909166,"tcdate":1779194154118,"writers":["aclweb.org/ACL/ARR/2026/May","aclweb.org/ACL/ARR/2026/May/Submission1931/Authors"],"signatures":["aclweb.org/ACL/ARR/2026/May/Submission1931/Authors"],"forum":"B6JRSLM4W9","license":"CC BY 4.0","number":1931,"cdate":1779194154118,"readers":["everyone"],"invitations":["aclweb.org/ACL/ARR/2026/May/-/Submission","aclweb.org/ACL/ARR/2026/May/-/Edit","aclweb.org/ACL/ARR/2026/May/-/Post_Submission","aclweb.org/ACL/ARR/2026/May/-/Preprint_Post_Submission"],"mdate":1790696909166,"odate":1780385598983,"domain":"aclweb.org/ACL/ARR/2026/May","id":"B6JRSLM4W9","version":2},{"content":{"research_area_keywords":{"value":"LLM Efficiency,pruning,code generation and understanding"},"languages_studied":{"value":"English"},"venue":{"value":"ACL ARR 2025 October Submission"},"_bibtex":{"value":"@inproceedings{\nanonymous2026tablecache,\ntitle={TableCache: Primary Foreign Key Guided {KV} Cache Precomputation for Low Latency Text-to-{SQL}},\nauthor={Anonymous},\nbooktitle={Submitted to ACL Rolling Review - October 2025},\nyear={2026},\nurl={https://openreview.net/forum?id=DzcKGz4NTq},\nnote={under review}\n}"},"title":{"value":"TableCache: Primary Foreign Key Guided KV Cache Precomputation for Low Latency Text-to-SQL"},"contribution_types":{"value":["NLP engineering experiment","Approaches low compute settings-efficiency"]},"abstract":{"value":"In Text-to-SQL tasks, existing LLM-based methods often include extensive database schemas in prompts, leading to long context lengths and increased prefilling latency. While user queries typically focus on recurrent table sets—offering an opportunity for KV cache sharing across queries—current inference engines, such as SGLang and vLLM, generate redundant prefix cache copies when processing user queries with varying table orders.\nTo address this inefficiency, we propose precomputing table representations as KV caches offline and querying the required ones online. A key aspect of our approach is the computation of table caches while preserving primary foreign key relationships between tables. Additionally, we construct a Table Trie structure to facilitate efficient KV cache lookups during inference.\nTo enhance cache performance, we introduce a cache management module with a query-reranking strategy to improve cache hit rates and a computation-loading pipeline for parallelizing model inference and cache loading.\nExperimental results show that our proposed TableCache achieves up to a 3.62× speedup in Time to First Token (TTFT) with negligible performance degradation."},"paper_type":{"value":"Long"},"pdf":{"value":"/pdf/ec3c8d30f39eb0aa910da1cf161002a6b3117fb6.pdf"},"research_area":{"value":"Efficient/Low-Resource Methods for NLP"},"venueid":{"value":"aclweb.org/ACL/ARR/2025/October/Submission"}},"tmdate":1782961432975,"tcdate":1759754058041,"writers":["aclweb.org/ACL/ARR/2025/October","aclweb.org/ACL/ARR/2025/October/Submission1403/Authors"],"signatures":["aclweb.org/ACL/ARR/2025/October/Submission1403/Authors"],"forum":"DzcKGz4NTq","license":"CC BY 4.0","number":1403,"cdate":1759754058041,"readers":["everyone"],"invitations":["aclweb.org/ACL/ARR/2025/October/-/Submission","aclweb.org/ACL/ARR/2025/October/-/Edit","aclweb.org/ACL/ARR/2025/October/-/Post_Submission","aclweb.org/ACL/ARR/2025/October/-/Preprint_Post_Submission"],"mdate":1782961432975,"odate":1782961377851,"domain":"aclweb.org/ACL/ARR/2025/October","id":"DzcKGz4NTq","version":2},{"content":{"venue":{"value":"CoRR 2026"},"pdf":{"value":"https://arxiv.org/pdf/2601.08743v1"},"venueid":{"value":"dblp.org/journals/CORR/2026"},"paperhash":{"value":"su|tablecache_primary_foreign_key_guided_kv_cache_precomputation_for_low_latency_texttosql"},"authorids":{"value":["~Jinbo_Su1","","","","","",""]},"html":{"value":"https://doi.org/10.48550/arXiv.2601.08743"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2601-08743,\n publtype={informal},\n author={Jinbo Su and Yuxuan Hu and Cuiping Li and Hong Chen and Jia Li and Lintao Ma and Jing Zhang},\n title={TableCache: Primary Foreign Key Guided KV Cache Precomputation for Low Latency Text-to-SQL},\n year={2026},\n month={January},\n cdate={1767225600000},\n journal={CoRR},\n volume={abs/2601.08743},\n url={https://doi.org/10.48550/arXiv.2601.08743}\n}\n"},"abstract":{"value":"In Text-to-SQL tasks, existing LLM-based methods often include extensive database schemas in prompts, leading to long context lengths and increased prefilling latency. While user queries typically focus on recurrent table sets-offering an opportunity for KV cache sharing across queries-current inference engines, such as SGLang and vLLM, generate redundant prefix cache copies when processing user queries with varying table orders. To address this inefficiency, we propose precomputing table representations as KV caches offline and querying the required ones online. A key aspect of our approach is the computation of table caches while preserving primary foreign key relationships between tables. Additionally, we construct a Table Trie structure to facilitate efficient KV cache lookups during inference. To enhance cache performance, we introduce a cache management system with a query reranking strategy to improve cache hit rates and a computation loading pipeline for parallelizing model inference and cache loading. Experimental results show that our proposed TableCache achieves up to a 3.62x speedup in Time to First Token (TTFT) with negligible performance degradation."},"title":{"value":"TableCache: Primary Foreign Key Guided KV Cache Precomputation for Low Latency Text-to-SQL"},"authors":{"value":["Jinbo Su","Yuxuan Hu","Cuiping Li","Hong Chen","Jia Li","Lintao Ma","Jing Zhang"]}},"tmdate":1773384222800,"pdate":1798675200000,"externalIds":["dblp:journals/corr/abs-2601-08743"],"tcdate":1773384218269,"writers":["~"],"signatures":["~Jinbo_Su1"],"forum":"JKZb39YTwM","license":"CC BY-SA 4.0","number":851473,"cdate":1767225600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1773384222800,"domain":"DBLP.org","id":"JKZb39YTwM","version":2},{"content":{"venue":{"value":"CVPR 2026"},"abstract":{"value":"Long streaming video QA remains challenging due to growing visual tokens and limited reasoning length of large language models (LLMs). KV-caching stores the Key-Value (KV) of the historical tokens via LLM prefill and enables more efficient streaming QA. However, existing methods cache every one or two frames, causing redundant memory usage and losing fine-grained spatial details within frame or temporal contexts across frames. This paper proposes MuKV, a method that features a multi-grained KV cache compression module and a semi-hierarchical retrieval approach to improve both efficiency and accuracy for long streaming VideoQA. For the offline KV cache, MuKV extracts visual representations at patch-, frame-, and segment-levels. The multiple levels of granularity preserve both local cues and global temporal context, while maintaining efficiency with a dual signal token compression mechanism guided by self-attention and frequency. For online QA, MuKV designs a semi-hierarchical retrieval method to retrieve relevant KV caches for answer generation. Experiments on long-streaming VideoQA benchmarks show that MuKV significantly improves answer accuracy, without sacrificing memory and online QA efficiency. Moreover, our compression mechanism alone brings consistent benefits across answer accuracy, memory, and QA efficiency over baselines, showcasing highly effective contribution."},"_bibtex":{"value":"@inproceedings{\nxiao2026mukv,\ntitle={Mu{KV}: Multi-Grained {KV} Cache Compression for Long Streaming Video Question-Answering},\nauthor={Junbin Xiao and Jiajun Chen and Tianxiang Sun and Xun Yang and Angela Yao},\nbooktitle={Conference on Computer Vision and Pattern Recognition 2026},\nyear={2026},\nurl={https://openreview.net/forum?id=IBNXXDFCrH}\n}"},"title":{"value":"MuKV: Multi-Grained KV Cache Compression for Long Streaming Video Question-Answering"},"pdf":{"value":"https://openaccess.thecvf.com/content/CVPR2026/papers/Xiao_MuKV_Multi-Grained_KV_Cache_Compression_for_Long_Streaming_Video_Question-Answering_CVPR_2026_paper.pdf"},"venueid":{"value":"thecvf.com/CVPR/2026/Conference"},"paperhash":{"value":"xiao|mukv_multigrained_kv_cache_compression_for_long_streaming_video_questionanswering"},"authorids":{"value":["~Junbin_Xiao1","~Jiajun_Chen10","~Tianxiang_Sun4","~Xun_Yang1","~Angela_Yao1"]},"authors":{"value":["Junbin Xiao","Jiajun Chen","Tianxiang Sun","Xun Yang","Angela Yao"]}},"tmdate":1789656523258,"pdate":1789656437487,"tcdate":1765223080231,"writers":["thecvf.com/CVPR/2026/Conference","thecvf.com/CVPR/2026/Conference/Submission42939/Authors"],"signatures":["thecvf.com/CVPR/2026/Conference/Submission42939/Authors"],"forum":"IBNXXDFCrH","license":"CC BY 4.0","number":42939,"cdate":1765223080231,"readers":["everyone"],"invitations":["thecvf.com/CVPR/2026/Conference/-/Submission","thecvf.com/CVPR/2026/Conference/Submission42939/-/Full_Submission","thecvf.com/CVPR/2026/Conference/-/Post_Submission","thecvf.com/CVPR/2026/Conference/Submission42939/-/Supplementary_Material","thecvf.com/CVPR/2026/Conference/-/Edit","thecvf.com/CVPR/2026/Conference/-/Compute_Flag"],"mdate":1789656523258,"odate":1789656437487,"domain":"thecvf.com/CVPR/2026/Conference","id":"IBNXXDFCrH","version":2},{"content":{"venue":{"value":"CoRR 2026"},"pdf":{"value":"https://arxiv.org/pdf/2605.22269v1"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"xiao|mukv_multigrained_kv_cache_compression_for_long_streaming_video_questionanswering"},"html":{"value":"https://doi.org/10.48550/arXiv.2605.22269"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2605-22269,\n publtype={informal},\n author={Junbin Xiao and Jiajun Chen and Tianxiang Sun and Xun Yang and Angela Yao},\n title={MuKV: Multi-Grained KV Cache Compression for Long Streaming Video Question-Answering},\n year={2026},\n month={May},\n cdate={1777593600000},\n journal={CoRR},\n volume={abs/2605.22269},\n url={https://doi.org/10.48550/arXiv.2605.22269}\n}\n"},"abstract":{"value":"Long streaming video QA remains challenging due to growing visual tokens and limited reasoning length of large language models (LLMs). KV-caching stores the Key-Value (KV) of the historical tokens via LLM prefill and enables more efficient streaming QA. However, existing methods cache every one or two frames, causing redundant memory usage and losing fine-grained spatial details within frame or temporal contexts across frames. This paper proposes MuKV, a method that features a multi-grained KV cache compression module and a semi-hierarchical retrieval approach to improve both efficiency and accuracy for long streaming VideoQA. For the offline KV cache, MuKV extracts visual representations at patch-, frame-, and segment-levels. The multiple levels of granularity preserve both local cues and global temporal context, while maintaining efficiency with a dual signal token compression mechanism guided by self-attention and frequency. For online QA, MuKV designs a semi-hierarchical retrieval method to retrieve relevant KV caches for answer generation. Experiments on long-streaming VideoQA benchmarks show that MuKV significantly improves answer accuracy, without sacrificing memory and online QA efficiency. Moreover, our compression mechanism alone brings consistent benefits across answer accuracy, memory, and QA efficiency over baselines, showcasing highly effective contribution."},"title":{"value":"MuKV: Multi-Grained KV Cache Compression for Long Streaming Video Question-Answering"},"authors":{"value":[{"fullname":"Junbin Xiao","username":"~Junbin_Xiao1"},{"fullname":"Jiajun Chen","username":""},{"fullname":"Tianxiang Sun","username":""},{"fullname":"Xun Yang","username":""},{"fullname":"Angela Yao","username":""}]}},"tmdate":1787736524644,"pdate":1798675200000,"externalIds":["dblp:journals/corr/abs-2605-22269"],"tcdate":1787736516890,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Junbin_Xiao1"],"forum":"V1WnvgeCiD","license":"CC BY-SA 4.0","number":142862,"cdate":1777593600000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1787736524644,"domain":"OpenReview.net/Public_Article","id":"V1WnvgeCiD","version":2},{"content":{"data_release":{"value":"We authorize the release of our submission and author names to the public in the event of acceptance."},"TLDR":{"value":"Sigmoid attention is a better substrate for learned gates that transfer cleanly from soft training-time attenuation to hard physical KV-cache deletion."},"venue":{"value":"AdaptFM Poster"},"email_sharing":{"value":"We authorize the sharing of all author emails with Program Chairs."},"keywords":{"value":["KV cache eviction","learned cache compression","sigmoid attention","attention normalization","Transformer inference","large language models"]},"abstract":{"value":"Learned KV-cache eviction often faces a soft-to-hard mismatch: during training, differentiable gates typically attenuate token contributions, whereas inference saves memory only when KV entries are physically removed. We ask whether the attention substrate affects this\nsoft-to-hard transition. Using GPT-2-scale Transformers trained on\nOpenWebText, we run a controlled $2\\times2\\times2$ comparison over\nattention type, learned gating, and positional encoding. Although\nsigmoid attention is worse as a dense language model, learned hard\neviction changes the useful operating points: sigmoid-gated models\ndelete KV entries with negligible PPL change relative to their own\nno-eviction references. Under a matched live-cache protocol on the same dense backbones,\nlearned sigmoid gates obtain lower PPL than our H$_2$O and KeyDiff\nimplementations, whereas softmax gates do not uniformly beat these\npost-hoc methods.\nThe results suggest that attention normalization can substantially affect\nwhether a training-time soft gate transfers cleanly to hard KV deletion."},"_bibtex":{"value":"@inproceedings{\nli2026sigmoid,\ntitle={Sigmoid Attention as a Better Substrate for Learned {KV} Cache Eviction},\nauthor={Isaac Li},\nbooktitle={AdaptFM: Resource-Adaptive Foundation Model Inference},\nyear={2026},\nurl={https://openreview.net/forum?id=NiWI3Hrnd5}\n}"},"title":{"value":"Sigmoid Attention as a Better Substrate for Learned KV Cache Eviction"},"pdf":{"value":"/pdf/b5524f508e242267184c4eafa28067f3acbe6c2c.pdf"},"venueid":{"value":"ICML.cc/2026/Workshop/AdaptFM"},"paperhash":{"value":"li|sigmoid_attention_as_a_better_substrate_for_learned_kv_cache_eviction"},"authorids":{"value":["~Isaac_Li2"]},"authors":{"value":["Isaac Li"]}},"tmdate":1780907945782,"pdate":1780353776095,"tcdate":1778293010093,"writers":["ICML.cc/2026/Workshop/AdaptFM","ICML.cc/2026/Workshop/AdaptFM/Submission168/Authors"],"signatures":["ICML.cc/2026/Workshop/AdaptFM/Submission168/Authors"],"forum":"NiWI3Hrnd5","license":"CC BY 4.0","number":168,"cdate":1778293010093,"readers":["everyone"],"invitations":["ICML.cc/2026/Workshop/AdaptFM/-/Submission","ICML.cc/2026/Workshop/AdaptFM/-/Submission_Change_Before_Reviewing","ICML.cc/2026/Workshop/AdaptFM/-/Submission_Release","ICML.cc/2026/Workshop/AdaptFM/Submission168/-/Camera_Ready_Revision","ICML.cc/2026/Workshop/AdaptFM/-/Edit"],"mdate":1780907945782,"odate":1780353776095,"domain":"ICML.cc/2026/Workshop/AdaptFM","id":"NiWI3Hrnd5","version":2},{"content":{"TLDR":{"value":"RESCUE improves KV cache eviction by learning policy-specific corrections for future-critical entries missed by a base policy and applying them only when a one-token fidelity probe verifies the correction."},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["KV cache eviction","long-context inference","large language models","attention","efficient inference"]},"primary_area":{"value":"infrastructure, software libraries, hardware, systems, etc."},"abstract":{"value":"KV cache eviction is essential for efficient long-context LLM inference. Existing policies score KV entries from evidence already visible at the eviction point, and share a blind spot: at an aggressive budget each misses most of what the continuation will attend to, and policies reading the same observation window fail on overlapping entries rather than independent ones. We propose RESCUE, which treats future-aware eviction as correcting a deployed policy rather than replacing it and then verifying the correction against the dense model before it is applied. A policy-conditioned residual scorer is supervised only on the entries its base policy missed, so it never relearns what that policy already ranks correctly; a verification step then decodes a probe from the uncompressed cache immediately after prefill and adopts the corrected cache only if it keeps the output distribution closer to the dense model's. Verification is the part that carries the result: applied unconditionally the correction is harmful on almost a quarter of the evaluated cells. Across all 80 cells of 16 LongBench tasks × five base policies on Llama-3.1-8B, RESCUE improves the average score of every policy it augments, by +1.28 points (95% CI [+0.51, +2.16]), and cuts the cells it harms from 18 to 3; the average stays positive on Mistral-7B (+1.08) and Qwen3-8B (+0.43), with the gain concentrated on the weakest base policy and falling towards zero for the strongest as the budget widens. RESCUE targets evict-once-after-prefill inference and buys that safety with one dense decoding step."},"_bibtex":{"value":"@inproceedings{\nanonymous2026rescue,\ntitle={{RESCUE}: Gated Correction for {KV} Cache Eviction},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=ngxyKpb52A},\nnote={under review}\n}"},"title":{"value":"RESCUE: Gated Correction for KV Cache Eviction"},"pdf":{"value":"/pdf/fd1ec54c2031f2e7ac327e0f2a98ec4437267f0a.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791233545544,"tcdate":1789743868512,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission44600/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission44600/Authors"],"forum":"ngxyKpb52A","license":"CC BY 4.0","number":44600,"cdate":1789743868512,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission44600/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791233545544,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"ngxyKpb52A","version":2},{"content":{"summary":{"value":"This paper presents IntelLLM, a novel KV cache eviction algorithm designed to alleviate the storage and computational burden of Transformer-based large language model inference. IntelLLM leverages the sparsity of attention mechanisms and strategically evicts certain tokens to improve inference efficiency. The authors propose two key strategies: Center of Gravity Eviction (CGE) and Remote Gap Localization (LGL). CGE addresses the semantic loss caused by dominant attention scores in softmax. LGL reorganizes token positions by creating a large gap between global and local tokens, further enhancing processing efficiency. IntelLLM is evaluated on LongBench with two models, Llama-3-8B and Mixtral-7B-instruct-v0.2, and outperformed all the baselines, including Full KV cache, StreamingLLM, and LM-Infinite. The algorithm achieved 2x KV cache compression ratio for Llama and an 8x compression ratio for Mixtral, maintaining strong inference efficiency despite the reduced cache size. Additionally, the paper includes ablation studies focusing on Head Gravity and the RGL Gap, demonstrating the soundness and effectiveness of IntelLLM's design."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"1. Suggestion on improving the overall presentation. First, there are some notation used without definition, e.g. $n, m, l_{head}, l_{tail}$ in Algorithm 1, and the operation $[:]$ (which dimension is applied to?) used in Algorithm 1. Second, terms such as \"Center of Gravity\" should be defined formally in the KV cache eviction topic. Third, there remains some minor grammar and expression mistakes in this paper, such as the misuse of citation format. Forth, the presentation of the tables and figures should be improved. The average score should be reported in Table 1 & 2. The meaning of the x-axis and y-axis of Figure 1 & 2 should be clarified.\n\n2. The description of Algorithm 1 is confusing. Could you provide a very detailed explanation of the calculation process in natural language? \n\n3. There isn't any model named \"Llama3-7B\". Do you mean Llama-3-8B?\n\n4. What are the detailed settings of the baselines, e.g. how much KV cache is retained? Could you provide more information on the hyperparameters used on baselines?\n\n5. The experimental results mentioned in line 316-319 should be included in this paper if such conclusion is drawn. If there is no space, it should be placed in the Appendix. The experiments in line 359-361 should also be included.\n\nOverall, I really like the idea proposed by this paper, which I personally find very inspiring. I will raise my rating on soundness and overall rating if good discussion is made with necessary experimental results during the rebuttal phase."},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"1. This paper provides a clear explanation of why the attention mechanism tends to allocate excessive weights to a few tokens with high attention scores while neglecting others. The analysis of the softmax function is insightful and well-reasoned.\n\n2. The proposed method, IntelLLM, demonstrates strong performance in benchmarks and shows compatibility across multiple models, including widely-used ones like Llama and Mixtral.\n\n3. The method is simple and easy to implement. IntelLLM can be integrated into various scenarios and inference frameworks."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The presentation is to some extent unclear and confusing. Some notations and terminologies are used without a clear definition. The description of Algorithm 1 is confusing and hard to understand. Tables and figures and be organized in a better format. Details of presentation issues are described in question Q1.\n\n2. Insufficient baseline selection. The proposed algorithm of this paper is an eviction algorithm that compresses KV cache, thus there should be comparison between other popular eviction methods, including H2O [1], SnapKV [2], SirLLM [3] and IntervalLLM (a baseline created in SirLLM). It would be better to compare IntelLLM with other end-to-end methods, including InfLLM [4] and MInference [5], as they utilize attention sparsity. (This is only a suggestion and the authors are not required to test all the methods above during the review process, but futher discussion is expected.)\n\n3. Insufficient benchmark datasets. Although LongBench is a classic benchmark for long context inference, the average length is relatively short and the hardness is limited. It is expected to benchmark IntelLLM on harder benchmark, e.g. L-Eval [6], and longer benchmark, e.g. RULER [7] or InfiniteBench [8]. It would also be better to test IntelLLM on accurate context retrieval tasks, e.g. Ret.PassKey in InfiniteBench and RULER. (This is also a suggestion, and the authors are not required to test all of the benchmarks mentioned above. However, some simple supplementary benchmarks are welcomed.)\n\n4. Lack of ablation studies. Ablations on $L_{comp}$ and $L_{near}$ are expected, as they can be used to prove which part of the retained KV cache is more important and to what extent can the hyperparameters affect the overall performance. Also, there should be comparison between RGL and other methods of assigning position information, such as assigning continuous position ids and not assigning any position information for distant tokens.\n\n5. Unclear presentation of research intention. KV cache compression is a technique developed to enhance model generation speed or reduce memory consumption. The paper should clearly state out which purposed is focused mainly, and conduct the corresponding experiments. For example, the model generation speed should be tested and it is expected to be faster than full KV cache inference. The peak memory is also expected to be lower if the system implementation is delicate enough. The speed reported in line 445-446 cannot state the research intention as the pre-fill speed is slightly lower. Reporting a faster generation speed (the speed only tested on the decoding stage and excluding the pre-fill stage) might be helpful.\n\n6. No limitation and future work discussion. This paper should include some vital limitations and potential future work of the proposed methods.\n\n\n[1] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models\n\n[2] SnapKV: LLM knows what you are looking for before generation\n\n[3] SirLLM: Streaming Infinite Retentive LLM\n\n[4] InfLLM: Unveiling the intrinsic capacity of LLMs for understanding extremely long sequences with training-free memory\n\n[5] Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention\n\n[6] L-eval: Instituting standardized evaluation for long context language models\n\n[7] RULER: What’s the Real Context Size of Your Long-Context Language Models?\n\n[8] ∞ Bench: Extending Long Context Evaluation Beyond 100K Tokens"}},"nonreaders":[],"tmdate":1731429311810,"tcdate":1729074040738,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission9602/Reviewer_4hs9"],"signatures":["ICLR.cc/2025/Conference/Submission9602/Reviewer_4hs9"],"forum":"4QWPCTLq20","number":1,"license":"CC BY 4.0","cdate":1729074040738,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission9602/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731429311810,"domain":"ICLR.cc/2025/Conference","replyto":"4QWPCTLq20","id":"Vxv6XKHR8E","forumContent":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["LLM","KV cache compression","CGE","RGL"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Large Language Models (LLMs) have demonstrated exceptional capabilities in integrating contextual knowledge, but their deployment is often constrained by the substantial computational resources required for long text sequences. To mitigate the inference time cost associated with attention mechanisms, LLMs utilize key-value embedding caching techniques (KV cache), which introduce significant storage pressure. In this paper, we propose IntelLLM, a novel and efficient approach to KV cache compression that strikes a balance between compression rate and performance. Drawing inspiration from sparse attention mechanism, we observe that only a small subset of tokens in lengthy texts capture the majority of attention weights. This sparsity, intrinsic to the attention mechanism, serves as the foundation for improving the KV compression ratio through a strategic eviction method. IntelLLM is composed of center of gravity eviction (CGE) strategy and remote gap localization (RGL) strategy. CGE is designed to address the potential loss of important semantic dependencies when evicting high-sparsity tokens, which prioritizes the retention of key tokens by shielding the center of gravity of attention during inference, thereby preserving critical information and optimizing the efficiency of attention computation. Additionally, RGL is proposed to leverage implicit positional features to maintain long-range dependencies, inspired by advancements in location encoding research. Our KV compression approach integrates seamlessly with existing LLMs, requiring minimal code modifications without the need for fine-tuning or model parameter changes. IntelLLM not only significantly reduces the storage requirements for KV cache but also consistently outperforms full KV models in long text processing tasks, while utilizing only 50% of the typical KV cache expenses."},"_bibtex":{"value":"@misc{\nli2024intelllm,\ntitle={Intel{LLM}: Little Hints Make a Big Difference for {LLM} {KV} Cache Compression},\nauthor={TingLong Li and Qiuyu Shao},\nyear={2024},\nurl={https://openreview.net/forum?id=4QWPCTLq20}\n}"},"title":{"value":"IntelLLM: Little Hints Make a Big Difference for LLM KV Cache Compression"},"pdf":{"value":"/pdf/e70bdc24592a789f5a144d6ea6d61377169446db.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"li|intelllm_little_hints_make_a_big_difference_for_llm_kv_cache_compression"},"authorids":{"value":["~TingLong_Li1","~Qiuyu_Shao1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["TingLong Li","Qiuyu Shao"]}},"version":2},{"content":{"comment":{"value":"We thank the reviewer for this important challenge. We address both points and the requested clarification below.\n\n**1. The Logical Paradox in Prefix Sharing**\n\nWe provide the requested end-to-end exploit chain under standard API constraints:\n\n1. Attacker crafts a prompt that collides with the victim's prefix hash\n2. Attacker's KV entries get cached under the colliding hash\n3. Victim's request hits the cache and retrieves attacker's entries\n4. Without GeoCache: victim's generation is corrupted (semantic poisoning)\n5. With GeoCache: attacker's entries are in a different coordinate system — victim's queries produce noise — attack fails\n\nThis is an API-level attack requiring no hardware access and no privilege escalation. **CVE-2025-25183** (vLLM, fixed in v0.7.2) and **CVE-2025-1953** (vLLM AIBrix) demonstrated exactly this — hash collisions in prefix caching allowed an attacker to serve poisoned KV entries to a victim through a different prompt. \"Cache Me, Catch You\" (Wu et al., NDSS 2026) documented additional vulnerabilities across six attack vectors in vLLM, SGLang, and GPTCache, showing that cache collision attacks lead to incorrect model outputs, information leakage, and content filter bypasses.\n\nWhile these have been patched (vLLM v0.11 now defaults to SHA-256), three observations: (1) This vulnerability class recurred independently — CVE-2025-25183 (vLLM), CVE-2025-1953 (AIBrix), and additional vulnerabilities across six attack vectors (Wu et al., NDSS 2026). A single patch does not eliminate a systemic pattern. (2) vLLM v0.11 offers xxhash as a faster alternative to SHA-256, and their own documentation warns this \"can cause undefined behavior or even leak private information in multi-tenant environments.\" Operators under latency pressure may choose speed over security. (3) Defense-in-depth is the security standard — no single layer is assumed infallible. GeoCache provides the content protection layer independent of hash implementation correctness.\n\n**2. The Missing Exploit Chain — Physical Access Vectors**\n\nBeyond hash collisions, KV tensors also become physically accessible in production infrastructure:\n\n**(a) GPU memory residuals.** LeftoverLocals (Sorensen & Khlaaf, 2024; CVE-2023-4969) demonstrated that GPU local memory is not cleared between workloads, allowing a subsequent tenant to read residual KV tensors.\n\n**(b) Disaggregated serving.** Production architectures (e.g., LMCache, llm-d, AIBrix) store and transmit KV tensors across nodes on remote DRAM/SSD. A compromised storage node or network interceptor can read tensor content.\n\n**(c) Externalized KV-cache outside TEE.** This is the same threat model as KV-Cloak (NDSS 2026) — the KV-cache sits outside the TEE boundary, readable by the CSP.\n\nLogical access controls (cache_salt, SHA-256 hashing) prevent unauthorized cache *hits* but do not protect tensor *content* once physically accessed. GeoCache protects both layers.\n\n**3. Regarding Timing Padding Sufficiency**\n\nTiming padding addresses existence leakage but does not protect tensor content under either vector above. If an attacker poisons the cache via hash collision, timing padding does not help — the victim still retrieves corrupted KV entries. If an attacker reads tensors from GPU memory, timing padding does not help — the raw content is exposed. GeoCache and timing padding address different layers.\n\nWe will rewrite Section 3.1 to explicitly describe both attack vectors (hash collision and physical access) with CVE references."},"title":{"value":"Author Response to Reviewer"}},"parentInvitations":"TMLR/-/Official_Comment","tmdate":1775998510279,"tcdate":1775998510279,"writers":["TMLR","TMLR/Paper8077/Authors"],"signatures":["TMLR/Paper8077/Authors"],"forum":"CNuH7VtNun","number":4,"license":"CC BY 4.0","cdate":1775998510279,"readers":["everyone"],"invitations":["TMLR/Paper8077/-/Official_Comment"],"mdate":1775998510279,"domain":"TMLR","replyto":"1PrdkCMnlM","id":"ww9v15efFC","forumContent":{"submission_length":{"value":"Long submission (more than 12 pages of main content)"},"venue":{"value":"Rejected by TMLR"},"abstract":{"value":"Key-value (KV) cache sharing across user sessions is a critical optimization for multi-tenant large language model (LLM) serving, reducing time-to-first-token (TTFT) latency by up to 73%. However, recent work has demonstrated that shared KV caches create exploitable vulnerabilities: an adversary can exfiltrate data or mount semantic poisoning attacks by accessing cross-tenant cache states. Existing mitigations either disable sharing entirely (negating performance gains) or employ heuristic content classifiers with inherent false-positive/negative tradeoffs. We propose GeoCache, a principled approach that applies session-specific orthogonal transformations to key and value attention matrices before caching. GeoCache exploits the isometry property of orthogonal operators to provide two complementary guarantees: (1) exact preservation of attention computation for authorized in-session cache reuse, proven by the identity q⊤M⊤MK = q⊤K (Theorem 1); and (2) geometric isolation of cross-session cache entries, with attention scores concentrating near zero under a BDOT-specific exponential bound (Theorem 2). The approach uses a Block-Diagonal Orthogonal Transform (BDOT) achieving O(d_k) per-token overhead and sub-microsecond (0.014 μs) latency per head-token on an NVIDIA T4 GPU. Experiments on Llama-2-7B and Mistral-7B confirm: (a) generation quality is preserved exactly (maximum float32 deviation 5.3 × 10⁻⁵); (b) controlled exfiltration and injection attacks via the forward attention path are substantially mitigated (ROUGE-L reduced from 0.917 to 0.043, 0/19 secret markers leaked; cross-session cosine similarity reduced from 1.0 to ≈ 0); and (c) throughput overhead is negligible. We position GeoCache as providing provable utility preservation together with computational content-level isolation (rather than cryptographic security): a separate algebraic recovery attack (Known-Plaintext Attack) on the BDOT operator is feasible if the attacker collects b linearly independent plaintext-ciphertext pairs per block. We therefore require operator rotation every b/2 tokens for production deployments serving structured prompts (JSON schemas, few-shot exemplars), which empirically reduces algebraic recovery to subspace-only directional leakage insufficient for exact token reconstruction; under freeform queries, hybrid deployment alone suffices. For deployments requiring timing-channel protection, GeoCache composes with constant-time response padding to eliminate existence leakage."},"_bibtex":{"value":"@article{\nanonymous2026geocache,\ntitle={GeoCache: Provably Lossless Inference and Computational Content-Level Isolation for Shared {KV}-Caches in Multi-Tenant {LLM} Inference via Isometric Orthogonal Transformation},\nauthor={Anonymous},\njournal={Submitted to Transactions on Machine Learning Research},\nyear={2026},\nurl={https://openreview.net/forum?id=CNuH7VtNun},\nnote={Rejected}\n}"},"title":{"value":"GeoCache: Provably Lossless Inference and Computational Content-Level Isolation for Shared KV-Caches in Multi-Tenant LLM Inference via Isometric Orthogonal Transformation"},"changes_since_last_submission":{"value":"Summary of Revisions (v3)\n\n1. Claims scoped. Title revised to \"Provably Lossless Inference and Computational Content-Level Isolation.\" Abstract and body reframed; \"eliminated\" and \"mathematically infeasible\" replaced with scoped empirical statements.\n\n2. Theorem 2 corrected. BDOT-specific proof added (Appendix A.4) with block-wise concentration bound depending on b, not dk. Variance analysis corrected for best/worst case.\n\n3. Real-template KPA evaluation. Five production templates on Mistral-7B (Table 10, Appendix D). Structured templates reach cosine 0.86-0.90 without rotation; operator rotation reduces to 0.51-0.64. Rotation reframed from \"additional hardening\" to required for production.\n\n4. Threat model clarified. Hash collisions split into poisoning/exfiltration directions. Plaintext K transience stated. New Limitation 6: confidentiality-to-availability degradation. KPA requires conjunction of ciphertext access AND plaintext knowledge.\n\n5. Public/private baseline added. New Table 8 row and Discussion paragraph. GeoCache positioned as defense-in-depth; simple partitioning acknowledged as sufficient when hash isolation is trusted and physical access excluded.\n\n6. KV-Cloak corrected. Removed inaccurate ~5% overhead claim. Differentiators reduced to three. \"Concurrent\" changed to \"recent work.\"\n\n7. Math/editorial fixes. Latency op count corrected (3→4). Per-head vs per-layer semantics reconciled. Theorem 1 proof clarified. GQA claim removed. Cryptographic Assumption justified via PRG chain.\n\n8. Artifacts. Two new KPA scripts added. README updated.\n\nCore methodology unchanged. Theorem 1 preserved; Theorem 2 proof corrected for BDOT."},"pdf":{"value":"/pdf/6989390b7dd1910414bb94c4012a07f477fe130d.pdf"},"venueid":{"value":"TMLR/Rejected"},"previous_TMLR_submission_url":{"value":"https://openreview.net/forum?id=oX8YpiElYp"},"assigned_action_editor":{"value":"~Yiming_Li1"}},"version":2},{"content":{"venue":{"value":"ACL (1) 2024"},"pdf":{"value":"https://aclanthology.org/2024.acl-long.602.pdf"},"venueid":{"value":"dblp.org/conf/ACL/2024"},"paperhash":{"value":"wu|layercondensed_kv_cache_for_efficient_inference_of_large_language_models"},"authorids":{"value":["~Haoyi_Wu1","https://dblp.org/search/pid/api?q=author:Kewei_Tu:"]},"html":{"value":"https://doi.org/10.18653/v1/2024.acl-long.602"},"_bibtex":{"value":"@inproceedings{DBLP:conf/acl/WuT24,\n author={Haoyi Wu and Kewei Tu},\n title={Layer-Condensed KV Cache for Efficient Inference of Large Language Models},\n year={2024},\n cdate={1704067200000},\n pages={11175-11188},\n url={https://doi.org/10.18653/v1/2024.acl-long.602},\n booktitle={ACL (1)},\n crossref={conf/acl/2024-1}\n}\n"},"abstract":{"value":"Huge memory consumption has been a major bottleneck for deploying high-throughput large language models in real-world applications. In addition to the large number of parameters, the key-value (KV) cache for the attention mechanism in the transformer architecture consumes a significant amount of memory, especially when the number of layers is large for deep language models. In this paper, we propose a novel method that only computes and caches the KVs of a small number of layers, thus significantly saving memory consumption and improving inference throughput. Our experiments on large language models show that our method achieves up to 26× higher throughput than standard transformers and competitive performance in language modeling and downstream tasks. In addition, our method is orthogonal to existing transformer memory-saving techniques, so it is straightforward to integrate them with our model, achieving further improvement in inference efficiency. Our code is available at https://github.com/whyNLP/LCKV."},"title":{"value":"Layer-Condensed KV Cache for Efficient Inference of Large Language Models"},"authors":{"value":["Haoyi Wu","Kewei Tu"]}},"tmdate":1729077558079,"pdate":1704067200000,"tcdate":1729077556301,"writers":["~"],"signatures":["~Haoyi_Wu1"],"forum":"P9CNir6qao","license":"CC BY-SA 4.0","number":153191,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1729077558079,"domain":"DBLP.org","id":"P9CNir6qao","version":2},{"content":{"summary":{"value":"This paper proposes a new attention kernel design, ChunkAttention, that can chunk, share the KV cache, and batch the attention computation for a multi-tenant LLM inference setting where the sequences from users may share long prompt prefixes. ChunkAttention identifies the matching prompt prefixes across several sequences and reuses their KV cache by chunking the KV cache and embedding it into the auxillary prefix tree. A 2-phase partitioning technique is also proposed to reduce the memory accesses on the KV cache during self-attention. Experimental results show ChunkAttention improves the self-attention speed by up to 3x."},"presentation":{"value":"3 good"},"contribution":{"value":"2 fair"},"soundness":{"value":"3 good"},"strengths":{"value":"1. The paper works on an important problem, i.e., accelerating self-attention kernels in LLMs.\n2. The paper flows well."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The tehniques proposed by this paper work for only the multi-tenant LLM inference setting where the sequences from users may share long prompt prefixes. However, how frequent can this case happen? How to identify the same prompts between different users is still a problem.\n\n2. This authors did NOT compare ChunkAttention agasint prior token pruning techniques, e.g.,\n\nSehoon Kim, Sheng Shen, David Thorsley, Amir Gholami, Woosuk Kwon, Joseph Hassoun, and\nKurt Keutzer. Learned token pruning for transformers. In Aidong Zhang and Huzefa Rangwala\n(eds.), KDD ’22: The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining,\nWashington, DC, USA, August 14 - 18, 2022, pp. 784–794. ACM, 2022. doi: 10.1145/3534678.\n3539260. URL https://doi.org/10.1145/3534678.3539260.\n\nor KV cache compression techniques, e.g., \nJesse Mu, Xiang Lisa Li, and Noah D. Goodman. Learning to compress prompts with gist tokens.\nCoRR, abs/2304.08467, 2023. doi: 10.48550/arXiv.2304.08467. URL https://doi.org/\n10.48550/arXiv.2304.08467."},"confidence":{"value":"3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked."},"questions":{"value":"Please comment on the points in the weakness section."},"rating":{"value":"5: marginally below the acceptance threshold"},"code_of_conduct":{"value":"Yes"}},"nonreaders":[],"tmdate":1699636957344,"tcdate":1698627354075,"writers":["ICLR.cc/2024/Conference","ICLR.cc/2024/Conference/Submission7819/Reviewer_ee1U"],"signatures":["ICLR.cc/2024/Conference/Submission7819/Reviewer_ee1U"],"forum":"9k27IITeAZ","number":2,"license":"CC BY 4.0","cdate":1698627354075,"readers":["everyone"],"invitations":["ICLR.cc/2024/Conference/Submission7819/-/Official_Review","ICLR.cc/2024/Conference/-/Edit"],"mdate":1699636957344,"domain":"ICLR.cc/2024/Conference","replyto":"9k27IITeAZ","id":"tICEz0XYcQ","forumContent":{"TLDR":{"value":"Batching long shared prompt prefixes to speed up self-attention based on chunked KV cache"},"venue":{"value":"Submitted to ICLR 2024"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["large language model","model inference","self attention"]},"supplementary_material":{"value":"/attachment/34a6aefe728c3e9a526d796dc036dc6b0d6a9571.zip"},"primary_area":{"value":"infrastructure, software libraries, hardware, etc."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide."},"abstract":{"value":"Self-attention is an essential component of GPT-style models and a significant cause of LLM inference latency for long sequences. In multi-tenant LLM inference servers, the compute and memory operation cost of self-attention can be amortized by making use of the probability that sequences from users may share long prompt prefixes. This paper introduces ChunkAttention, a unique self-attention kernel built on chunking, sharing the KV cache, and batching the attention computation. ChunkAttention recognizes matching prompt prefixes across several sequences and shares their KV cache in memory by chunking the KV cache and structuring it into the auxiliary prefix tree. To significantly improve the memory reuse of KV cache and consequently the speed of self-attention for long shared prompts, we design an efficient computation kernel on this new storage structure, where two-phased partitioning is implemented to reduce memory operations on shared KV cache during self-attention. Experiments show that ChunkAttention can speed up self-attention of long shared prompts 1.6-3 times, with lengths ranging from 1024 to 8192."},"_bibtex":{"value":"@misc{\nye2024chunkattention,\ntitle={ChunkAttention: Efficient Attention on {KV} Cache with Chunking Sharing and Batching},\nauthor={Lu Ye and Ze Tao and Yong Huang and Yang Li},\nyear={2024},\nurl={https://openreview.net/forum?id=9k27IITeAZ}\n}"},"title":{"value":"ChunkAttention: Efficient Attention on KV Cache with Chunking Sharing and Batching"},"pdf":{"value":"/pdf/4527081429f9d579d4db9e8df518075eb6db0de5.pdf"},"venueid":{"value":"ICLR.cc/2024/Conference/Rejected_Submission"},"paperhash":{"value":"ye|chunkattention_efficient_attention_on_kv_cache_with_chunking_sharing_and_batching"},"authorids":{"value":["~Lu_Ye1","~Ze_Tao2","~Yong_Huang4","~Yang_Li83"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors' identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Lu Ye","Ze Tao","Yong Huang","Yang Li"]}},"version":2},{"content":{"TLDR":{"value":"CacheRepair learns to restore cross-chunk attention information in independently prefilled KV caches, accelerating multi-document RAG while preserving answer quality."},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["Retrieval-Augmented Generation","KV Cache Fusion","Efficient LLM Inference"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Multi-document retrieval-augmented generation (RAG) requires a language model to process multiple retrieved text chunks before answering a question. Precomputing each chunk's KV cache independently and concatenating the caches when the chunks are retrieved can accelerate this step. However, the assembled cache lacks cross-chunk attention information, reducing answer quality. Selective recomputation methods recover the missing cross-chunk context by rerunning the target LLM on selected tokens, incurring substantial\nonline computation. We introduce CacheRepair, a lightweight network that learns the difference between independently computed KV caches and those produced by processing the chunks together. The network combines compressed KV features with token embeddings and uses attention that is bidirectional within each chunk and flows from earlier to later chunks. Each repair block receives the compressed cache features, and the predicted residual is added to every document token's cache. Each repair network is trained for a specific frozen target LLM on a generic retrieval corpus and reused across downstream datasets. Our analysis shows that repair reduces KV errors both near chunk boundaries and throughout chunk interiors. Evaluation across three target LLMs and four downstream datasets places\nCacheRepair on the measured answer-quality--latency Pareto frontier in eleven of twelve model--dataset combinations. Reported time to first token (TTFT) includes online cache transfer and repair. Across all twelve combinations, the largest repairers achieve 1.69--4.61$\\times$ speedups in median TTFT over full prefill and improve mean F1 by 2.1--26.1 percentage points over direct cache reuse."},"_bibtex":{"value":"@inproceedings{\nanonymous2026cacherepair,\ntitle={CacheRepair: Learning to Repair Cross-Chunk Context in {RAG} for {KV} Cache Fusion},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=WRp9Klu5s7},\nnote={under review}\n}"},"title":{"value":"CacheRepair: Learning to Repair Cross-Chunk Context in RAG for KV Cache Fusion"},"pdf":{"value":"/pdf/b7e1d67ab6f98f957ebce44f0b6ce68d6b1e19e4.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791224370356,"tcdate":1788778238857,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission10644/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission10644/Authors"],"forum":"WRp9Klu5s7","license":"CC BY 4.0","number":10644,"cdate":1788778238857,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Edit","ICLR.cc/2027/Conference/Submission10644/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing"],"mdate":1791224370356,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"WRp9Klu5s7","version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2505.10951v2"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"zhu|subgcache_accelerating_graphbased_rag_with_subgraphlevel_kv_cache"},"authorids":{"value":["~Qiuyu_Zhu3","https://dblp.org/search/pid/api?q=author:Liang_Zhang:","","https://dblp.org/search/pid/api?q=author:Cheng_Long:","https://dblp.org/search/pid/api?q=author:Jie_Zhang:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2505.10951"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2505-10951,\n publtype={informal},\n author={Qiuyu Zhu and Liang Zhang and Qianxiong Xu and Cheng Long and Jie Zhang},\n title={SubGCache: Accelerating Graph-based RAG with Subgraph-level KV Cache},\n year={2025},\n month={May},\n cdate={1746057600000},\n journal={CoRR},\n volume={abs/2505.10951},\n url={https://doi.org/10.48550/arXiv.2505.10951}\n}\n"},"abstract":{"value":"Graph-based retrieval-augmented generation (RAG) enables large language models (LLMs) to incorporate structured knowledge via graph retrieval as contextual input, enhancing more accurate and context-aware reasoning. We observe that for different queries, it could retrieve similar subgraphs as prompts, and thus we propose SubGCache, which aims to reduce inference latency by reusing computation across queries with similar structural prompts (i.e., subgraphs). Specifically, SubGCache clusters queries based on subgraph embeddings, constructs a representative subgraph for each cluster, and pre-computes the key-value (KV) cache of the representative subgraph. For each query with its retrieved subgraph within a cluster, it reuses the pre-computed KV cache of the representative subgraph of the cluster without computing the KV tensors again for saving computation. Experiments on two new datasets across multiple LLM backbones and graph-based RAG frameworks demonstrate that SubGCache consistently reduces inference latency with comparable and even improved generation quality, achieving up to 6.68$\\times$ reduction in time-to-first-token (TTFT)."},"title":{"value":"SubGCache: Accelerating Graph-based RAG with Subgraph-level KV Cache"},"authors":{"value":["Qiuyu Zhu","Liang Zhang","Qianxiong Xu","Cheng Long","Jie Zhang"]}},"tmdate":1791959971772,"pdate":1735689600000,"externalIds":["dblp:journals/corr/abs-2505-10951"],"tcdate":1767785510110,"writers":["~"],"signatures":["~Qianxiong_Xu1"],"forum":"IvYfvKKVLj","license":"CC BY-SA 4.0","number":725127,"cdate":1746057600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1791959971772,"domain":"DBLP.org","id":"IvYfvKKVLj","version":2},{"content":{"venue":{"value":"AAAI 2026"},"pdf":{"value":"https://ojs.aaai.org/index.php/AAAI/article/download/40827/44788"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"zhu|subgcache_accelerating_graphbased_rag_with_subgraphlevel_kv_cache"},"html":{"value":"https://doi.org/10.1609/aaai.v40i41.40827"},"_bibtex":{"value":"@inproceedings{DBLP:conf/aaai/ZhuZXLZ26,\n author={Qiuyu Zhu and Liang Zhang and Qianxiong Xu and Cheng Long and Jie Zhang},\n title={SubGCache: Accelerating Graph-based RAG with Subgraph-level KV Cache},\n year={2026},\n cdate={1767225600000},\n pages={35204-35212},\n url={https://doi.org/10.1609/aaai.v40i41.40827},\n booktitle={AAAI},\n crossref={conf/aaai/2026}\n}\n"},"abstract":{"value":"Graph-based retrieval-augmented generation (RAG) enables large language models (LLMs) to incorporate structured knowledge via graph retrieval as contextual input, enhancing more accurate and context-aware reasoning. We observe that for different queries, it could retrieve similar subgraphs as prompts, and thus we propose SubGCache, which aims to reduce inference latency by reusing computation across queries with similar structural prompts (i.e., subgraphs). Specifically, SubGCache clusters queries based on subgraph embeddings, constructs a representative subgraph for each cluster, and pre-computes the key-value (KV) cache of the representative subgraph. For each query with its retrieved subgraph within a cluster, it reuses the pre-computed KV cache of the representative subgraph of the cluster without computing the KV tensors again for saving computation. Extensive experiments on three datasets across multiple LLM backbones and graph-based RAG frameworks demonstrate that SubGCache consistently reduces inference latency with comparable and even improved generation quality, achieving up to 6.68x reduction in time-to-first-token (TTFT)."},"title":{"value":"SubGCache: Accelerating Graph-based RAG with Subgraph-level KV Cache"},"authors":{"value":[{"fullname":"Qiuyu Zhu","username":"~Qiuyu_Zhu3"},{"fullname":"Liang Zhang","username":""},{"fullname":"Qianxiong Xu","username":""},{"fullname":"Cheng Long","username":""},{"fullname":"Jie Zhang","username":""}]}},"tmdate":1779798108340,"pdate":1798675200000,"externalIds":["dblp:conf/aaai/ZhuZXLZ26"],"tcdate":1779798105372,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~ZHU_QIUYU1"],"forum":"XUMmoNQUXC","license":"CC BY-SA 4.0","number":28902,"cdate":1767225600000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1779798108340,"domain":"OpenReview.net/Public_Article","id":"XUMmoNQUXC","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2407.18003v4"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"shi|keep_the_cost_down_a_review_on_methods_to_optimize_llm_s_kvcache_consumption"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Luohe_Shi:","https://dblp.org/search/pid/api?q=author:Hongyi_Zhang:","~Yao_Yao8","https://dblp.org/search/pid/api?q=author:Zuchao_Li:","https://dblp.org/search/pid/api?q=author:Hai_Zhao_0001:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2407.18003"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2407-18003,\n publtype={informal},\n author={Luohe Shi and Hongyi Zhang and Yao Yao and Zuchao Li and Hai Zhao},\n title={Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption},\n year={2024},\n cdate={1704067200000},\n journal={CoRR},\n volume={abs/2407.18003},\n url={https://doi.org/10.48550/arXiv.2407.18003}\n}\n"},"abstract":{"value":"Large Language Models (LLMs), epitomized by ChatGPT's release in late 2022, have revolutionized various industries with their advanced language comprehension. However, their efficiency is challenged by the Transformer architecture's struggle with handling long texts. KV Cache has emerged as a pivotal solution to this issue, converting the time complexity of token generation from quadratic to linear, albeit with increased GPU memory overhead proportional to conversation length. With the development of the LLM community and academia, various KV Cache compression methods have been proposed. In this review, we dissect the various properties of KV Cache and elaborate on various methods currently used to optimize the KV Cache space usage of LLMs. These methods span the pre-training phase, deployment phase, and inference phase, and we summarize the commonalities and differences among these methods. Additionally, we list some metrics for evaluating the long-text capabilities of large language models, from both efficiency and capability perspectives. Our review thus sheds light on the evolving landscape of LLM optimization, offering insights into future advancements in this dynamic field. Links to the papers mentioned in this review can be found in our Github Repo https://github.com/zcli-charlie/Awesome-KV-Cache."},"title":{"value":"Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption"},"authors":{"value":["Luohe Shi","Hongyi Zhang","Yao Yao","Zuchao Li","Hai Zhao"]}},"tmdate":1744273203166,"pdate":1704067200000,"tcdate":1744273201107,"writers":["~"],"signatures":["~Yao_Yao8"],"forum":"ED898ig7RZ","license":"CC BY-SA 4.0","number":387485,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1744273203166,"domain":"DBLP.org","id":"ED898ig7RZ","version":2},{"content":{"TLDR":{"value":"Proposes $\\ell_2$-norm KV-cache eviction (no attention tracking). Matches full cache at $\\ge$ 512 tokens, but at 256 tokens underperforms sliding window—recency dominates at tight budgets."},"venue":{"value":"ICLR 2026 Workshop MemAgents"},"pdf":{"value":"/pdf/44d92c5b30de18b32cc26abc9e6e239e33beafd1.pdf"},"keywords":{"value":["KV-cache compression","Heavy-Hitter Oracle","autonomous agents","memory efficiency","transformer optimization","VRAM management","sparse attention","foundation models"]},"venueid":{"value":"ICLR.cc/2026/Workshop/MemAgent"},"paperhash":{"value":"prasanth|normguided_kvcache_eviction_for_memoryefficient_reasoning"},"authorids":{"value":["~Prasanth1"]},"abstract":{"value":"Large language models deployed as autonomous agents face a fundamental memory\nconstraint: the KV-cache required for autoregressive generation scales\nquadratically with context length. We propose \\textbf{$\\ell_2$-Norm Eviction},\na novel gradient-free KV-cache compression method that scores tokens by the\nmean $\\ell_2$-norm of their key vectors across attention heads, retaining a\nhybrid of high-norm heavy hitters and recent tokens. Unlike H2O~\\cite{h2o},\nwhich requires accumulating explicit attention scores across all decoding steps,\nour method operates with a single pass over key tensors and imposes no\nattention-tracking overhead. We evaluate $\\ell_2$-Norm Eviction against a\nfull-cache baseline and a StreamingLLM-style sliding window on the GSM8K\nmathematical reasoning benchmark and curated logic prompts, using automated\nExact Match scoring across four cache budgets (256--2048 tokens) on\nMistral-7B-Instruct-v0.3. At budgets 512--2048, the eviction condition ($T > B$) is never satisfied\nbecause total sequence lengths remain below 512 tokens in our evaluation set;\nno tokens are dropped and all methods match the full-cache baseline exactly.\nAt the extreme budget of 256 (87.5\\% reduction), where eviction does fire,\nthe sliding window (EM=0.25) outperforms $\\ell_2$-Norm Eviction\n(EM=0.05) on GSM8K, indicating that recency dominates global token\nimportance at very tight budgets. We characterise this as a minimum viable\nbudget effect and identify adaptive pool sizing as the key direction for\nclosing this gap."},"_bibtex":{"value":"@inproceedings{\nprasanth2026normguided,\ntitle={Norm-Guided {KV}-Cache Eviction for Memory-Efficient Reasoning},\nauthor={Prasanth},\nbooktitle={ICLR 2026 Workshop on Memory for LLM-Based Agentic Systems},\nyear={2026},\nurl={https://openreview.net/forum?id=xOW2jXDKG3}\n}"},"title":{"value":"Norm-Guided KV-Cache Eviction for Memory-Efficient Reasoning"},"authors":{"value":["Prasanth"]}},"tmdate":1777131174253,"pdate":1772542116343,"tcdate":1771056679448,"writers":["ICLR.cc/2026/Workshop/MemAgent","ICLR.cc/2026/Workshop/MemAgent/Submission101/Authors"],"signatures":["ICLR.cc/2026/Workshop/MemAgent/Submission101/Authors"],"forum":"xOW2jXDKG3","license":"CC BY 4.0","number":101,"cdate":1771056679448,"readers":["everyone"],"invitations":["ICLR.cc/2026/Workshop/MemAgent/-/Submission","ICLR.cc/2026/Workshop/MemAgent/-/Post_Submission","ICLR.cc/2026/Workshop/MemAgent/-/Edit","ICLR.cc/2026/Workshop/MemAgent/Submission101/-/Revision"],"mdate":1777131174253,"odate":1772770085981,"domain":"ICLR.cc/2026/Workshop/MemAgent","id":"xOW2jXDKG3","version":2},{"content":{"TLDR":{"value":"KVShareArena is an open evaluation framework for KV-cache reuse when the preceding text or the LLM that wrote the cache changes. It compares methods under the same conditions on quality and cost, with a common interface and a leaderboard."},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["KV cache reuse","large language models","benchmarking","retrieval-augmented generation","multi-agent systems","cross-model transfer"]},"supplementary_material":{"value":"/attachment/380b88f24d58d58940979c8c96fc475a3fac5fe5.zip"},"primary_area":{"value":"datasets and benchmarks"},"abstract":{"value":"Reusing key–value (KV) caches speeds up LLM inference by avoiding repeated computation on shared text. The standard method, prefix caching, reuses a KV cache only when the LLM is the same and all text before the reused content is identical. However, real workloads often break both conditions. The preceding text changes when a RAG system places different documents before the same document, or when agents with different system prompts read the same file or tool output. The LLM changes when multi-agent workflows use specialized LLMs on shared material, or when an updated model reads documents cached by its previous version. Because KV caches depend on both the preceding text and the model weights, direct reuse in these settings can reduce answer quality. Many methods repair the reused cache or compress it, each with its own balance between quality and cost, so choosing among them requires a comparison under the same conditions. However, each method paper uses its own tasks, models, and cost measures, so the reported results cannot be compared directly. Existing benchmarks do not provide such a comparison either, because they mainly test long-context processing or reuse of an unchanged prefix. To fill this gap, we introduce KVShareArena, a benchmark and open evaluation framework that compares these methods under the same conditions. KVShareArena has (1) reuse tests on 2,150 questions from three QA datasets, in which the preceding text, the LLM that wrote the cache, or both change while the answering LLM and the input text stay fixed; (2) five dense and mixture-of-experts LLMs from 4B to 30B parameters and six LLM pairs in which one version of an LLM reads caches written by another, for 33 model–dataset settings in total; (3) 11 repair and compression methods from six method classes; (4) four evaluation perspectives: answer quality, prefill computation, KV-cache memory, and latency; and (5) a common interface for adding new methods and an interactive leaderboard that lets users set their own balance between quality and speed. Experiments on KVShareArena yield two findings. First, both the quality loss from reuse and which repairs help depend on the LLM, even between two 8B models. Second, most repairs keep their quality when another version of the LLM wrote the cache, but a trained repair adapter loses quality in 12 of the 18 pair–dataset tests."},"_bibtex":{"value":"@inproceedings{\nanonymous2026kvsharearena,\ntitle={{KVS}hareArena: {KV}-Cache Reuse Across Contexts and Model Checkpoints},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=VeFsdb2Do6},\nnote={under review}\n}"},"title":{"value":"KVShareArena: KV-Cache Reuse Across Contexts and Model Checkpoints"},"pdf":{"value":"/pdf/72c4784d68c79f76f08062348b850e833a957013.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791225572562,"tcdate":1788987559946,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission12634/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission12634/Authors"],"forum":"VeFsdb2Do6","license":"CC BY 4.0","number":12634,"cdate":1788987559946,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Edit","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission12634/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing"],"mdate":1791225572562,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"VeFsdb2Do6","version":2},{"content":{"summary":{"value":"This paper introduces dLLM-Cache, an adaptive caching framework designed to accelerate diffusion-based large language models (dLLMs) without retraining. The key idea is to cache intermediate representations during inference, leveraging redundancy in prompts and generated tokens. The proposed cache selection strategy dynamically balances accuracy and efficiency using prompt similarity and token reuse mechanisms. Experiments on various LLM-based diffusion tasks (e.g., LLaDA, DreamFusion) show that dLLM-Cache achieves substantial inference speedup with minimal degradation in performance."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"N/A"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. The paper is clearly written and well-structured, making the method easy to understand.\n2. The adaptive caching mechanism is simple yet effective, requiring no retraining or architectural modification.\n3. Comprehensive experiments demonstrate strong empirical improvements across multiple diffusion-based LLMs.\n4. Ablation studies support the robustness of the cache selection mechanism."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The theoretical complexity reduction is not analyzed in sufficient depth. A more formal discussion or derivation of the speed–accuracy trade-off would greatly enhance clarity.\n2. Since dLLM-Cache reuses outdated KV pairs, it would strengthen the work to mathematically and empirically analyze the upper bound of the approximation error under different values of K.\n3. The limitations of the proposed method, particularly under dynamic or semantically diverse prompts, are not fully explored. A deeper investigation of such edge cases would improve the completeness of the study.\n4. Several tables (e.g., Table 1, Table 2, and Table 3) present overlapping or repetitive results, with only minor variations in metrics or experimental settings. These could be merged or reorganized to improve readability and reduce redundancy.\n5. The observation that using outdated cache can sometimes improve accuracy is intriguing but insufficiently discussed. A more detailed explanation or hypothesis about why this occurs would make the findings more convincing."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762924081986,"tcdate":1762236418943,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission13465/Reviewer_S22Y"],"signatures":["ICLR.cc/2026/Conference/Submission13465/Reviewer_S22Y"],"forum":"DriG3hgh42","number":3,"license":"CC BY 4.0","cdate":1762236418943,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission13465/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762924081986,"domain":"ICLR.cc/2026/Conference","replyto":"DriG3hgh42","id":"DTfnIRFsL7","forumContent":{"TLDR":{"value":"We accelerate diffusion LLMs with a training-free adaptive caching strategy, reusing prompt and response features to significantly reduce inference latency."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Diffusion Large Language Models","Model Acceleration","Adaptive Caching"]},"supplementary_material":{"value":"/attachment/7fbf8792aa1a3f3f392030e87abf6d6ef0aa7d3e.zip"},"primary_area":{"value":"generative models"},"abstract":{"value":"Autoregressive Models (ARMs) have long dominated the landscape of Large Language Models. Recently, a new paradigm has emerged in the form of diffusion-based Large Language Models (dLLMs), which generate text by iteratively denoising masked segments. This approach has shown significant advantages and potential. However, dLLMs suffer from high inference latency. Traditional ARM acceleration techniques, such as Key-Value caching, are incompatible with dLLMs due to their bidirectional attention mechanism. To address this specific challenge, our work begins with a key observation that dLLM inference involves a static prompt and a partially dynamic response, where most tokens remain stable across adjacent denoising steps. Based on this, we propose dLLM-Cache, a training-free adaptive caching framework that combines long-interval prompt caching with partial response updates guided by feature similarity. This design enables efficient reuse of intermediate computations without compromising model performance. Extensive experiments on representative dLLMs, including LLaDA 8B and Dream 7B, show that dLLM-Cache achieves up to *9.1*$\\times$ speedup over standard inference without compromising output quality. Notably, our method brings dLLM inference latency close to that of ARMs under many settings. *Codes are provided in the supplementary material and will be released publicly on GitHub.*"},"_bibtex":{"value":"@misc{\nliu2026dllmcache,\ntitle={d{LLM}-Cache: Accelerating Diffusion Large Language Models with Adaptive Caching},\nauthor={Zhiyuan Liu and Yicun Yang and Yaojie Zhang and Junjie Chen and Chang Zou and Qingyan Wei and Shaobo Wang and Yichen Zhu and Linfeng Zhang},\nyear={2026},\nurl={https://openreview.net/forum?id=DriG3hgh42}\n}"},"title":{"value":"dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Caching"},"pdf":{"value":"/pdf/bbb0d922478ec6b430e560769f62a1cb1fcbcc0e.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"liu|dllmcache_accelerating_diffusion_large_language_models_with_adaptive_caching"},"authorids":{"value":["~Zhiyuan_Liu15","~Yicun_Yang1","~Yaojie_Zhang2","~Junjie_Chen9","~Chang_Zou2","~Qingyan_Wei1","~Shaobo_Wang1","~Yichen_Zhu1","~Linfeng_Zhang2"]},"authors":{"value":["Zhiyuan Liu","Yicun Yang","Yaojie Zhang","Junjie Chen","Chang Zou","Qingyan Wei","Shaobo Wang","Yichen Zhu","Linfeng Zhang"]}},"version":2},{"content":{"venue":{"value":"CoRR 2026"},"pdf":{"value":"https://arxiv.org/pdf/2604.19351v4"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"guo|dashkv_accelerating_longcontext_llm_inference_via_asymmetric_kv_cache_hashing"},"html":{"value":"https://doi.org/10.48550/arXiv.2604.19351"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2604-19351,\n publtype={informal},\n author={Jinyu Guo and Zhihan Zhang and Yutong Li and Jiehui Xie and Md. Tamim Iqbal and Dongshen Han and Lik-Hang Lee and Sung-Ho Bae and Jie Zou and Yang Yang and Chaoning Zhang},\n title={DASH-KV: Accelerating Long-Context LLM Inference via Asymmetric KV Cache Hashing},\n year={2026},\n month={April},\n cdate={1775001600000},\n journal={CoRR},\n volume={abs/2604.19351},\n url={https://doi.org/10.48550/arXiv.2604.19351}\n}\n"},"abstract":{"value":"The quadratic computational complexity of the standard attention mechanism constitutes a fundamental bottleneck for large language models in long-context inference. While existing KV cache compression methods alleviate memory pressure, they often sacrifice generation quality and fail to address the high overhead of floating-point arithmetic. This paper introduces DASH-KV, an innovative acceleration framework that reformulates attention as approximate nearest-neighbor search via asymmetric deep hashing. Under this paradigm, we design an asymmetric encoding architecture that differentially maps queries and keys to account for their distinctions in precision and reuse characteristics. To balance efficiency and accuracy, we further introduce a dynamic mixed-precision mechanism that adaptively retains full-precision computation for critical tokens. Extensive experiments on LongBench demonstrate that DASH-KV significantly outperforms state-of-the-art baseline methods while matching the performance of full attention, all while reducing inference complexity from O(N^2) to linear O(N)."},"title":{"value":"DASH-KV: Accelerating Long-Context LLM Inference via Asymmetric KV Cache Hashing"},"authors":{"value":[{"fullname":"Jinyu Guo","username":""},{"fullname":"Zhihan Zhang","username":""},{"fullname":"Yutong Li","username":""},{"fullname":"Jiehui Xie","username":"~Jiehui_Xie1"},{"fullname":"Md. Tamim Iqbal","username":""},{"fullname":"Dongshen Han","username":""},{"fullname":"Lik-Hang Lee","username":""},{"fullname":"Sung-Ho Bae","username":""},{"fullname":"Jie Zou","username":""},{"fullname":"Yang Yang","username":""},{"fullname":"Chaoning Zhang","username":""}]}},"tmdate":1784517319947,"pdate":1798675200000,"externalIds":["dblp:journals/corr/abs-2604-19351"],"tcdate":1784517317637,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Jiehui_Xie1"],"forum":"244SWinDDs","license":"CC BY-SA 4.0","number":65902,"cdate":1775001600000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1784517319947,"domain":"OpenReview.net/Public_Article","id":"244SWinDDs","version":2},{"content":{"venue":{"value":"ICLR 2027 Conference Submission"},"TLDR":{"value":"TaSQ tailors the VQ target space for accurate one-bit KV cache compression, outperforming prior KV cache VQ methods with negligible serving overhead."},"keywords":{"value":["KV cache compression","vector quantization","low-bit quantization","large language model inference","efficient serving"]},"supplementary_material":{"value":"/attachment/30bc9f8a57f9668022cd3ca6ec91c8335529e96a.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"The key-value (KV) cache becomes a major memory bottleneck in long-context LLM inference, placing substantial pressure on memory capacity and bandwidth.\nTo mitigate this bottleneck, vector quantization (VQ) has emerged as a promising approach for aggressive KV cache compression.\nHowever, existing VQ methods degrade substantially in the 1-bit regime.\nAt such extreme compression, each codebook must represent a larger group of channels with a limited set of centroids, making effective use of its capacity increasingly challenging.\nTo address this, we introduce $\\textbf{TaSQ}$, which tailors the VQ target space by combining query-guided channel weighting, cross-head normalization, and covariance-aware channel grouping to better reflect the error sensitivity and statistical structure of cached activations.\nSince these transforms are RoPE-compatible and can be easily merged into projection weights and codebooks, TaSQ preserves the conventional VQ lookup structure and adds negligible serving overhead.\nAcross general, long-chain-of-thought reasoning, and long-context retrieval benchmarks, TaSQ consistently outperforms existing low-bit KV cache VQ baselines while preserving reasoning stability.\nOn a single RTX 6000 Ada GPU, its SGLang implementation supports up to $14\\times$ larger batch sizes and achieves $1.87\\times$ higher peak throughput compared to the BF16 baseline."},"_bibtex":{"value":"@inproceedings{\nanonymous2026tailoring,\ntitle={Tailoring the Quantization Space for 1-Bit {KV} Cache Compression},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=lWUM9JOGww},\nnote={under review}\n}"},"title":{"value":"Tailoring the Quantization Space for 1-Bit KV Cache Compression"},"pdf":{"value":"/pdf/0f9d7b4a7554408f04dbefadab056fe76f6c966a.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791232666988,"tcdate":1789717416295,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission39044/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission39044/Authors"],"forum":"lWUM9JOGww","license":"CC BY 4.0","number":39044,"cdate":1789717416295,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission39044/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791232666988,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"lWUM9JOGww","version":2},{"content":{"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["KV Cache eviction","coverage calibration","latent Memory"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Long-context decoding is increasingly constrained by key--value (KV) cache memory and bandwidth. Existing fixed-budget compression methods typically separate retention from compensation, while a retention ranking specifies neither discarded attention mass nor the direction of induced output error. We start from an exact factorization: eviction error equals evicted attention $\\textit{mass}$ times the $\\textit{directional gap}$ between the evicted centroid and retained output, highlighting the importance of set-level coverage in retention and mass-preserving memory writing. We introduce $\\textbf{CORE}$ ($\\underline{\\textbf{CO}}$verage Calibration and Evicted-Mass $\\underline{\\textbf{RE}}$distribution for KV Cache), which distills an offline allocation combining query utility and log-determinant coverage into a lightweight cache-aware indexer. At inference, one calibrated distribution drives both channels: its Top-$B$ ordering retains complementary KV states, while its excluded allocation mass and conditional weights parameterize latent-memory writes without a separate write-weight predictor or online log-determinant evaluation. Our analysis provides a four-term pre-compensation error certificate, characterizes non-additive coverage interactions, and establishes mass-independent\nwrite stability with hierarchical bounds through recurrent and\nquery-adaptive normalization. Across three backbones, CORE exceeds the strongest RULER baseline by up to 3.78 points at 90\\% compression; LongBench and repeated-eviction evaluations further demonstrate strong effectiveness and decoding efficiency."},"_bibtex":{"value":"@inproceedings{\nanonymous2026core,\ntitle={{CORE}: {CO}verage {CA}libration and Evicted-Mass {RE}distribution for {KV} Cache},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=Y00XNwRb0F},\nnote={under review}\n}"},"title":{"value":"CORE: COverage CAlibration and Evicted-Mass REdistribution for KV Cache"},"pdf":{"value":"/pdf/72db8458746e884c0e0b9e216ffbf042e9971316.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791218692110,"tcdate":1787147700599,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission50/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission50/Authors"],"forum":"Y00XNwRb0F","license":"CC BY 4.0","number":50,"cdate":1787147700599,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Edit","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission50/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing"],"mdate":1791218692110,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"Y00XNwRb0F","version":2},{"content":{"summary":{"value":"The paper proposed a method to minimize the risk of KV cache eviction by efficiently and soundly offloading some of them into external memory, which is realized by page organization, page digest, and digest ranking/scoring. The method gets much better performance in context retrieval tasks compared to other KV eviction methods."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"I'm wondering if there will be an open-sourced implementation/PR to vLLM?"},"rating":{"value":7},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"The paper proposes a reliable way to do KV cache eviction with minimal risk.\nThe system is sound and intuitive. It seems compatible with real world frameworks like vLLM.\nExperiments show good results."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"It's not very clear if the dynamics of importance will change along the decoding in other tasks, so the observations may be limited.\nIt would be interesting to see how the page size affect the methods, as vLLM's default page size is 16. Is there any reason to scale to 32?\nWould the method still work when TP>1?"},"limitations":{"value":"It's desirable to see normal length tasks, e.g., GSM8K."}},"nonreaders":[],"tmdate":1730879877509,"tcdate":1721030926755,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission16833/Reviewer_ntoN"],"signatures":["NeurIPS.cc/2024/Conference/Submission16833/Reviewer_ntoN"],"forum":"4oAt5L4lYe","number":3,"license":"CC BY 4.0","cdate":1721030926755,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission16833/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730879877509,"domain":"NeurIPS.cc/2024/Conference","replyto":"4oAt5L4lYe","id":"ggq2UPwcXq","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"TLDR":{"value":"We propose a method to dynamically evict some KV cache tokens (pages) or recall evicted ones for different query tokens, reducing the memory usage of the KV cache and the latency of attention computation with few accuracy loss."},"keywords":{"value":["Machine Learning System","Large Language Model Inference","Key-Value Cache Eviction"]},"primary_area":{"value":"infrastructure"},"abstract":{"value":"Large Language Models (LLMs) are widely used in today's tasks of natural language processing. \nTo support applications like multi-turn chats, document understanding, and content generation, models with long context lengths are growing in importance.\nHowever, managing long contexts brings substantial challenges due to the expansion of key-value cache (KV cache). Longer KV cache requires larger memory, limiting the batch-size thus decreasing throughput. Also, computing attention over long KV cache incurs more memory access, hurting the end-to-end latency.\nPrior works find that it is sufficient to use only the recent and high-impact tokens for attention computation, allowing the eviction of less vital tokens to shrink cache size.\nNonetheless, we observe a dynamic shift in token importance across different decoding steps. Tokens initially evicted might regain importance after certain decoding steps.\nTo address this, we propose ArkVale, a page-based KV cache manager that can recognize and recall currently important tokens evicted before. We asynchronously copy the filled page into external memory (e.g., CPU memory) as backup and summarize it into a much smaller digest by constructing the bounding-volume of its keys. Before attention computation, we measure all pages' importance based on their digests, recall the important ones, evict the unimportant ones, and select the top-ranked pages for attention computation. \nExperiment results show that ArkVale performs well on various long context tasks with negligible accuracy loss under 2k$\\sim$4k cache budget and can improve decoding latency to $2.2\\times$ and batching throughput to $4.6\\times$ because it applies attention on only a small subset of pages and reduce per-sample memory usage of KV cache."},"_bibtex":{"value":"@inproceedings{\nchen2024arkvale,\ntitle={ArkVale: Efficient Generative {LLM} Inference with Recallable Key-Value Eviction},\nauthor={Renze Chen and Zhuofeng Wang and Beiquan Cao and Tong Wu and Size Zheng and Xiuhong Li and Xuechao Wei and Shengen Yan and Meng Li and Yun Liang},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=4oAt5L4lYe}\n}"},"title":{"value":"ArkVale: Efficient Generative LLM Inference with Recallable Key-Value Eviction"},"pdf":{"value":"/pdf/fc7d6c105320063bd046924651f65ecd45ef8647.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"chen|arkvale_efficient_generative_llm_inference_with_recallable_keyvalue_eviction"},"authorids":{"value":["~Renze_Chen1","~Zhuofeng_Wang2","~Beiquan_Cao1","~Tong_Wu21","~Size_Zheng1","~Xiuhong_Li1","~Xuechao_Wei1","~Shengen_Yan1","~Meng_Li1","~Yun_Liang1"]},"authors":{"value":["Renze Chen","Zhuofeng Wang","Beiquan Cao","Tong Wu","Size Zheng","Xiuhong Li","Xuechao Wei","Shengen Yan","Meng Li","Yun Liang"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2410.08584v2"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"he|zipvl_efficient_large_visionlanguage_models_with_dynamic_token_sparsification_and_kv_cache_compression"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Yefei_He:","https://dblp.org/search/pid/api?q=author:Feng_Chen:","https://dblp.org/search/pid/api?q=author:Jing_Liu:","~Wenqi_Shao1","https://dblp.org/search/pid/api?q=author:Hong_Zhou:","https://dblp.org/search/pid/api?q=author:Kaipeng_Zhang:","~Bohan_Zhuang1"]},"html":{"value":"https://doi.org/10.48550/arXiv.2410.08584"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2410-08584,\n publtype={informal},\n author={Yefei He and Feng Chen and Jing Liu and Wenqi Shao and Hong Zhou and Kaipeng Zhang and Bohan Zhuang},\n title={ZipVL: Efficient Large Vision-Language Models with Dynamic Token Sparsification and KV Cache Compression},\n year={2024},\n cdate={1704067200000},\n journal={CoRR},\n volume={abs/2410.08584},\n url={https://doi.org/10.48550/arXiv.2410.08584}\n}\n"},"abstract":{"value":"The efficiency of large vision-language models (LVLMs) is constrained by the computational bottleneck of the attention mechanism during the prefill phase and the memory bottleneck of fetching the key-value (KV) cache in the decoding phase, particularly in scenarios involving high-resolution images or videos. Visual content often exhibits substantial redundancy, resulting in highly sparse attention maps within LVLMs. This sparsity can be leveraged to accelerate attention computation or compress the KV cache through various approaches. However, most studies focus on addressing only one of these bottlenecks and do not adequately support dynamic adjustment of sparsity concerning distinct layers or tasks. In this paper, we present ZipVL, an efficient inference framework designed for LVLMs through a dynamic ratio allocation strategy of important tokens. This ratio is adaptively determined based on the layer-specific distribution of attention scores, rather than fixed hyper-parameters, thereby improving efficiency for less complex tasks while maintaining high performance for more challenging ones. Then we select important tokens based on their normalized attention scores and perform sparse attention mechanism solely on those important tokens, reducing the latency in the prefill phase. Tokens deemed less important will be discarded to reduce KV cache size, alleviating the memory bottleneck in the decoding phase. Our experiments demonstrate that ZipVL can accelerate the prefill phase by 2.3$\\times$ and improve decoding throughput by 2.8$\\times$, with a minimal accuracy reduction of only 0.5\\% on VQAv2 benchmark over LLaVA-Next-13B model, effectively enhancing the generation efficiency of LVLMs."},"title":{"value":"ZipVL: Efficient Large Vision-Language Models with Dynamic Token Sparsification and KV Cache Compression"},"authors":{"value":["Yefei He","Feng Chen","Jing Liu","Wenqi Shao","Hong Zhou","Kaipeng Zhang","Bohan Zhuang"]}},"tmdate":1767793105038,"pdate":1704067200000,"tcdate":1740890568032,"writers":["~"],"signatures":["~Wenqi_Shao2"],"forum":"XmqBgUBSUG","license":"CC BY-SA 4.0","number":343548,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1767793105038,"domain":"DBLP.org","id":"XmqBgUBSUG","version":2},{"content":{"summary":{"value":"This paper introduces kvtc, a lightweight transform coder designed to compress KV cache for efficient storage in LLM inference. Drawing on classical media compression techniques, kvtc integrates PCA-based feature decorrelation, adaptive quantization, and entropy coding, requiring only brief initial calibration without modifying model parameters . It achieves up to 20× compression ratio on KV cache of models like Llama 3.1, Mistral NeMo, and Qwen 2.5 R1, while maintaining reasoning and long-context accuracy across benchmarks such as GSM8K, MMLU, and LiveCodeBench."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"See above"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"+ The paper presents an interesting approach by using a transform-coding framework for KV cache compression. Experiments demonstrate that this method can maintain model performance even with a high compression rate.\n\n+ The ablation experiments are very thorough."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"+ The paper uses a limited number of evaluation benchmarks. Recent related works typically conduct comprehensive experiments on benchmarks like LongBench and RULER, whereas this paper only uses one dataset from each (Qasper and VT).\n\n+ The \"Related Work\" section should also include content related to transform coding, as it is central to the proposed approach in this paper."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762931266841,"tcdate":1762002726967,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission19321/Reviewer_rALC"],"signatures":["ICLR.cc/2026/Conference/Submission19321/Reviewer_rALC"],"forum":"aNVKROYpLB","number":4,"license":"CC BY 4.0","cdate":1762002726967,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission19321/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762931266841,"domain":"ICLR.cc/2026/Conference","replyto":"aNVKROYpLB","id":"swskDkvLw6","forumContent":{"TLDR":{"value":"We present KVTC, a lightweight transform coder that allows for extended retention of transformer KV-cache via compression."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["transformer","kv cache","compression"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Serving large language models (LLMs) at scale necessitates efficient key-value (KV) cache management. KV caches can be reused across conversation turns via shared-prefix prompts that are common in iterative code editing and chat. However, stale caches consume scarce GPU memory, require offloading, or force recomputation. We present KVTC, a lightweight transform coder that compresses KV caches for compact on-GPU and off-GPU storage. Drawing on classical media compression, KVTC combines PCA-based feature decorrelation, adaptive quantization, and entropy coding. It requires only a brief initial calibration and leaves model parameters unchanged. By exploiting redundancies in KV caches, KVTC achieves up to 20x compression while maintaining reasoning and long-context accuracy, and 40x or higher for specific use cases. We test KVTC with Llama 3, Mistral NeMo, and R1-Qwen 2.5 models across benchmarks\nincluding AIME25, GSM8K, LiveCodeBench, LongBench, MATH-500, MMLU, Qasper and RULER.\nIt consistently outperforms inference-time baselines such as token eviction, quantization, and SVD-based methods, while achieving higher compression ratios. These results support KVTC as a practical building block for memory-efficient LLM serving with reusable KV caches."},"_bibtex":{"value":"@inproceedings{\nstaniszewski2026kv,\ntitle={{KV} Cache Transform Coding for Compact Storage in {LLM} Inference},\nauthor={Konrad Staniszewski and Adrian {\\L}a{\\'n}cucki},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=aNVKROYpLB}\n}"},"title":{"value":"KV Cache Transform Coding for Compact Storage in LLM Inference"},"pdf":{"value":"/pdf/1cef9774f0f0cf7bb9e4b167882e3ad3ef8cde16.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"staniszewski|kv_cache_transform_coding_for_compact_storage_in_llm_inference"},"authorids":{"value":["~Konrad_Staniszewski1","~Adrian_Łańcucki1"]},"authors":{"value":["Konrad Staniszewski","Adrian Łańcucki"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2406.03482v2"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"zandieh|qjl_1bit_quantized_jl_transform_for_kv_cache_quantization_with_zero_overhead"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Amir_Zandieh:","~Majid_Daliri1","https://dblp.org/search/pid/api?q=author:Insu_Han:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2406.03482"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2406-03482,\n publtype={informal},\n author={Amir Zandieh and Majid Daliri and Insu Han},\n title={QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overhead},\n year={2024},\n cdate={1704067200000},\n journal={CoRR},\n volume={abs/2406.03482},\n url={https://doi.org/10.48550/arXiv.2406.03482}\n}\n"},"abstract":{"value":"Serving LLMs requires substantial memory due to the storage requirements of Key-Value (KV) embeddings in the KV cache, which grows with sequence length. An effective approach to compress KV cache is quantization. However, traditional quantization methods face significant memory overhead due to the need to store quantization constants (at least a zero point and a scale) in full precision per data block. Depending on the block size, this overhead can add 1 or 2 bits per quantized number. We introduce QJL, a new quantization approach that consists of a Johnson-Lindenstrauss (JL) transform followed by sign-bit quantization. In contrast to existing methods, QJL eliminates memory overheads by removing the need for storing quantization constants. We propose an asymmetric estimator for the inner product of two vectors and demonstrate that applying QJL to one vector and a standard JL transform without quantization to the other provides an unbiased estimator with minimal distortion. We have developed an efficient implementation of the QJL sketch and its corresponding inner product estimator, incorporating a lightweight CUDA kernel for optimized computation. When applied across various LLMs and NLP tasks to quantize the KV cache to only 3 bits, QJL demonstrates a more than fivefold reduction in KV cache memory usage without compromising accuracy, all while achieving faster runtime. Codes are available at \\url{https://github.com/amirzandieh/QJL}."},"title":{"value":"QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overhead"},"authors":{"value":["Amir Zandieh","Majid Daliri","Insu Han"]}},"tmdate":1722661897259,"pdate":1704067200000,"tcdate":1722661894552,"writers":["~"],"signatures":["~Majid_Daliri1"],"forum":"fDzqgyrlRz","license":"CC BY-SA 4.0","number":49765,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1722661897259,"domain":"DBLP.org","id":"fDzqgyrlRz","version":2},{"content":{"venue":{"value":"CoRR 2026"},"pdf":{"value":"https://arxiv.org/pdf/2604.19157v1"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"jia|sawint4_systemaware_4bit_kvcache_quantization_for_realworld_llm_serving"},"html":{"value":"https://doi.org/10.48550/arXiv.2604.19157"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2604-19157,\n publtype={informal},\n author={Jinda Jia and Jisen Li and Zhongzhu Zhou and Jung Hwan Heo and Jue Wang and Tri Dao and Shuaiwen Leon Song and Ben Athiwaratkun and Chenfeng Xu and Tianyi Zhang and Xiaoxia Wu},\n title={SAW-INT4: System-Aware 4-Bit KV-Cache Quantization for Real-World LLM Serving},\n year={2026},\n month={April},\n cdate={1775001600000},\n journal={CoRR},\n volume={abs/2604.19157},\n url={https://doi.org/10.48550/arXiv.2604.19157}\n}\n"},"abstract":{"value":"KV-cache memory is a major bottleneck in real-world LLM serving, where systems must simultaneously support latency-sensitive small-batch requests and high-throughput concurrent workloads. Although many KV-cache compression methods improve offline accuracy or compression ratio, they often violate practical serving constraints such as paged memory layouts, regular memory access, and fused attention execution, limiting their effectiveness in deployment. In this work, we identify the minimal set of 4-bit KV-cache quantization methods that remain viable under these constraints. Our central finding is that a simple design--token-wise INT4 quantization with block-diagonal Hadamard rotation--consistently achieves the best accuracy-efficiency trade-off. Across multiple models and benchmarks, this approach recovers nearly all of the accuracy lost by naive INT4, while more complex methods such as vector quantization and Hessian-aware quantization provide only marginal additional gains once serving compatibility is taken into account. To make this practical, we implement a fused rotation-quantization kernel that integrates directly into paged KV-cache layouts and introduces zero measurable end-to-end overhead, matching plain INT4 throughput across concurrency levels. Our results show that effective KV-cache compression is fundamentally a systems co-design problem: under real serving constraints, lightweight block-diagonal Hadamard rotation is a viable method that delivers near-lossless accuracy without sacrificing serving efficiency."},"title":{"value":"SAW-INT4: System-Aware 4-Bit KV-Cache Quantization for Real-World LLM Serving"},"authors":{"value":[{"fullname":"Jinda Jia","username":""},{"fullname":"Jisen Li","username":"~Jisen_Li2"},{"fullname":"Zhongzhu Zhou","username":""},{"fullname":"Jung Hwan Heo","username":""},{"fullname":"Jue Wang","username":""},{"fullname":"Tri Dao","username":""},{"fullname":"Shuaiwen Leon Song","username":""},{"fullname":"Ben Athiwaratkun","username":""},{"fullname":"Chenfeng Xu","username":""},{"fullname":"Tianyi Zhang","username":""},{"fullname":"Xiaoxia Wu","username":""}]}},"tmdate":1785886554877,"pdate":1798675200000,"externalIds":["dblp:journals/corr/abs-2604-19157"],"tcdate":1785886551645,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Jisen_Li2"],"forum":"BTNY9CDtTl","license":"CC BY-SA 4.0","number":119913,"cdate":1775001600000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1785886554877,"domain":"OpenReview.net/Public_Article","id":"BTNY9CDtTl","version":2},{"content":{"venue":{"value":"AAAI 2026"},"pdf":{"value":"https://ojs.aaai.org/index.php/AAAI/article/download/39285/43246"},"venueid":{"value":"dblp.org/conf/AAAI/2026"},"paperhash":{"value":"gu|david_dualstage_adaptive_visiontext_integrated_decoupling_for_multimodal_kv_cache_eviction"},"authorids":{"value":["","","","~Xiangmin_Xu1"]},"html":{"value":"https://doi.org/10.1609/aaai.v40i26.39285"},"_bibtex":{"value":"@inproceedings{DBLP:conf/aaai/GuJGX26,\n author={Yifeng Gu and Jianxiu Jin and Kailing Guo and Xiangmin Xu},\n title={DAVID: Dual-stage Adaptive Vision-text Integrated Decoupling for Multimodal KV Cache Eviction},\n year={2026},\n cdate={1767225600000},\n pages={21387-21395},\n url={https://doi.org/10.1609/aaai.v40i26.39285},\n booktitle={AAAI},\n crossref={conf/aaai/2026}\n}\n"},"abstract":{"value":"With the rapid development of multimodal large language models (MLLMs), deploying them on low-resource devices remains challenging. Beyond the model size, long multimodal inputs cause substantial memory overhead in the KV cache, making efficient cache management critical. In this paper, we propose DAVID, a KV cache eviction strategy that adapts to the degree of modality fusion across layers. By analyzing the feature distributions of vision and text tokens, we observe low fusion in early layers and high fusion in deeper layers. Based on this observation, DAVID adopts a decoupled eviction strategy in shallow layers and a super-modal eviction strategy in deeper layers. To support this dynamic switching, we design a lightweight metric that quantifies cross-modal fusion and uses a threshold to determine which layers require decoupling. Experimental results show that DAVID achieves state-of-the-art performance on multiple benchmarks and offers a new perspective on KV cache eviction for MLLMs."},"title":{"value":"DAVID: Dual-stage Adaptive Vision-text Integrated Decoupling for Multimodal KV Cache Eviction"},"authors":{"value":["Yifeng Gu","Jianxiu Jin","Kailing Guo","Xiangmin Xu"]}},"tmdate":1775737594850,"pdate":1798675200000,"tcdate":1775737592618,"writers":["~"],"signatures":["~Xiangmin_Xu1"],"forum":"1gSI15Hmaa","license":"CC BY-SA 4.0","number":865618,"cdate":1767225600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1775737594850,"domain":"DBLP.org","id":"1gSI15Hmaa","version":2},{"content":{"summary":{"value":"The paper introduces Palu, a KV-Cache compression framework that reduces memory usage in LLM inference by applying low-rank projection to the hidden dimension. Palu decomposes linear layers into low-rank matrices, caching compressed intermediate states and reconstructing KV values on the fly. Key features include medium-grained low-rank decomposition, automated rank search, quantization compatibility enhancements, and a GPU-optimized matrix fusion kernel. Experiments show that Palu compresses KV-Cache by 50% and boosts attention module speed by up to 1.87×."},"soundness":{"value":4},"confidence":{"value":5},"questions":{"value":"Q1: Corresponding to W1, provide theoretical or empirical evidence for the statement in Section 3.1 that \" However, this approach poses significant computational challenges during runtime that make it impractical for deployments \"\n\nQ2: Corresponding to W2, add a theoretical derivation of the algorithm's error bounds.\n\nQ3: Corresponding to W3, include an analysis of the algorithm's time complexity.\n\nQ4: Corresponding to W4, clarify the relationship between \"compression rate\" and \"Avg. Bits\" in Section 4, along with definitions of related metrics.\n\nQ5: It is recommended to open-source the code."},"rating":{"value":6},"details_of_ethics_concerns":{"value":"I believe this paper does not require an ethics review."},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":3},"strengths":{"value":"S1: The method creatively combines SVD algorithms with KV-cache compression, optimizing speed and grouping different attention heads with varied compression rates, which is insightful.\n\nS2: The experiments are comprehensive, measuring both the algorithm’s accuracy and speed.\n\nS3: The figures and tables are clear, and the descriptions are precise."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"W1: In Section 3.1, the statement \"However, this approach poses significant computational challenges during runtime that make it impractical for deployments\" lacks experimental results and theoretical support, even though this claim is foundational to the proposed method.\n\nW2: There is no theoretical proof for the accuracy of the proposed algorithm, such as bounded error analysis.\n\nW3: The time complexity of the algorithm is not analyzed.\n\nW4: In Section 4, the relationship between \"compression rate\" and \"Avg. Bits\" is unclear. Additionally, definitions of metrics such as \"compression rate\" are not specified.\n\nW5: The code is not open-sourced, making reproducibility uncertain."}},"nonreaders":[],"tmdate":1731429114548,"tcdate":1730380107545,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission12412/Reviewer_2i3G"],"signatures":["ICLR.cc/2025/Conference/Submission12412/Reviewer_2i3G"],"forum":"LWMS4pk2vK","number":2,"license":"CC BY 4.0","cdate":1730380107545,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission12412/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731429114548,"domain":"ICLR.cc/2025/Conference","replyto":"LWMS4pk2vK","id":"KtcgPx9TSx","forumContent":{"venue":{"value":"ICLR 2025 Poster"},"TLDR":{"value":"We propose a novel low-rank KV-Cache compression method to reduce the memory footprint and accelerate decoding efficiency"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["KV-Cache","Low-Rank Compression","Large Language Model"]},"supplementary_material":{"value":"/attachment/05d223ca327a41c923d2e670b96806342f64cf22.zip"},"primary_area":{"value":"other topics in machine learning (i.e., none of the above)"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Post-training KV-Cache compression methods typically either sample a subset of effectual tokens or quantize the data into lower numerical bit width. However, these methods cannot exploit redundancy in the hidden dimension of the KV tenors. This paper presents a hidden dimension compression approach called Palu, a KV-Cache compression framework that utilizes low-rank projection to reduce inference-time LLM memory usage. Palu decomposes the linear layers into low-rank matrices, caches compressed intermediate states, and reconstructs the full keys and values on the fly. To improve accuracy, compression rate, and efficiency, Palu further encompasses (1) a medium-grained low-rank decomposition scheme, (2) an efficient rank search algorithm, (3) low-rank-aware quantization compatibility enhancements, and (4) an optimized GPU kernel with matrix fusion. Extensive experiments with popular LLMs show that Palu compresses KV-Cache by 50% while maintaining strong accuracy and delivering up to 1.89× speedup on the RoPE-based attention module. When combined with quantization, Palu’s\ninherent quantization-friendly design yields small to negligible extra accuracy degradation while saving additional memory than quantization-only methods and achieving up to 2.91× speedup for the RoPE-based attention. Moreover, it maintains comparable or even better accuracy (up to 1.19 lower perplexity) compared to quantization-only methods. These results demonstrate Palu’s superior capability to effectively address the efficiency and memory challenges of LLM inference posed by KV-Cache. Our code is publicly available at: https://github.com/shadowpa0327/Palu."},"_bibtex":{"value":"@inproceedings{\nchang2025palu,\ntitle={Palu: {KV}-Cache Compression with Low-Rank Projection},\nauthor={Chi-Chih Chang and Wei-Cheng Lin and Chien-Yu Lin and Chong-Yan Chen and Yu-Fang Hu and Pei-Shuo Wang and Ning-Chi Huang and Luis Ceze and Mohamed S. Abdelfattah and Kai-Chiang Wu},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=LWMS4pk2vK}\n}"},"title":{"value":"Palu: KV-Cache Compression with Low-Rank Projection"},"pdf":{"value":"/pdf/bcb7e2da88d0937fb5299d480da43bbc390f57bc.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"chang|palu_kvcache_compression_with_lowrank_projection"},"authorids":{"value":["~Chi-Chih_Chang1","~Wei-Cheng_Lin2","~Chien-Yu_Lin1","~Chong-Yan_Chen2","~Yu-Fang_Hu1","~Pei-Shuo_Wang1","~Ning-Chi_Huang1","~Luis_Ceze1","~Mohamed_S._Abdelfattah1","~Kai-Chiang_Wu1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Chi-Chih Chang","Wei-Cheng Lin","Chien-Yu Lin","Chong-Yan Chen","Yu-Fang Hu","Pei-Shuo Wang","Ning-Chi Huang","Luis Ceze","Mohamed S. Abdelfattah","Kai-Chiang Wu"]}},"version":2},{"content":{"venue":{"value":"ICLR 2027 Conference Submission"},"TLDR":{"value":"DeSKV combines lightweight approximated routing with deferred token selection to reduce GPU memory usage and accelerate long-context decoding while retaining near-full-cache accuracy."},"keywords":{"value":["KV cache compression","long-context LLM inference","dynamic KV cache selection","KV cache offloading","sparse attention"]},"primary_area":{"value":"infrastructure, software libraries, hardware, systems, etc."},"abstract":{"value":"Long-context generation is limited by KV-cache storage and access. Routing-based offloading reduces GPU memory use by keeping exact K/V in host memory and retrieving only a query-dependent working set. This requires a compact router that preserves token rankings as queries change, while minimizing the routing overhead for latency. This paper presents DeSKV, a low-rank routing method with step-deferred selection. The router is fitted from query and key moments, with more weight on keys; Step deferral selection utilizes the preceding step's token selection while preparing the next working set. On Qwen3-8B, DeSKV achieves a 48.3 average LongBench score at a 256-token prompt-KV budget versus 48.5 with full cache, and reduces decode latency by 33.6\\% relative to full-cache execution at a 97K-token context."},"_bibtex":{"value":"@inproceedings{\nanonymous2026deskv,\ntitle={De{SKV}: Low-Rank Routing and Deferred Selection for {KV}-Cache Offloading},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=OKtrN1LHwH},\nnote={under review}\n}"},"title":{"value":"DeSKV: Low-Rank Routing and Deferred Selection for KV-Cache Offloading"},"pdf":{"value":"/pdf/03c8c95fb7e9c93a0e274c87bd2285fd74276705.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791226467055,"tcdate":1789122214051,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission14178/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission14178/Authors"],"forum":"OKtrN1LHwH","license":"CC BY 4.0","number":14178,"cdate":1789122214051,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/Submission14178/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791226467055,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"OKtrN1LHwH","version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"https://arxiv.org/pdf/2412.08890v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"kim|lexico_extreme_kv_cache_compression_via_sparse_coding_over_universal_dictionaries"},"authorids":{"value":["","https://dblp.org/search/pid/api?q=author:Jongho_Park:","https://dblp.org/search/pid/api?q=author:Jaewoong_Cho:","https://dblp.org/search/pid/api?q=author:Dimitris_Papailiopoulos:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2412.08890"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2412-08890,\n publtype={informal},\n author={Junhyuck Kim and Jongho Park and Jaewoong Cho and Dimitris Papailiopoulos},\n title={Lexico: Extreme KV Cache Compression via Sparse Coding over Universal Dictionaries},\n year={2024},\n cdate={1704067200000},\n journal={CoRR},\n volume={abs/2412.08890},\n url={https://doi.org/10.48550/arXiv.2412.08890}\n}\n"},"abstract":{"value":"We introduce Lexico, a novel KV cache compression method that leverages sparse coding with a universal dictionary. Our key finding is that key-value cache in modern LLMs can be accurately approximated using sparse linear combination from a small, input-agnostic dictionary of ~4k atoms, enabling efficient compression across different input prompts, tasks and models. Using orthogonal matching pursuit for sparse approximation, Lexico achieves flexible compression ratios through direct sparsity control. On GSM8K, across multiple model families (Mistral, Llama 3, Qwen2.5), Lexico maintains 90-95% of the original performance while using only 15-25% of the full KV-cache memory, outperforming both quantization and token eviction methods. Notably, Lexico remains effective in low memory regimes where 2-bit quantization fails, achieving up to 1.7x better compression on LongBench and GSM8K while maintaining high accuracy."},"title":{"value":"Lexico: Extreme KV Cache Compression via Sparse Coding over Universal Dictionaries"},"authors":{"value":["Junhyuck Kim","Jongho Park","Jaewoong Cho","Dimitris Papailiopoulos"]}},"tmdate":1769422554029,"pdate":1735603200000,"externalIds":["dblp:journals/corr/abs-2412-08890"],"tcdate":1769422324125,"writers":["~"],"signatures":["~Junhyuck_Kim1"],"forum":"RJSZFeOiwH","license":"CC BY-SA 4.0","number":802058,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1769422554029,"domain":"DBLP.org","id":"RJSZFeOiwH","version":2},{"content":{"venue":{"value":"ICML 2025 poster"},"keywords":{"value":["transformer","kv cache","compression"]},"_bibtex":{"value":"@inproceedings{\nkim2025lexico,\ntitle={Lexico: Extreme {KV} Cache Compression via Sparse Coding over Universal Dictionaries},\nauthor={Junhyuck Kim and Jongho Park and Jaewoong Cho and Dimitris Papailiopoulos},\nbooktitle={Forty-second International Conference on Machine Learning},\nyear={2025},\nurl={https://openreview.net/forum?id=Yh9vxlxnjA}\n}"},"title":{"value":"Lexico: Extreme KV Cache Compression via Sparse Coding over Universal Dictionaries"},"paperhash":{"value":"kim|lexico_extreme_kv_cache_compression_via_sparse_coding_over_universal_dictionaries"},"TLDR":{"value":"We introduce a novel KV cache compression method that leverages sparse coding with a universal dictionary."},"primary_area":{"value":"deep_learning->large_language_models"},"abstract":{"value":"We introduce Lexico, a novel KV cache compression method that leverages sparse coding with a universal dictionary. Our key finding is that key-value cache in modern LLMs can be accurately approximated using sparse linear combination from a small, input-agnostic dictionary of ~4k atoms, enabling efficient compression across different input prompts, tasks and models. Using orthogonal matching pursuit for sparse approximation, Lexico achieves flexible compression ratios through direct sparsity control. On GSM8K, across multiple model families (Mistral, Llama 3, Qwen2.5), Lexico maintains 90-95% of the original performance while using only 15-25% of the full KV-cache memory, outperforming both quantization and token eviction methods. Notably, Lexico remains effective in low memory regimes where 2-bit quantization fails, achieving up to 1.7x better compression on LongBench and GSM8K while maintaining high accuracy."},"link_to_code":{"value":"https: //github.com/krafton-ai/lexico"},"pdf":{"value":"/pdf/84290473dfb8f3fce5797c405210f5c7829cd325.pdf"},"lay_summary":{"value":"Modern AI language models like ChatGPT need to remember everything you’ve said so far in a conversation. They store this information in what’s called a \"KV cache,\" which takes up a lot of memory—especially as conversations get longer or more complex. This creates a problem when trying to run these models efficiently on devices with limited memory, such as a single GPU.\n\nOur method, Lexico, reduces this memory cost by compressing the stored information using a technique called sparse coding. Instead of storing all the data in full, Lexico breaks it down into a small set of building blocks (like a dictionary of reusable parts) that can represent the original information with high accuracy. These dictionaries are universal, meaning they work across different tasks without needing to be retrained.\n\nEven under tight memory limits, Lexico maintains strong performance—matching the original model’s accuracy 90–95% of the time while using only 15–25% of the memory. It also outperforms popular alternatives like quantization and token deletion in challenging tasks. This makes Lexico a practical solution for running large language models more efficiently, especially when memory is a bottleneck."},"venueid":{"value":"ICML.cc/2025/Conference"},"authorids":{"value":["~Junhyuck_Kim1","~Jongho_Park2","~Jaewoong_Cho1","~Dimitris_Papailiopoulos1"]},"authors":{"value":["Junhyuck Kim","Jongho Park","Jaewoong Cho","Dimitris Papailiopoulos"]}},"tmdate":1753292789960,"pdate":1746105093057,"tcdate":1736492551036,"writers":["ICML.cc/2025/Conference","ICML.cc/2025/Conference/Submission592/Authors"],"signatures":["ICML.cc/2025/Conference/Submission592/Authors"],"forum":"Yh9vxlxnjA","license":"CC BY 4.0","number":592,"cdate":1736492551036,"readers":["everyone"],"invitations":["ICML.cc/2025/Conference/-/Submission","ICML.cc/2025/Conference/-/Post_Submission","ICML.cc/2025/Conference/Submission592/-/Full_Submission","ICML.cc/2025/Conference/-/Edit","ICML.cc/2025/Conference/Submission592/-/Camera_Ready_Revision"],"mdate":1753292789960,"odate":1750230902392,"domain":"ICML.cc/2025/Conference","id":"Yh9vxlxnjA","version":2},{"content":{"TLDR":{"value":"We introduce a novel KV cache compression method that leverages sparse coding with a universal dictionary."},"venue":{"value":"SLLM"},"keywords":{"value":["transformer","kv cache","compression"]},"abstract":{"value":"We introduce Lexico, a novel KV cache compression method that leverages sparse coding with a universal dictionary. Our key finding is that key-value cache in modern LLMs can be accurately approximated using sparse linear combination from a small, input-agnostic dictionary of ~4k atoms, enabling efficient compression across different input prompts, tasks and models. Using orthogonal matching pursuit for sparse approximation, Lexico achieves flexible compression ratios through direct sparsity control. On GSM8K, across multiple model families (Mistral, Llama 3, Qwen2.5), Lexico maintains 90-95% of the original performance while using only 15-25% of the full KV-cache memory, outperforming both quantization and token eviction methods. Notably, Lexico remains effective in low memory regimes where 2-bit quantization fails, achieving up to 1.7x better compression on LongBench and GSM8K while maintaining high accuracy."},"_bibtex":{"value":"@inproceedings{\nkim2025lexico,\ntitle={{LEXICO}: Extreme {KV} Cache Compression via Sparse Coding over Universal Dictionaries},\nauthor={Junhyuck Kim and Jongho Park and Jaewoong Cho and Dimitris Papailiopoulos},\nbooktitle={Sparsity in LLMs (SLLM): Deep Dive into Mixture of Experts, Quantization, Hardware, and Inference},\nyear={2025},\nurl={https://openreview.net/forum?id=p5rLOe9USf}\n}"},"title":{"value":"Lexico: Extreme KV Cache Compression via Sparse Coding over Universal Dictionaries"},"Anonymization":{"value":"This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs."},"pdf":{"value":"/pdf/081639a92f3b96d69baa48dab48c664eb292725d.pdf"},"venueid":{"value":"ICLR.cc/2025/Workshop/SLLM"},"paperhash":{"value":"kim|lexico_extreme_kv_cache_compression_via_sparse_coding_over_universal_dictionaries"},"authorids":{"value":["~Junhyuck_Kim1","~Jongho_Park2","~Jaewoong_Cho1","~Dimitris_Papailiopoulos1"]},"Track":{"value":"long paper (up to 4 pages)"},"authors":{"value":["Junhyuck Kim","Jongho Park","Jaewoong Cho","Dimitris Papailiopoulos"]}},"tmdate":1744632854469,"pdate":1741181472916,"tcdate":1739000297068,"writers":["ICLR.cc/2025/Workshop/SLLM","ICLR.cc/2025/Workshop/SLLM/Submission68/Authors"],"signatures":["ICLR.cc/2025/Workshop/SLLM/Submission68/Authors"],"forum":"p5rLOe9USf","license":"CC BY 4.0","number":68,"cdate":1739000297068,"readers":["everyone"],"invitations":["ICLR.cc/2025/Workshop/SLLM/-/Submission","ICLR.cc/2025/Workshop/SLLM/-/Post_Submission","ICLR.cc/2025/Workshop/SLLM/-/Edit","ICLR.cc/2025/Workshop/SLLM/Submission68/-/Camera_Ready"],"mdate":1744632854469,"odate":1741196736392,"domain":"ICLR.cc/2025/Workshop/SLLM","id":"p5rLOe9USf","version":2},{"content":{"venue":{"value":"ACL (1) 2025"},"pdf":{"value":"https://aclanthology.org/2025.acl-long.631.pdf"},"venueid":{"value":"dblp.org/conf/ACL/2025"},"paperhash":{"value":"su|accurate_kv_cache_quantization_with_outlier_tokens_tracing"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Yi_Su_0006:","~Yuechi_Zhou1","~Quantong_Qiu1","https://dblp.org/search/pid/api?q=author:Juntao_Li:","https://dblp.org/search/pid/api?q=author:Qingrong_Xia:","~Ping_Li16","https://dblp.org/search/pid/api?q=author:Xinyu_Duan:","https://dblp.org/search/pid/api?q=author:Zhefeng_Wang_0001:","https://dblp.org/search/pid/api?q=author:Min_Zhang_0005:"]},"html":{"value":"https://aclanthology.org/2025.acl-long.631/"},"_bibtex":{"value":"@inproceedings{DBLP:conf/acl/0006ZQLXLD0Z25,\n author={Yi Su and Yuechi Zhou and Quantong Qiu and Juntao Li and Qingrong Xia and Ping Li and Xinyu Duan and Zhefeng Wang and Min Zhang},\n title={Accurate KV Cache Quantization with Outlier Tokens Tracing},\n year={2025},\n cdate={1735689600000},\n pages={12895-12915},\n url={https://aclanthology.org/2025.acl-long.631/},\n booktitle={ACL (1)},\n crossref={conf/acl/2025-1}\n}\n"},"abstract":{"value":"The impressive capabilities of Large Language Models (LLMs) come at the cost of substantial computational resources during deployment. While KV Cache can significantly reduce recomputation during inference, it also introduces additional memory overhead. KV Cache quantization presents a promising solution, striking a good balance between memory usage and accuracy. Previous research has shown that the Keys are distributed by channel, while the Values are distributed by token. Consequently, the common practice is to apply channel-wise quantization to the Keys and token-wise quantization to the Values. However, our further investigation reveals that a small subset of unusual tokens exhibit unique characteristics that deviate from this pattern, which can substantially impact quantization accuracy. To address this, we develop a simple yet effective method to identify these tokens accurately during the decoding process and exclude them from quantization as outlier tokens, significantly improving overall accuracy. Extensive experiments show that our method achieves significant accuracy improvements under 2-bit quantization and can deliver a 6.4 times reduction in memory usage and a 2.3 times increase in throughput."},"title":{"value":"Accurate KV Cache Quantization with Outlier Tokens Tracing"},"authors":{"value":["Yi Su","Yuechi Zhou","Quantong Qiu","Juntao Li","Qingrong Xia","Ping Li","Xinyu Duan","Zhefeng Wang","Min Zhang"]}},"tmdate":1787407275173,"pdate":1735689600000,"externalIds":["dblp:conf/acl/0006ZQLXLD0Z25"],"tcdate":1760154065104,"writers":["~"],"signatures":["~Ping_Li16"],"forum":"SxB221Z9YD","license":"CC BY-SA 4.0","number":642689,"cdate":1735689600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1787407275173,"domain":"DBLP.org","id":"SxB221Z9YD","version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"http://arxiv.org/pdf/2505.10938v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"su|accurate_kv_cache_quantization_with_outlier_tokens_tracing"},"authorids":{"value":["~Yi_Su3","~Yuechi_Zhou1","~Quantong_Qiu1","https://dblp.org/search/pid/api?q=author:Juntao_Li:","https://dblp.org/search/pid/api?q=author:Qingrong_Xia:","~Ping_Li16","https://dblp.org/search/pid/api?q=author:Xinyu_Duan:","https://dblp.org/search/pid/api?q=author:Zhe-Feng_Wang_0001:","~Min_Zhang9"]},"html":{"value":"https://doi.org/10.48550/arXiv.2505.10938"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2505-10938,\n publtype={informal},\n author={Yi Su and Yuechi Zhou and Quantong Qiu and Juntao Li and Qingrong Xia and Ping Li and Xinyu Duan and Zhe-Feng Wang and Min Zhang},\n title={Accurate KV Cache Quantization with Outlier Tokens Tracing},\n year={2025},\n month={May},\n cdate={1746057600000},\n journal={CoRR},\n volume={abs/2505.10938},\n url={https://doi.org/10.48550/arXiv.2505.10938}\n}\n"},"abstract":{"value":"The impressive capabilities of Large Language Models (LLMs) come at the cost of substantial computational resources during deployment. While KV Cache can significantly reduce recomputation during inference, it also introduces additional memory overhead. KV Cache quantization presents a promising solution, striking a good balance between memory usage and accuracy. Previous research has shown that the Keys are distributed by channel, while the Values are distributed by token. Consequently, the common practice is to apply channel-wise quantization to the Keys and token-wise quantization to the Values. However, our further investigation reveals that a small subset of unusual tokens exhibit unique characteristics that deviate from this pattern, which can substantially impact quantization accuracy. To address this, we develop a simple yet effective method to identify these tokens accurately during the decoding process and exclude them from quantization as outlier tokens, significantly improving overall accuracy. Extensive experiments show that our method achieves significant accuracy improvements under 2-bit quantization and can deliver a 6.4 times reduction in memory usage and a 2.3 times increase in throughput."},"title":{"value":"Accurate KV Cache Quantization with Outlier Tokens Tracing"},"authors":{"value":["Yi Su","Yuechi Zhou","Quantong Qiu","Juntao Li","Qingrong Xia","Ping Li","Xinyu Duan","Zhe-Feng Wang","Min Zhang"]}},"tmdate":1787407274600,"pdate":1735689600000,"tcdate":1751012400694,"writers":["~"],"signatures":["~Quantong_Qiu1"],"forum":"zwI8KAI2XC","license":"CC BY-SA 4.0","number":565235,"cdate":1746057600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1787407274600,"domain":"DBLP.org","id":"zwI8KAI2XC","version":2},{"content":{"research_area_keywords":{"value":"Large Language Model, KV Cache Compression"},"languages_studied":{"value":"English"},"venue":{"value":"ACL ARR 2025 February Submission"},"_bibtex":{"value":"@inproceedings{\nanonymous2025cokv,\ntitle={Co{KV}: Optimizing {KV} Cache Allocation via Cooperative Game},\nauthor={Anonymous},\nbooktitle={Submitted to ACL Rolling Review - February 2025},\nyear={2025},\nurl={https://openreview.net/forum?id=erVsTXKapY},\nnote={under review}\n}"},"title":{"value":"CoKV: Optimizing KV Cache Allocation via Cooperative Game"},"contribution_types":{"value":["Approaches to low-resource settings","Approaches low compute settings-efficiency"]},"abstract":{"value":"Large language models (LLMs) have achieved remarkable success on various aspects of human life. However, one of the major challenges in deploying these models is the substantial memory consumption required to store key-value pairs (KV), which imposes significant resource demands. Recent research has focused on KV cache budget allocation, with several approaches proposing head-level budget distribution by evaluating the importance of individual attention heads. These methods, however, assess the importance of heads independently, overlooking their cooperative contributions within the model, which may result in a deviation from their true impact on model performance. In light of this limitation, we propose CoKV, a novel method that models the cooperation between heads in model inference as a cooperative game. By evaluating the contribution of each head within the cooperative game, CoKV can allocate the cache budget more effectively. Extensive experiments show that CoKV achieves state-of-the-art performance on the LongBench benchmark using LLama 3-8B-Instruct and Mistral-7B models. Code is provided in \\url{https://anonymous.4open.science/r/CoKV-40AC}."},"paper_type":{"value":"Long"},"pdf":{"value":"/pdf/94b1a3c08407f45a96bfa5658c56c357d4195f3e.pdf"},"research_area":{"value":"Efficient/Low-Resource Methods for NLP"},"venueid":{"value":"aclweb.org/ACL/ARR/2025/February/Submission"}},"tmdate":1782962059699,"tcdate":1739687688244,"writers":["aclweb.org/ACL/ARR/2025/February","aclweb.org/ACL/ARR/2025/February/Submission5741/Authors"],"signatures":["aclweb.org/ACL/ARR/2025/February/Submission5741/Authors"],"forum":"erVsTXKapY","license":"CC BY 4.0","number":5741,"cdate":1739687688244,"readers":["everyone"],"invitations":["aclweb.org/ACL/ARR/2025/February/-/Submission","aclweb.org/ACL/ARR/2025/February/-/Edit","aclweb.org/ACL/ARR/2025/February/-/Post_Submission","aclweb.org/ACL/ARR/2025/February/Submission5741/-/Change_Reviewer_Nomination","aclweb.org/ACL/ARR/2025/February/-/Preprint_Release_Submission","aclweb.org/ACL/ARR/2025/February/-/Preprint_Post_Submission"],"mdate":1782962059699,"odate":1746821355472,"domain":"aclweb.org/ACL/ARR/2025/February","id":"erVsTXKapY","version":2},{"content":{"TLDR":{"value":"RelaxKV jointly allocates selective cache repair over repair targets and query-relevant recomputation context, improving the quality--TTFT trade-off for cross-request KV reuse, especially at long context lengths."},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["Large Language Model","Sparse Attention","Position-Independent Caching"]},"supplementary_material":{"value":"/attachment/4e01e48d91c1e4b4cce8193c4079531f3976f0d0.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Cross-request KV caching reduces the prefill cost of Retrieval-Augmented Generation (RAG), but conventional prefix caching severely limits cache reuse across requests. Position-Independent Caching (PIC) removes this constraint by reusing independent chunks, but their KV states miss cross-chunk interactions. Existing methods selectively recompute token states to recover these missing interactions, but primarily allocate the recomputation budget to selecting which states to recompute, while fixing the recomputation context to the full causal prefix. We introduce \\textbf{RelaxKV}, which formulates selective cache repair as a joint allocation problem over repair targets and recomputation context. Guided by the user query, RelaxKV identifies layer-specific repair targets and restricts their recomputation to a query-relevant context, reducing attention computation. Across four decoder models, RelaxKV at a 15\\% anchor ratio improves aggregate LongBench performance over ProphetKV on all models. On Qwen3-14B, RelaxKV provides a stronger quality–TTFT trade-off than ProphetKV across a 5\\%–30\\% anchor-ratio sweep, and achieves the best selective results on RULER-MV and LV-Eval at 16K and 32K context lengths. Controlled ablations further demonstrate the importance of recomputation context selection. \nOur code will be released upon acceptance."},"_bibtex":{"value":"@inproceedings{\nanonymous2026relaxkv,\ntitle={Relax{KV}: Query-Aware Recomputation with Sparse Context Attention for Efficient {KV} Cache Reuse},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=gSzB27MDTc},\nnote={under review}\n}"},"title":{"value":"RelaxKV: Query-Aware Recomputation with Sparse Context Attention for Efficient KV Cache Reuse"},"pdf":{"value":"/pdf/336b5ca50b45115ae96ed701f2a5bf4a3c20e2b1.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791224297382,"tcdate":1788767877262,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission10492/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission10492/Authors"],"forum":"gSzB27MDTc","license":"CC BY 4.0","number":10492,"cdate":1788767877262,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/Submission10492/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791224297382,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"gSzB27MDTc","version":2},{"content":{"venue":{"value":"ICLR 2026 Workshop LLM Reasoning"},"keywords":{"value":["Machine Learning","Caching"]},"Funding":{"value":"No, the presenting author of this submission does *not* fall under ICLR’s funding aims, or has sufficient alternate funding."},"abstract":{"value":"Large language models often produce inconsistent answers across multiple related questions when earlier premises are partially forgotten or distorted in long contexts. We argue this is not only a modeling issue but a working-memory issue: KV cache policy controls which premises remain accessible for attention and thus mediates logical consistency under finite memory. Current practice sits at two extremes: retain everything (wasteful) or evict uniformly (premise-destructive). This ignores decades of memory-hierarchy results on working sets and locality. We synthesize empirical evidence that attention working sets are sparse and structurally constrained (heavy hitters, attention sinks, layer heterogeneity), implying that premise-preserving retention is achievable. We provide a small proof-of-concept cache manager with content-aware retention and show favorable memory–quality tradeoffs on a premise-retrieval stress test (passkey retrieval). We then propose a “consistency bundle” evaluation protocol for measuring cross-question contra-\ndictions as a function of memory policy. Our conclusion is practical: memory policies should be designed and reported as reasoning controls, not just serving optimizations."},"_bibtex":{"value":"@inproceedings{\natri2026kv,\ntitle={{KV} Cache as a Reasoning Primitive for Long Context Reasoning},\nauthor={Rian Atri},\nbooktitle={ICLR 2026 Workshop on Logical Reasoning of Large Language Models},\nyear={2026},\nurl={https://openreview.net/forum?id=vs2qwVfU2C}\n}"},"title":{"value":"KV Cache as a Reasoning Primitive for Long Context Reasoning"},"Anonymization":{"value":"This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs."},"pdf":{"value":"/pdf/97be7e9f86697f753871470d243cbb3f9738e04f.pdf"},"venueid":{"value":"ICLR.cc/2026/Workshop/LLM_Reasoning"},"paperhash":{"value":"atri|kv_cache_as_a_reasoning_primitive_for_long_context_reasoning"},"authorids":{"value":["~Rian_Atri1"]},"Track":{"value":"tiny / short paper (up to 4 pages)"},"authors":{"value":["Rian Atri"]}},"tmdate":1777096991126,"pdate":1772712938637,"tcdate":1771821634123,"writers":["ICLR.cc/2026/Workshop/LLM_Reasoning","ICLR.cc/2026/Workshop/LLM_Reasoning/Submission150/Authors"],"signatures":["ICLR.cc/2026/Workshop/LLM_Reasoning/Submission150/Authors"],"forum":"vs2qwVfU2C","license":"CC BY 4.0","number":150,"cdate":1771821634123,"readers":["everyone"],"invitations":["ICLR.cc/2026/Workshop/LLM_Reasoning/-/Submission","ICLR.cc/2026/Workshop/LLM_Reasoning/-/Post_Submission","ICLR.cc/2026/Workshop/LLM_Reasoning/-/Edit","ICLR.cc/2026/Workshop/LLM_Reasoning/Submission150/-/Revision"],"mdate":1777096991126,"odate":1772711556419,"domain":"ICLR.cc/2026/Workshop/LLM_Reasoning","id":"vs2qwVfU2C","version":2},{"content":{"summary":{"value":"This work introduces a dynamic layer-skipping framework that enables each input to traverse different Transformer blocks while satisfying a given budget constraint. Given an input sequence, the KV vectors from the first layer are extracted to compute a global context, which is then fed into a decision module to determine the most important layers to execute. During the decoding phase, the global context is updated using the latest token’s KV cache, allowing the decision module to refresh the depth path dynamically. When no user-specified budget is provided, a budget predictor is trained to estimate an appropriate number of layers based on the input’s KV cache. Experiments are conducted on the LLaMA family."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"Please refer to the Weakness section."},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"This work effectively analyzes the varying importance of Transformer blocks across different inputs and throughout the decoding process. It presents a principled and lightweight approach to dynamic layer selection and budget-aware inference."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- While I find the motivation of this work compelling, I am concerned about whether batch processing can be efficiently realized for higher throughput, particularly when different samples (e.g., sample 1 and sample 2) have distinct depth paths. This may also lead to inefficient parallelism and increased complexity in cache management.\n- The main results in Table 2 do not appear remarkably superior. The performance gap between ShortGPT (static pruning) and the proposed method seems rather marginal.\n- It appears that in the Decision Module, the model-predicted scores from the lightweight MLP are combined with the prior scores. However, it was difficult to understand whether the MLP in the Decision Module itself is trained through any learning process. In contrast, the Budget Predictor is clearly described as being trained using GRPO.\n- I believe Table 2 reports results under specified budgets, and I could not find corresponding results or analysis for the Budget Predictor.\n- The experimental validation seems relatively weak. It would strengthen the work to demonstrate the applicability of the proposed method on other popular LLMs such as Qwen or Phi, which would further support the generality and effectiveness of the approach.\n- Minor\n * Softmsax in Eqn (5), 5p -> Softmax\n * I was not able to find which specific LLaMA model was used for the analysis in Figures 1 and 2."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1764359082280,"tcdate":1761554134695,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission5872/Reviewer_rpEZ"],"signatures":["ICLR.cc/2026/Conference/Submission5872/Reviewer_rpEZ"],"forum":"a4sxpU37LJ","number":3,"license":"CC BY 4.0","cdate":1761554134695,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission5872/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1764359082280,"domain":"ICLR.cc/2026/Conference","replyto":"a4sxpU37LJ","id":"OzxNx6CX6j","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Model Compression","Dynamic Inference"]},"supplementary_material":{"value":"/attachment/f728ce2ed4d2ceee487c102604f0d5e73048ba46.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Large language models require substantial computational resources for inference due to their massive number of parameters. Model layer pruning accelerates inference by eliminating redundant layers. However, existing layer pruning methods fail to meet users' flexible budget constraints and lack the ability to adaptively adjust the inference path. To address these issues, we propose Buddy, a budget-driven and adaptive inference framework. Specifically, we design a Decision Module that adaptively selects important layers to execute based on user input while satisfying a given budget constraint. Additionally, Buddy reuses the KV cache from the first layer and dynamically updates the context during inference, enabling adaptive adjustments to the inference path based on evolving contextual information. Furthermore, when no explicit budget is provided, a Budget Predictor automatically determines an appropriate inference cost to achieve an optimal trade-off between performance and computational efficiency. Extensive experiments on the Llama model demonstrate that Buddy consistently outperforms baseline methods under various pruning configurations."},"_bibtex":{"value":"@misc{\nzhou2026buddy,\ntitle={{BUDDY}: {BU}dget-Driven {DY}namic Depth Routing for Adaptive Large Language Models Inference},\nauthor={Yuhua Zhou and Shaoqi Yu and Shichao Weng and Changhai Zhou and Mingze Yin and Fei Yang and Aimin PAN},\nyear={2026},\nurl={https://openreview.net/forum?id=a4sxpU37LJ}\n}"},"title":{"value":"BUDDY: BUdget-Driven DYnamic Depth Routing for Adaptive Large Language Models Inference"},"pdf":{"value":"/pdf/5bdd979948fada865f3ca9971924cf0c92451f15.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"zhou|buddy_budgetdriven_dynamic_depth_routing_for_adaptive_large_language_models_inference"},"authorids":{"value":["~Yuhua_Zhou1","~Shaoqi_Yu1","~Shichao_Weng1","~Changhai_Zhou2","~Mingze_Yin1","~Fei_Yang3","~Aimin_PAN2"]},"authors":{"value":["Yuhua Zhou","Shaoqi Yu","Shichao Weng","Changhai Zhou","Mingze Yin","Fei Yang","Aimin PAN"]}},"version":2},{"content":{"TLDR":{"value":"Learned KV compression preserves document utility but can degrade an LLM on unrelated queries; we expose this hidden cost and show that relevance routing and off-document training data can mitigate it."},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["KV cache compression","Long-context LLMs","Context interference","Capability preservation","Learned memories"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Serving long documents to a Large Language Model (LLM) repeatedly is expensive: computations grow with context length, and the memory footprint of the key-value (KV) cache balloons.\n*Compressed* KV (CKV) representations aim to mimic the cache of a document and are typically computed once and for all, ahead of inference time.\nMethods to obtain CKVs range from drop mechanisms that reduce their number of columns, to learned approaches. Among the latter, *Cartridges* have emerged as a leading compression method, learning compact KV representations through distillation on relevant Q/A pairs.\nWhile existing evaluations focus primarily on whether Cartridges and other CKVs yield approximately similar responses to document-related, *on-context* queries, we investigate the crucial deployment question of whether they can handle *off-context* queries, something the native KV representation is particularly good at, thanks to the mechanics of attention.\nWe observe a fundamental trade-off: while Cartridges perform better for on-context queries, heuristic-variants preserve better the original LLM's ability to operate off-context. We measure this through their capability to avoid context contamination in their response, retain general knowledge, and follow instructions.\nWe propose *Cartridges++*, simple modifications to cartridges that retain off-context abilities at small or negligible cost. The *router* variant decides at inference time whether the query should use the learned long-context memory, while the *data-mixing* variant allocates a small fraction of training Q/As to queries outside the reference long document.\nOur study shows that assessing CKVs on document utility alone can mask substantial degradation in broader model capabilities, yet those issues can be fixed with benign changes to CKV inference or training."},"_bibtex":{"value":"@inproceedings{\nanonymous2026cartridges,\ntitle={Cartridges++: {KV} Cache Compression without Off-Context Derailment},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=GaKi2aZXvs},\nnote={under review}\n}"},"title":{"value":"Cartridges++: KV Cache Compression without Off-Context Derailment"},"pdf":{"value":"/pdf/7561f39d7d4b5440551f46d601ebe7e1a3c585f2.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791232925506,"tcdate":1789724922452,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission40756/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission40756/Authors"],"forum":"GaKi2aZXvs","license":"CC BY 4.0","number":40756,"cdate":1789724922452,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Edit","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission40756/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing"],"mdate":1791232925506,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"GaKi2aZXvs","version":2},{"content":{"summary":{"value":"The paper propose a pipeline for both training and inference named FIRST. The proposed framework includes layer-specific routers which are input-dependent selectors that can decide which layers are worth for inference. The proposed framework is compatible to the KV cache and batch decoding, which means there is no additional computation needed for handling both processes.\nThe paper also import LoRA method for further improvement of the model performance and selection strategy.\nThe experimental results on Machine Translation and Summarization tasks show the effectiveness of the proposed method."},"soundness":{"value":2},"confidence":{"value":5},"questions":{"value":"1. How about the model performance on other normal tasks and datasets including WikiText2, C4, and other common sense reasoning datasets including BoolQ, PIQA, HellaSwag, WinoGrande, ARC-easy, ARC-challenge, and OpenbookQA?\n2. How about the model performance of other layer pruning LLM works [1] [2] [3] ?\n3. What is the motivation of adopting this framework for Translation and Summarization tasks?\n\n\n\n[1] Compressing Large Language Models by Streamlining the Unimportant Layer \\\n[2] Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methods \\\n[3] Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning"},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. This paper proposes the router for the layer selection, which allows the fine-tuning to optimize the selection strategy.\n2. The LoRA is adopted for further optimization, which shows helpful for the selection strategy optimization."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The paper novelty is limited. The layerly redundancy has been explored by a lot of works [1] [2] [3], while none of those works were introduced for comparison in the results tables.\n2. The proposed routers require additional parameters and corresponding training process, which is not kind of efficient and effective compared to those works [1] [2] [3] that do not require additional parameters and training. Meanwhile, the router selection has been adopted for DiT models in work [4].\n3. The method proposed by the paper is kind of general method, while the paper only focuses on the Machine Translation and Summarization tasks, which hurts the motivation of this method.\n4. According to the Table 1, Table 2, Table 3, the LoRA method seems to have strong improvement for the proposed method, while the ablation study of LoRA only results is absence, which may cause the misunderstanding that the effectiveness is brought by the LoRA rather than the proposed method.\n\n\n[1] Compressing Large Language Models by Streamlining the Unimportant Layer \\\n[2] Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methods \\\n[3] Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning \\\n[4] Learning-to-Cache: Accelerating Diffusion Transformer via Layer Caching"}},"nonreaders":[],"tmdate":1731427855295,"tcdate":1730399625167,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission13480/Reviewer_D1SP"],"signatures":["ICLR.cc/2025/Conference/Submission13480/Reviewer_D1SP"],"forum":"ulGwcj1egv","number":1,"license":"CC BY 4.0","cdate":1730399625167,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission13480/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427855295,"domain":"ICLR.cc/2025/Conference","replyto":"ulGwcj1egv","id":"Jc4xngb5ho","forumContent":{"TLDR":{"value":"We propose FiRST, an algorithm that reduces inference latency in Large Language Models by adaptively selecting transformer layers retaining performance and compatibility with KV-caching"},"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Input-Adaptive Layer Selection; Resource-Constrained Environments; Latency Reduction; Finetuning"]},"primary_area":{"value":"generative models"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Auto-regressive Large Language Models (LLMs) demonstrate remarkable performance across domanins such as vision and language processing. However, due to sequential processing through a stack of transformer layers, autoregressive decoding faces significant computation/latency challenges, particularly in resource-constrained environments like mobile and edge devices. Existing approaches in literature that aim to improve latency via skipping layers have two distinct flavors - 1) Early exit 2) Input-agnostic heuristics where tokens exit at pre-determined layers irrespective of input sequence. Both the above strategies have limitations - the former cannot be applied to handle KV Caching necessary for speed-ups in modern framework and the latter does not capture the variation in layer importance across tasks or more generally, across input sequences. To address both limitations, we propose \\textsc{FiRST}, an algorithm that reduces inference latency by using layer-specific routers to select a subset of transformer layers adaptively for each input sequence - the prompt (during prefill stage) decides which layers will be skipped during decoding. \\textsc{FiRST} preserves compatibility with KV caching enabling faster inference while being quality-aware. \\textsc{FiRST} is model-agnostic and can be easily enabled on any pre-trained LLM. We further improve performance by incorporating LoRA adapters for fine-tuning on external datasets, enhancing task-specific accuracy while maintaining latency benefits. Our approach reveals that input adaptivity is critical - indeed, different task-specific middle layers play a crucial role in evolving hidden representations depending on task. Extensive experiments show that \\textsc{FiRST} significantly reduces latency while retaining competitive performance (as compared to baselines), making our approach an efficient solution for LLM deployment in low-resource environments."},"_bibtex":{"value":"@misc{\njain2024first,\ntitle={Fi{RST}: Finetuning Router-Selective Transformers for Input-Adaptive Latency Reduction},\nauthor={Akriti Jain and Saransh Sharma and Koyel Mukherjee and Soumyabrata Pal},\nyear={2024},\nurl={https://openreview.net/forum?id=ulGwcj1egv}\n}"},"title":{"value":"FiRST: Finetuning Router-Selective Transformers for Input-Adaptive Latency Reduction"},"pdf":{"value":"/pdf/3d0953a1b0b51cd3b8d84b9db6cb34aa5839ddb3.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"jain|first_finetuning_routerselective_transformers_for_inputadaptive_latency_reduction"},"authorids":{"value":["~Akriti_Jain1","~Saransh_Sharma1","~Koyel_Mukherjee2","~Soumyabrata_Pal1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Akriti Jain","Saransh Sharma","Koyel Mukherjee","Soumyabrata Pal"]}},"version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2504.11816v1"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"kim|costefficient_llm_serving_in_the_cloud_vm_selection_with_kv_cache_offloading"},"html":{"value":"https://doi.org/10.48550/arXiv.2504.11816"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2504-11816,\n publtype={informal},\n author={Kihyun Kim and Jinwoo Kim and Hyunsun Chung and Myung-Hoon Cha and Hong-Yeon Kim and Youngjae Kim},\n title={Cost-Efficient LLM Serving in the Cloud: VM Selection with KV Cache Offloading},\n year={2025},\n month={April},\n cdate={1743465600000},\n journal={CoRR},\n volume={abs/2504.11816},\n url={https://doi.org/10.48550/arXiv.2504.11816}\n}\n"},"abstract":{"value":"LLM inference is essential for applications like text summarization, translation, and data analysis, but the high cost of GPU instances from Cloud Service Providers (CSPs) like AWS is a major burden. This paper proposes InferSave, a cost-efficient VM selection framework for cloud based LLM inference. InferSave optimizes KV cache offloading based on Service Level Objectives (SLOs) and workload charac teristics, estimating GPU memory needs, and recommending cost-effective VM instances. Additionally, the Compute Time Calibration Function (CTCF) improves instance selection accuracy by adjusting for discrepancies between theoretical and actual GPU performance. Experiments on AWS GPU instances show that selecting lower-cost instances without KV cache offloading improves cost efficiency by up to 73.7% for online workloads, while KV cache offloading saves up to 20.19% for offline workloads."},"title":{"value":"Cost-Efficient LLM Serving in the Cloud: VM Selection with KV Cache Offloading"},"authors":{"value":[{"fullname":"Kihyun Kim","username":""},{"fullname":"Jinwoo Kim","username":""},{"fullname":"Hyunsun Chung","username":""},{"fullname":"Myung-Hoon Cha","username":""},{"fullname":"Hong-Yeon Kim","username":""},{"fullname":"Youngjae Kim","username":"~Youngjae_Kim3"}]}},"tmdate":1785943993649,"pdate":1767139200000,"externalIds":["dblp:journals/corr/abs-2504-11816"],"tcdate":1785943978910,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Youngjae_Kim3"],"forum":"D1uR8WiHWF","license":"CC BY-SA 4.0","number":123188,"cdate":1743465600000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1785943993649,"domain":"OpenReview.net/Public_Article","id":"D1uR8WiHWF","version":2},{"content":{"summary":{"value":"The authors present ReKV (Retrieve In-context Video KV-Cache) for streaming video question-answering. The authors incorporate a sliding-window attention mechanism on existing VideoLLMs, introduce a retrieval method that leverages an external retriever or the parameters within Video-LLMs to retrieve only queryrelevant KV-Caches. The authors evaluates the model on both long video QA and streaming videoqa."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"All weakness, and:\n\n1. The citation format is inconsistent over the paper, the authors should unify this format. \n\n2. Since the model is claimed to integrate seamlessly with existing Video-LLMs, is it possible to apply to a bigger VideoLLM backbone, like around 70B scale? \n\n3. In Table 4, for offline video question-answering, could the authors elaborate more on the baseline setting of the LLaVA-ov model, like frame numbers? Also, could the authors compare the efficiency of the proposed ReKV compared to the original LLaVA-ov model in the table?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. The number of evaluation benchmark for the proposed method is adequate. \n\n2. The improvements against standard VideoLLMs are substantial."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Lack of fair comparison against existing memory-based models (including VideoStreaming and Flash-VStream). It would be better if the author could provide results for ReKV and previous memory-based models under the same VideoLLM backbone to show the effectiveness of the proposed method, for both long video benchmarks and streaming video benchmarks. \n\n2. Missing related works. The authors should discuss the novel contribution compared to the paper VideoLLM-online[1], MC-ViT[2]. \n\n3. The “External Video KV-Cache Retrieval” is confusing, do the authors mean selecting the keyframes using the query information via the CLIP-based models (like a cross-modal matching)? This is already investigated by a number of works, including ATP [3], SeViLA[4] and so on. It would be if for the authors to clarify how \"External Video KV-Cache Retrieval\" differs from or improves upon the keyframe selection methods. \n\n[1] VideoLLM-online: Online Video Large Language Model for Streaming Video\n\n[2] Memory Consolidation Enables Long-Context Video Understanding\n\n[3] Revisiting the \"Video\" in Video-Language Understanding\n\n[4] Self-Chained Image-Language Model for Video Localization and Question Answering"}},"nonreaders":[],"tmdate":1732730450084,"tcdate":1730491331640,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission861/Reviewer_g3YS"],"signatures":["ICLR.cc/2025/Conference/Submission861/Reviewer_g3YS"],"forum":"8g9fs6mdEG","number":2,"license":"CC BY 4.0","cdate":1730491331640,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission861/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1732730450084,"domain":"ICLR.cc/2025/Conference","replyto":"8g9fs6mdEG","id":"EABgkTuzVt","forumContent":{"venue":{"value":"ICLR 2025 Poster"},"TLDR":{"value":"We propose a training-free approach that integrates with existing Video-LLMs for streaming video question-answering."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Video Understanding","Multimodal Large Language Models","Streaming Video Question-answering"]},"supplementary_material":{"value":"/attachment/1ccf8ba392ddeea6d8f0843d1d4a65497e295ce6.pdf"},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"We propose ReKV, a novel training-free approach that enables efficient streaming video question-answering (StreamingVQA), by seamlessly integrating with existing Video Large Language Models (Video-LLMs). Traditional VideoQA systems struggle with long videos, as they must process entire videos before responding to queries, and repeat this process for each new question. In contrast, our approach analyzes long videos in a streaming manner, allowing for prompt responses as soon as user queries are received. Building on a common Video-LLM, we first incorporate a sliding-window attention mechanism, ensuring that input frames attend to a limited number of preceding frames, thereby reducing computational overhead. To prevent information loss, we store processed video key-value caches (KV-Caches) in RAM and disk, reloading them into GPU memory as needed. Additionally, we introduce a retrieval method that leverages an external retriever or the parameters within Video-LLMs to retrieve only query-relevant KV-Caches, ensuring both efficiency and accuracy in question answering. ReKV enables the separation of video analyzing and question-answering across different processes and GPUs, significantly enhancing the efficiency of StreamingVQA. Through comprehensive experimentation, we validate the efficacy and practicality of our approach, which significantly boosts efficiency and enhances applicability over existing VideoQA models."},"_bibtex":{"value":"@inproceedings{\ndi2025streaming,\ntitle={Streaming Video Question-Answering with In-context Video {KV}-Cache Retrieval},\nauthor={Shangzhe Di and Zhelun Yu and Guanghao Zhang and Haoyuan Li and TaoZhong and Hao Cheng and Bolin Li and Wanggui He and Fangxun Shu and Hao Jiang},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=8g9fs6mdEG}\n}"},"title":{"value":"Streaming Video Question-Answering with In-context Video KV-Cache Retrieval"},"pdf":{"value":"/pdf/e491e49e823ff5b84c75cc8daad0e1b80ebfd206.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"di|streaming_video_questionanswering_with_incontext_video_kvcache_retrieval"},"authorids":{"value":["~Shangzhe_Di1","~Zhelun_Yu1","~Guanghao_Zhang1","~Haoyuan_Li1","~TaoZhong1","~Hao_Cheng9","~Bolin_Li2","~Wanggui_He1","~Fangxun_Shu1","~Hao_Jiang13"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Shangzhe Di","Zhelun Yu","Guanghao Zhang","Haoyuan Li","TaoZhong","Hao Cheng","Bolin Li","Wanggui He","Fangxun Shu","Hao Jiang"]}},"version":2},{"content":{"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["Quantization","KV Cache","LLM Inference"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"The growing key--value (KV) cache limits long-context large language model (LLM) inference through its memory footprint and bandwidth demand. Quantization reduces these costs, but attention typically still requires dequantizing or unpacking the stored KV states into wider numerical operands for computation, introducing conversion overhead and limiting the computational benefits of low-bit storage.\nTo bridge this gap between compact storage and computation, we present \\textbf{PiCoKV}, a training-free, data-aware KV-cache quantization method for dequantization-free attention on packed binary codes. PiCoKV factorizes keys and values into binary codes and compact decoding matrices. By folding these matrices into attention's query and output paths, PiCoKV performs both query--key scoring and value aggregation directly on packed codes through word-level bitwise operations, processing multiple binary entries in parallel without unpacking them into wider numerical operands.\nEvaluations on long-context benchmarks for Llama-3.1-8B-Instruct and Qwen3-8B show that PiCoKV achieves accuracy comparable to full attention and outperforms low-bit quantization baselines. \nThe combination of compact storage and direct computation yields practical efficiency gains: PiCoKV achieves higher end-to-end decoding throughput than the baselines, reaching $3.45\\times$ that of FlashAttention-2 at a 256K context while compressing the KV cache by up to $6.3\\times$. Our code is available at https://anonymous.4open.science/r/PiCoKV-BFC1/."},"_bibtex":{"value":"@inproceedings{\nanonymous2026picokv,\ntitle={PiCo{KV}: Dequantization-Free {KV} Cache Quantization via Binary Principal-Component Coding},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=9anJkPeyEc},\nnote={under review}\n}"},"title":{"value":"PiCoKV: Dequantization-Free KV Cache Quantization via Binary Principal-Component Coding"},"pdf":{"value":"/pdf/57b7f04dc7b9a4fc55f01a1d59c62acc8f2fa0ef.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791223579332,"tcdate":1788567684692,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission9360/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission9360/Authors"],"forum":"9anJkPeyEc","license":"CC BY 4.0","number":9360,"cdate":1788567684692,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission9360/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791223579332,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"9anJkPeyEc","version":2},{"content":{"TLDR":{"value":"Why keep 1% of a 256K KV cache? Budgets depend on how many tokens outrank the evidence, not on length: the need stays at 24-28 tokens from 16K to 256K, yet grows 220x when distractors are added at fixed length."},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["KV cache compression","KV cache eviction","long-context inference","efficient LLM inference","memory budget","token retention","evaluation methodology"]},"supplementary_material":{"value":"/attachment/cc83ef7236958fd791167037fc31084186b7317d.zip"},"primary_area":{"value":"infrastructure, software libraries, hardware, systems, etc."},"abstract":{"value":"Every KV-cache eviction method must decide how many tokens to keep, and practice sets that budget as a fraction of the prompt. We show that the fraction is the wrong unit. The budget is set by competition: evidence survives eviction exactly when the budget covers the evidence plus the tokens the scorer ranks above it, and prompt length matters only through those competitors. Prior work misses this for two reasons. It evaluates budgets on prompts where length and competing content grow together, so their effects cannot be separated, and it treats keeping the evidence as enough to answer. Separating length from competition exposes costly errors in both directions: on long prompts with little competing content, a fixed fraction keeps about 100 times more cache than needed, while at a fixed length, added distractors raise the required budget more than 200-fold. Counts frozen before verification forecast budgets within a factor of two at up to 512K tokens, where a constant fraction fails. Keeping the evidence is also not enough: answers depend on keeping the rows that bind the evidence to the question. Compacting the cache to these budgets cuts KV memory by up to two orders of magnitude with no loss of accuracy on the tested requests. The rule for sizing a cache is simple: count competitors, not context."},"_bibtex":{"value":"@inproceedings{\nanonymous2026count,\ntitle={Count Competitors, Not Context: When {KV}-Cache Budgets Transfer},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=NMAQl4WMdo},\nnote={under review}\n}"},"title":{"value":"Count Competitors, Not Context: When KV-Cache Budgets Transfer"},"pdf":{"value":"/pdf/eb0dc3c2b4f5581fb3eac568c484d87fd189b345.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791234913286,"tcdate":1789793770293,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission54439/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission54439/Authors"],"forum":"NMAQl4WMdo","license":"CC BY 4.0","number":54439,"cdate":1789793770293,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Edit","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission54439/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing"],"mdate":1791234913286,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"NMAQl4WMdo","version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2508.15212v3"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"liao|spark_queryaware_unstructured_sparsity_with_recoverable_kv_cache_channel_pruning"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Huanxuan_Liao:","~Yixing_Xu2","https://dblp.org/search/pid/api?q=author:Shizhu_He:","https://dblp.org/search/pid/api?q=author:Guanchen_Li:","https://dblp.org/search/pid/api?q=author:Xuanwu_Yin:","https://dblp.org/search/pid/api?q=author:Dong_Li_0025:","https://dblp.org/search/pid/api?q=author:Emad_Barsoum:","https://dblp.org/search/pid/api?q=author:Jun_Zhao:","https://dblp.org/search/pid/api?q=author:Kang_Liu:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2508.15212"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2508-15212,\n publtype={informal},\n author={Huanxuan Liao and Yixing Xu and Shizhu He and Guanchen Li and Xuanwu Yin and Dong Li and Emad Barsoum and Jun Zhao and Kang Liu},\n title={SparK: Query-Aware Unstructured Sparsity with Recoverable KV Cache Channel Pruning},\n year={2025},\n month={August},\n cdate={1754006400000},\n journal={CoRR},\n volume={abs/2508.15212},\n url={https://doi.org/10.48550/arXiv.2508.15212}\n}\n"},"abstract":{"value":"Long-context inference in large language models (LLMs) is increasingly constrained by the KV cache bottleneck: memory usage grows linearly with sequence length, while attention computation scales quadratically. Existing approaches address this issue by compressing the KV cache along the temporal axis through strategies such as token eviction or merging to reduce memory and computational overhead. However, these methods often neglect fine-grained importance variations across feature dimensions (i.e., the channel axis), thereby limiting their ability to effectively balance efficiency and model accuracy. In reality, we observe that channel saliency varies dramatically across both queries and positions: certain feature channels carry near-zero information for a given query, while others spike in relevance. To address this oversight, we propose SPARK, a training-free plug-and-play method that applies unstructured sparsity by pruning KV at the channel level, while dynamically restoring the pruned entries during attention score computation. Notably, our approach is orthogonal to existing KV compression and quantization techniques, making it compatible for integration with them to achieve further acceleration. By reducing channel-level redundancy, SPARK enables processing of longer sequences within the same memory budget. For sequences of equal length, SPARK not only preserves or improves model accuracy but also reduces KV cache storage by over 30% compared to eviction-based methods. Furthermore, even with an aggressive pruning ratio of 80%, SPARK maintains performance with less degradation than 5% compared to the baseline eviction method, demonstrating its robustness and effectiveness. Our code will be available at https://github.com/Xnhyacinth/SparK."},"title":{"value":"SparK: Query-Aware Unstructured Sparsity with Recoverable KV Cache Channel Pruning"},"authors":{"value":["Huanxuan Liao","Yixing Xu","Shizhu He","Guanchen Li","Xuanwu Yin","Dong Li","Emad Barsoum","Jun Zhao","Kang Liu"]}},"tmdate":1767785442098,"pdate":1735689600000,"externalIds":["dblp:journals/corr/abs-2508-15212"],"tcdate":1767785416723,"writers":["~"],"signatures":["~Yixing_Xu2"],"forum":"MWHVcgzD9l","license":"CC BY-SA 4.0","number":724939,"cdate":1754006400000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1767785442098,"domain":"DBLP.org","id":"MWHVcgzD9l","version":2},{"content":{"TLDR":{"value":"A comprehensive review on KV-Cache compression, involving various methods in the training, deployment, and inference stages."},"venue":{"value":"COLM"},"code_of_ethics":{"value":["I acknowledge that I and all co-authors of this work have read and commit to adhering to the COLM Code of Ethics on https://colmweb.org/CoE.html"]},"keywords":{"value":"efficient, KV-Cache, LLM, inference"},"abstract":{"value":"Large Language Models (LLMs), epitomized by ChatGPT's release in late 2022, have revolutionized various industries with their advanced language comprehension. However, their efficiency is challenged by the Transformer architecture's struggle with handling long texts. KV-Cache has emerged as a pivotal solution to this issue, converting the time complexity of token generation from quadratic to linear, albeit with increased GPU memory overhead proportional to conversation length. With the development of the LLM community and academia, various KV-Cache compression methods have been proposed. In this review, we dissect the various properties of KV-Cache and elaborate on various methods currently used to optimize the KV-Cache space usage of LLMs. These methods span the pre-training phase, deployment phase, and inference phase, and we summarize the commonalities and differences among these methods. Additionally, we list some metrics for evaluating the long-text capabilities of large language models, from both efficiency and capability perspectives. Our review thus sheds light on the evolving landscape of LLM optimization, offering insights into future advancements in this dynamic field."},"_bibtex":{"value":"@inproceedings{\nluohe2024keep,\ntitle={Keep the Cost Down: A Review on Methods to Optimize {LLM}{\\textquoteright}s {KV}-Cache Consumption},\nauthor={Shi Luohe and Hongyi Zhang and Yao Yao and Zuchao Li and hai zhao},\nbooktitle={First Conference on Language Modeling},\nyear={2024},\nurl={https://openreview.net/forum?id=8tKjqqMM5z}\n}"},"title":{"value":"Keep the Cost Down: A Review on Methods to Optimize LLM’s KV-Cache Consumption"},"research_area":{"value":["Compute efficient LMs","Engineering for large LMs","Inference algorithms for LMs"]},"pdf":{"value":"/pdf/3ca30515feafea32986aa165fc2377d4c09f6b76.pdf"},"venueid":{"value":"colmweb.org/COLM/2024/Conference"},"paperhash":{"value":"luohe|keep_the_cost_down_a_review_on_methods_to_optimize_llms_kvcache_consumption"},"authorids":{"value":["~Shi_Luohe1","~Hongyi_Zhang5","~Yao_Yao8","~Zuchao_Li1","~hai_zhao1"]},"author_guide":{"value":["I certify that this submission complies with the submission instructions as described on https://colmweb.org/AuthorGuide.html"]},"authors":{"value":["Shi Luohe","Hongyi Zhang","Yao Yao","Zuchao Li","hai zhao"]}},"tmdate":1724633534628,"pdate":1720608914943,"tcdate":1711093414809,"writers":["colmweb.org/COLM/2024/Conference","colmweb.org/COLM/2024/Conference/Submission663/Authors"],"signatures":["colmweb.org/COLM/2024/Conference/Submission663/Authors"],"forum":"8tKjqqMM5z","license":"CC BY 4.0","number":663,"cdate":1711093414809,"readers":["everyone"],"invitations":["colmweb.org/COLM/2024/Conference/-/Submission","colmweb.org/COLM/2024/Conference/-/Post_Submission","colmweb.org/COLM/2024/Conference/Submission663/-/Revision","colmweb.org/COLM/2024/Conference/-/Edit","colmweb.org/COLM/2024/Conference/Submission663/-/Camera_Ready"],"mdate":1724633534628,"odate":1724633534615,"domain":"colmweb.org/COLM/2024/Conference","id":"8tKjqqMM5z","version":2},{"content":{"summary":{"value":"The paper presents SINKQ, a KV Cache quantization designed for efficient deployment of large language models (LLMs) by balancing memory use and accuracy. SINKQ identifies unusual tokens with distinct distribution and higher attention scores, which often lead to quantization errors."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"1. Will the code be made publicly available for reproducibility?\n2. Can SINKQ be extended for higher-bit quantization, or is its design specifically optimized for low-bit scenarios?"},"rating":{"value":3},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":1},"strengths":{"value":"The paper introduces an approach to improve quantization accuracy by dynamically excluding high-error tokens."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The major claim of this paper lacks sufficient justification. The foundation of the work is based on the statement, \"Previous work suggests that attention sinks occur only in the initial tokens; however, our observations reveal that they can appear at any position within a sentence.\" However, there is no concrete evidence or examples provided to support this strong claim. To strengthen the paper, it would be beneficial if the authors included specific evidence, such as visualizations or quantitative analysis, to demonstrate instances where attention sinks appear at various positions within a sentence.\n2. In the experiments, Figure 3(b) and Figure 3(c) suggest that the proposed method may not be as memory-efficient as Kiwi, despite claims of improved efficiency. It would be helpful for the authors to directly address this observation and clarify the reasons behind this apparent discrepancy. A more detailed analysis of the trade-offs between the proposed method and Kiwi in terms of memory efficiency would strengthen the discussion.\n3. There are other state-of-the-art KV cache methods, such as those referenced in [1] and [2], which should be considered as additional baselines. Including these baselines would provide a more comprehensive evaluation. It would also be beneficial for the authors to explain why these specific baselines were not included initially and how their method compares to the key innovations introduced in [1] and [2].\n\n[1] Hooper C, Kim S, Mohammadzadeh H, et al. Kvquant: Towards 10 million context length llm inference with kv cache quantization[J]. arXiv preprint arXiv:2401.18079, 2024.\n[2] Zhang T, Yi J, Xu Z, et al. KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization[J]. arXiv preprint arXiv:2405.03917, 2024."}},"nonreaders":[],"tmdate":1731428555510,"tcdate":1730661166515,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission14060/Reviewer_MqTg"],"signatures":["ICLR.cc/2025/Conference/Submission14060/Reviewer_MqTg"],"forum":"bJ33TvbJW0","number":3,"license":"CC BY 4.0","cdate":1730661166515,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission14060/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731428555510,"domain":"ICLR.cc/2025/Conference","replyto":"bJ33TvbJW0","id":"HHd0p0v5Va","forumContent":{"venue":{"value":"ICLR 2025 Conference Withdrawn Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["KV Cache","Large Language Models","Quantization"]},"supplementary_material":{"value":"/attachment/431387138fed652b3cf97922f5c35da2f4fd21ac.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The impressive capabilities of large language models (LLMs) come at the cost of substantial computational resources during deployment. While KV Cache can significantly reduce recomputation during inference, it also introduces additional memory overhead. KV Cache quantization presents a promising solution, striking a good balance between memory usage and accuracy.\nPrevious research has shown that the Keys are distributed by channel, while the Values are distributed by token. Consequently, the common practice is to apply channel-wise quantization to the Keys and token-wise quantization to the Values. However, our further investigation reveals that a small subset of unusual tokens exhibit unique characteristics that deviate from this pattern, which can substantially impact quantization accuracy. Furthermore, these tokens often have higher attention scores, exacerbating their quantization errors.\nTo address this, we develop a simple yet effective method to identify these tokens accurately during the decoding process and exclude them from quantization, significantly improving overall accuracy. Extensive experiments show that our method achieves significant accuracy improvements under 2-bit quantization and can deliver a 6.4× reduction in memory usage and a 2.3× increase in throughput. Our code will be released upon acceptance."},"_bibtex":{"value":"@misc{\nsu2024sinkq,\ntitle={SinkQ: Accurate 2-bit {KV} Cache Quantization with Dynamic Sink Tracking},\nauthor={Yi Su and Yuechi Zhou and Juntao Li and Qingrong Xia and Ping Li and Xinyu Duan and Zhefeng Wang and Min Zhang},\nyear={2024},\nurl={https://openreview.net/forum?id=bJ33TvbJW0}\n}"},"title":{"value":"SinkQ: Accurate 2-bit KV Cache Quantization with Dynamic Sink Tracking"},"pdf":{"value":"/pdf/3116bda2b92bc73967c37ed846d9e4b814c14e2c.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Withdrawn_Submission"},"paperhash":{"value":"su|sinkq_accurate_2bit_kv_cache_quantization_with_dynamic_sink_tracking"},"authorids":{"value":["~Yi_Su3","~Yuechi_Zhou1","~Juntao_Li2","~Qingrong_Xia1","~Ping_Li16","~Xinyu_Duan1","~Zhefeng_Wang1","~Min_Zhang9"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Yi Su","Yuechi Zhou","Juntao Li","Qingrong Xia","Ping Li","Xinyu Duan","Zhefeng Wang","Min Zhang"]}},"version":2},{"content":{"TLDR":{"value":"ProjCache unifies KV cache eviction and compensation under a shared reconstruction objective after attention output projection, combining exact single-pair eviction scores with OBS-inspired compensation updates."},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["KV Cache Eviction","KV Cache Compression","Efficient LLM Inference"]},"supplementary_material":{"value":"/attachment/7fb67b580cfc0f23993fbfc9db7364f4b7175663.zip"},"primary_area":{"value":"generative models"},"abstract":{"value":"Evicting key-value (KV) pairs reduces the memory cost of large language model inference but perturbs attention outputs. We introduce ProjCache, which unifies KV cache eviction and compensation under a shared layer-wise objective: the reconstruction error measured\nafter the attention output projection. We parameterize both operations using token masks applied to attention weights and extend the saliency analysis of Optimal Brain Damage (OBD) to these mask coordinates. By summing all terms in the Taylor expansion of the\nreconstruction error, we derive an exact single-pair eviction score. This score accounts for attention renormalization and retains cross terms between query heads sharing a KV head in grouped-query attention. For compensation, we adapt Optimal Brain Surgeon (OBS) to jointly optimize the masks of retained tokens using a quadratic approximation of the same reconstruction objective. Experiments on LongBench and RULER with LLaMA-3.1-8B and Qwen3-8B show that replacing heuristic attention scores with ProjCache scores improves performance in 106 of the 108 evaluated settings, with gains of up to 3.4 points on LongBench and 25.6 points on RULER. ProjCache achieves performance comparable to state-of-the-art methods. For compensation, ProjCache lowers the reconstruction error by 17\\% to 44\\% in all 24 LongBench and RULER-4K settings, whereas prior compensation methods increase it in every setting."},"_bibtex":{"value":"@inproceedings{\nanonymous2026projcache,\ntitle={ProjCache: Output-Projected {KV} Cache Eviction with {OBS}-Inspired Compensation},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=JBy9Au1cJ5},\nnote={under review}\n}"},"title":{"value":"ProjCache: Output-Projected KV Cache Eviction with OBS-Inspired Compensation"},"pdf":{"value":"/pdf/4a73a969dbe80c9fab179cdd45e5a131b6d6e65c.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791234358780,"tcdate":1789773360022,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission50153/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission50153/Authors"],"forum":"JBy9Au1cJ5","license":"CC BY 4.0","number":50153,"cdate":1789773360022,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission50153/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791234358780,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"JBy9Au1cJ5","version":2},{"content":{"summary":{"value":"This paper introduces Deep Low-Rank Projector (DLRP), an adapter module designed to compress the Key-Value (KV) cache of large language models (LLMs), particularly in parameter-efficient fine-tuning (PEFT) settings such as LoRA. The key innovation lies in leveraging Deep Linear Networks (DLNs) to model the projection process and introducing a nuclear-norm-inspired regularizer to promote low-rank structure without performing expensive SVD. After training, the method selects the smallest energy-preserving rank, constructs the DLRP, and fine-tunes it on downstream tasks. Experiments on Qwen3-4B, Qwen3-8B, and Mistral-7B show that DLRP significantly outperforms existing methods such as SnapKV, StreamingLLM, PyramidKV, and KeyDiff, especially in PEFT settings."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"See weaknesses"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. Novel formulation:\nThe use of Deep Linear Networks for cache compression is novel and theoretically grounded. The proposed regularizer offers an elegant and computationally efficient surrogate to the nuclear norm.\n2. Comprehensive experiments:\nEvaluation across multiple benchmarks (GSM8K, PIQA, HellaSwag, XSum, CNN/DM) and models (Qwen, Mistral) demonstrates strong empirical robustness.\n3. Clear motivation:\nThe paper clearly identifies and addresses the overlooked interaction between KV-cache compression and PEFT."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Generalization beyond a single task or dataset:\nIt appears that the DLRP needs to be trained separately for each downstream task, since the rank-selection step is based on task-specific energy thresholds and the adapter is fine-tuned afterward.\nQuestion: Does this imply that if we want to apply DLRP to another downstream task, we must re-train the projector from scratch?\nIf so, this may limit scalability and increase computational overhead. I suggest the authors explore task-agnostic training or transferability across related domains.\n2. Inference-time efficiency vs. compression ratio:\nWhile the paper demonstrates strong accuracy retention, it does not explicitly quantify inference latency reduction.\nQuestion: As the compression ratio increases, does the actual inference time decrease proportionally?\nGiven that the DLRP folds into a single linear map, it would be useful to report speedups or GPU memory usage during real-time decoding.\n3. Integration with other compression paradigms:\nDLRP currently focuses on the head-dimension compression only. The authors acknowledge this in the Limitations section.\n→ Suggestion: It would be interesting to discuss or experiment with combining DLRP with sequence-length compression methods (e.g., StreamingLLM, H2O) or quantization-based approaches (e.g., bit-level cache compression). Such hybrid methods may yield higher compression without retraining."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762927220410,"tcdate":1761992173466,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission17276/Reviewer_mc12"],"signatures":["ICLR.cc/2026/Conference/Submission17276/Reviewer_mc12"],"forum":"EbIL1Po0Vf","number":3,"license":"CC BY 4.0","cdate":1761992173466,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission17276/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762927220410,"domain":"ICLR.cc/2026/Conference","replyto":"EbIL1Po0Vf","id":"rPvrt4b7mD","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Large Language Model","KV Cache Compression","Deep Linear Network"]},"supplementary_material":{"value":"/attachment/0df15aad8c3c1755f3ea821c65f8339cbd911df1.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Large Language Models (LLMs) have become integral to a wide range of natural language processing tasks. A key component enabling fast autoregressive inference in LLMs is the Key-Value~(KV) cache, which stores hidden states across decoding steps. However, the KV cache imposes substantial memory overhead, especially in long-context generation. While recent studies have proposed various compression techniques to mitigate this issue, they largely overlook the interaction between the techniques Parameter-Efficient Fine-Tuning (PEFT) methods such as LoRA, under which models are significantly more sensitive to KV cache compression. To address this issue, we propose the Deep Low-Rank Projector (DLRP), a novel adapter that compresses the KV cache along the head dimension while preserving downstream performance in PEFT-adapted models. We introduce the Deep Linear Projector (DLP), which is realized leveraging a Deep Linear Network (DLN). We also propose a novel regularizer that approximates the nuclear norm of the DLP, thereby promoting low-rank structure in the learned projection. After training with the proposed regularizer, we inspect the singular‑value spectrum and select the minimum rank satisfying a predefined energy threshold, yielding a compact head dimension that balances compression and accuracy. Based on this rank, we construct the DLRP, fine-tune it on the target task, and merge its factorized layers into a single linear operator for efficient inference. Empirical evaluation confirms that DLRP achieves substantial KV cache compression while maintaining strong performance across diverse LLM benchmarks, offering a practical solution for deploying PEFT-adapted models in memory-constrained settings."},"_bibtex":{"value":"@misc{\nkang2025deep,\ntitle={Deep Low Rank Projector for {KV} Cache Compression},\nauthor={Suhyun Kang and Yujee Song and Kyenghun Lee and Hyeonmok Ko},\nyear={2025},\nurl={https://openreview.net/forum?id=EbIL1Po0Vf}\n}"},"title":{"value":"Deep Low Rank Projector for KV Cache Compression"},"pdf":{"value":"/pdf/f11140fbdbc279a8534fad491d34bed87701a853.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"kang|deep_low_rank_projector_for_kv_cache_compression"},"authorids":{"value":["~Suhyun_Kang1","~Yujee_Song2","~Kyenghun_Lee1","~Hyeonmok_Ko1"]},"authors":{"value":["Suhyun Kang","Yujee Song","Kyenghun Lee","Hyeonmok Ko"]}},"version":2},{"content":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Vision-Language Model","Token Dropping","Multimodal Reasoning"]},"supplementary_material":{"value":"/attachment/86a58b24a84d2e1fd195b3b00ee4af9171a8096f.zip"},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"Recent advances in large language models (LLMs) have unlocked remarkable long-context capabilities, enabling breakthroughs across diverse NLP tasks. However, despite architectural progress and compression techniques such as quantization, the key-value (KV) cache remains a critical memory bottleneck during inference. Prior work has explored cache optimization via eviction strategies, yet most rely on heuristic or single-axis importance metrics, neglecting the nuanced and dynamic interplay between layers and attention heads. In this paper, we propose SCORE (Similarity-aware Contextual Overlap-Redundancy Eviction), a novel framework that introduces a distance-based multi-level similarity metric to quantify and eliminate structural redundancy within the KV cache. By dynamically reallocating cache budgets across layers and heads and employing a redundancy-aware greedy token selection mechanism, SCORE preserves semantic diversity while minimizing memory overhead. Extensive experiments on long-context benchmarks such as LongBench and NeedleBench show that SCORE retains 95\\% of full KV cache performance using only 1.5\\% of the cache, consistently outperforming state-of-the-art baselines under strict memory constraints. These results underscore the value of fine-grained, context-aware cache management for scalable and efficient long-context inference in LLMs."},"_bibtex":{"value":"@misc{\nlee2025score,\ntitle={{SCORE}: Similarity-Aware Contextual Overlap-Redundancy Eviction for Efficient {KV} Cache Compression in {LLM}s},\nauthor={Gilha lee and Seungil Lee and Hyun Kim},\nyear={2025},\nurl={https://openreview.net/forum?id=HTlkxdEDWo}\n}"},"title":{"value":"SCORE: Similarity-Aware Contextual Overlap-Redundancy Eviction for Efficient KV Cache Compression in LLMs"},"pdf":{"value":"/pdf/652add9448518949efe3ae07acae0fb25d0284f9.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"lee|score_similarityaware_contextual_overlapredundancy_eviction_for_efficient_kv_cache_compression_in_llms"},"authorids":{"value":["~Gilha_lee1","~Seungil_Lee1","~Hyun_Kim3"]},"authors":{"value":["Gilha lee","Seungil Lee","Hyun Kim"]}},"tmdate":1763517836999,"tcdate":1758251308627,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission15434/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission15434/Authors"],"forum":"HTlkxdEDWo","license":"CC BY 4.0","number":15434,"cdate":1758251308627,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/-/Submission","ICLR.cc/2026/Conference/-/Post_Submission","ICLR.cc/2026/Conference/Submission15434/-/Full_Submission","ICLR.cc/2026/Conference/-/Withdrawn_Submission"],"mdate":1763517836999,"odate":1759896705795,"domain":"ICLR.cc/2026/Conference","id":"HTlkxdEDWo","version":2},{"content":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["LLMs","KV cache retrieval","LLM inference acceleration"]},"primary_area":{"value":"generative models"},"abstract":{"value":"While Key-Value (KV) cache succeeds in reducing redundant computations in auto-regressive models, it introduces significant memory overhead, limiting its practical deployment in long-sequence scenarios. Existing KV retrieval methods attempt to mitigate this by dynamically retaining only a subset of KV entries on the GPU. However, they still suffer from notable efficiency and accuracy bottlenecks due to per-token retrieval and coarse-grained page-level KV management strategy, especially in long-output reasoning scenarios. With the emergence of large reasoning models, efficiently handling such scenarios has become increasingly important. To address this issue, we present two key observations: (1) critical KVs exhibit strong temporal locality during decoding, and (2) these KVs exhibit distinct distribution patterns across the input prompt and the generated output. Building on these observations, we propose \\emph{LouisKV}, an efficient KV cache retrieval framework designed for various long-sequence scenarios. Specifically, LouisKV introduces a semantic-aware retrieval strategy that leverages temporal locality to trigger retrieval only at semantic boundaries, drastically reducing computation and data transfer overhead. LouisKV also designs a decoupled, fine-grained management scheme that tailors differentiated strategies for input and output sequences to create retrieval units that better match the model's attention patterns, thereby enabling the precise identification of critical KVs. Furthermore, to boost system efficiency, LouisKV incorporates several kernel-level optimizations, including custom Triton and CUDA kernels to accelerate the KV clustering and retrieval. Evaluation results show that LouisKV achieves up to 4.7$\\times$ speedup over state-of-the-art KV retrieval methods while maintaining near-lossless accuracy across diverse long-sequence tasks, including long-input short-output, short-input long-output, and long-input long-output scenarios."},"_bibtex":{"value":"@inproceedings{\nwu2026louiskv,\ntitle={Louis{KV}: Efficient {KV} Cache Retrieval for Long Input-Output Sequences},\nauthor={Wenbo Wu and Qingyi Si and Xiurui Pan and Ye Wang and Jie Zhang},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=6RJ8fZwm4P}\n}"},"title":{"value":"LouisKV: Efficient KV Cache Retrieval for Long Input-Output Sequences"},"pdf":{"value":"/pdf/e52b710edf11b6acbcce32310c6f2710f8fc2c92.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"wu|louiskv_efficient_kv_cache_retrieval_for_long_inputoutput_sequences"},"authorids":{"value":["~Wenbo_Wu2","~Qingyi_Si1","~Xiurui_Pan1","~Ye_Wang30","~Jie_Zhang88"]},"authors":{"value":["Wenbo Wu","Qingyi Si","Xiurui Pan","Ye Wang","Jie Zhang"]}},"tmdate":1775877085549,"pdate":1769436069300,"tcdate":1758198942053,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission11436/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission11436/Authors"],"forum":"6RJ8fZwm4P","license":"CC BY 4.0","number":11436,"cdate":1758198942053,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/-/Submission","ICLR.cc/2026/Conference/-/Post_Submission","ICLR.cc/2026/Conference/Submission11436/-/Full_Submission","ICLR.cc/2026/Conference/Submission11436/-/Rebuttal_Revision","ICLR.cc/2026/Conference/-/Edit","ICLR.cc/2026/Conference/Submission11436/-/Camera_Ready_Revision"],"mdate":1775877085549,"odate":1759896705795,"domain":"ICLR.cc/2026/Conference","id":"6RJ8fZwm4P","version":2},{"content":{"summary":{"value":"This paper presents LongFlow, a lightweight KV cache compression algorithm specifically designed for reasoning models with long decoding outputs. \n\nIt introduces an efficient importance estimation metric that is computed directly from intermediate attention results using only the current query, incurring negligible computational overhead. \nThe authors also implement a custom Triton kernel that fuses FlashAttention, importance estimation, and token eviction into a single optimized operator. \n\nExperimental results show that LongFlow achieves up to an 11.8× throughput improvement and an 80% reduction in KV cache size, with only minor loss in model accuracy."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"1. Could the authors **provide an ablation study** on the accuracy loss caused by each approximation in LongFlow, including the **query approximation**, **attention denominator simplification**, and **omission of the running maximum in $qk$ scores**?\n \n2. If these approximations are shown to introduce substantial errors, could the method be **enhanced by incorporating correction mechanisms**?\n \n3. Is **omitting the running maximum of $qk$ scores** truly necessary for efficiency? Have the authors evaluated the trade-off between numerical stability and performance?\n \n4. What are the **evaluation metrics and decoding settings** used in the accuracy experiments (e.g., _pass@k_, temperature, top-p, sampling strategy)?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"LongFlow introduces a novel and lightweight importance metric for token eviction, which is directly derived from intermediate attention weights and value vector norms without additional storage overhead. \nIt also implements an Triton kernel that fuses FlashAttention, importance estimation, and token eviction."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"While LongFlow achieves notable efficiency improvements, its accuracy degradation is non-trivial, particularly on AIME24/25, AMC, and GPQA benchmarks with the Qwen3-8B model. \nThis degradation may stem from the inherent limitations of KV eviction methods and the **aggressive approximations** LongFlow applies to query vectors and attention denominators. \nDue to the existence of **outlier key channels** (where $\\max_j |k^j|$ is large) \\[1\\], the theoretical error bound can grow substantially, undermining eviction accuracy. \nMoreover, LongFlow **omits the running maximum of $qk$ scores** in its FlashAttention-based kernel, which weakens **softmax numerical stability** and can further degrade attention precision.\n\nIn addition, the **efficiency experiments are limited to Qwen3-1.7B**, and the scalability of the efficiency improvements to larger models remains unverified.\n\n\\[1\\] Liu et al., KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762926574966,"tcdate":1761928957515,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission16469/Reviewer_wUq4"],"signatures":["ICLR.cc/2026/Conference/Submission16469/Reviewer_wUq4"],"forum":"rz6WybXjgk","number":3,"license":"CC BY 4.0","cdate":1761928957515,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission16469/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762926574966,"domain":"ICLR.cc/2026/Conference","replyto":"rz6WybXjgk","id":"LhS79xBuPS","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["LLM","KV cache compression","Reasoning models"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Reasoning models like OpenAI-o1 and DeepSeek-R1 have demonstrated strong capabilities in complex tasks such as mathematical reasoning and code generation.\nHowever, this leap in performance is achieved by generating a significantly greater number of output tokens, which dramatically increases deployment costs.\nThe generation of extremely long sequences necessitates a longer KV cache, which in turn results in a substantial memory footprint and severe bandwidth pressure during attention computation.\nWhile there are numerous techniques to optimize KV cache, they are predominantly designed for long-input, short-output scenarios and are ineffective for the long-output nature of these reasoning models.\nThe high computational cost of their importance estimation is severely exacerbated in long-output scenarios by the need for continuous context re-evaluation.\nTo overcome this challenge, we introduce LongFlow, a novel KV cache compression method that employs an efficient importance estimation metric derived from an intermediate result in the attention computation using only the current query. This elegant design requires no auxiliary storage and adds negligible computational overhead. \nFurthermore, we implement a custom kernel that integrates Flash-Attention, importance estimation, and token eviction into a single, highly optimized operator to enhance system-level efficiency.\nExtensive experiments demonstrate that our method can achieve an 11.8x increase in throughput with an 80\\% compression of the KV\ncache while incurring negligible loss in model accuracy."},"_bibtex":{"value":"@misc{\nsu2025longflow,\ntitle={LongFlow: Efficient {KV} Cache Compression for Reasoning Models},\nauthor={Yi Su and Zhenxu Tian and Yuechi Zhou and Haijia Gui and Juntao Li and Min Zhang},\nyear={2025},\nurl={https://openreview.net/forum?id=rz6WybXjgk}\n}"},"title":{"value":"LongFlow: Efficient KV Cache Compression for Reasoning Models"},"pdf":{"value":"/pdf/73f64f7718a427df1e006976a5d14940131d1d86.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"su|longflow_efficient_kv_cache_compression_for_reasoning_models"},"authorids":{"value":["~Yi_Su3","~Zhenxu_Tian1","~Yuechi_Zhou1","~Haijia_Gui1","~Juntao_Li2","~Min_Zhang9"]},"authors":{"value":["Yi Su","Zhenxu Tian","Yuechi Zhou","Haijia Gui","Juntao Li","Min Zhang"]}},"version":2},{"content":{"summary":{"value":"This paper presents a frame-level AR framework for real-time interactive long video generation. The paper mainly focuses on efficiency design. The authors introduce a KV-recache mechanism that refreshes cached states to ensure smooth, adherent transitions when prompts change. They also propose a tuning strategy to address quality degradation in long videos, aligning training with inference. For efficiency, the model combines short window attention with a frame sink to maintain long-range consistency while accelerating generation. The resulting 1.3B parameter model achieves 20.7 FPS on an H100 GPU, supports videos up to 240 seconds, and demonstrates strong performance on VBench for both short and long video benchmarks."},"soundness":{"value":3},"confidence":{"value":2},"questions":{"value":"Please see weaknesses."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":3},"strengths":{"value":"1. A new KV-recache strategy is proposed for frame-level AR with multiple prompts.\n2. Fairly comprehensive evluation, demonstrating the performance on different lengths of video.\n3. Some efficient strategies for training and inference are proposed for long videos."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Generalization ability. Though the authors show some long video cases, the prompt still remains in the same scene. How does it handle multiple, rapid-fire prompts? More importantly, how does it respond to prompts that introduce abrupt semantic shifts (e.g., changing from \"Iron Man fighting\" to \"a peaceful beach\") versus the logical continuations shown in the examples?\n2. The \"train-long-test-long\" strategy is a key strength, but its supervision (DMD loss) relies on a teacher model that itself was only trained on short clips. While the student model is exposed to long rollouts, the supervision for each new clip is still provided by this short-clip teacher. This may create a ceiling for true long-range coherence, as the teacher lacks the long-horizon context the student is trying to learn.\n3. The ablation on time consuming about the KV cache, no KV Cache, and KV recache for different lengths of video is suggested to add to prove the effiency.\n4. The ablation about frame sink strategy (first frame or others) is encouraged to add."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762916058076,"tcdate":1761920619406,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission2157/Reviewer_tdeq"],"signatures":["ICLR.cc/2026/Conference/Submission2157/Reviewer_tdeq"],"forum":"nCAODkpsPJ","number":3,"license":"CC BY 4.0","cdate":1761920619406,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission2157/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762916058076,"domain":"ICLR.cc/2026/Conference","replyto":"nCAODkpsPJ","id":"OCYrrMe5N6","forumContent":{"TLDR":{"value":"We present LongLive, that is a real-time, interactive, and AR framework for long video generation."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Real-time","Interactive","Long Video Generation"]},"supplementary_material":{"value":"/attachment/6fe8f481a0c438177d3866f80a6f3b3989b24501.zip"},"primary_area":{"value":"generative models"},"abstract":{"value":"We present LongLive, a frame-level autoregressive (AR) framework for real-time and interactive long video generation. Long video generation presents challenges in both efficiency and quality. Diffusion and Diffusion-Forcing models can produce high-quality videos but suffer from low efficiency due to bidirectional attention. Causal attention AR models support KV caching for faster inference but often degrade in quality on long videos due to memory challenges during long-video training. In addition, beyond static prompt-based generation, interactive capabilities, such as streaming prompt inputs, are critical for dynamic content creation, enabling users to guide narratives in real time. This interactive requirement significantly increases the complexity, especially in ensuring visual consistency and semantic coherence during prompt transitions. To address these challenges, LongLive adopts a causal, frame-level AR design that integrates a KV-recache mechanism that refreshes cached states with the new prompt for smooth, adherent switches streaming long tuning to enable long video training and to align training and inference (train-long–test-long); and short window attention paired with a frame-level attention sink, preserving long-range consistency while enabling faster generation. With these key designs, LongLive fine-tunes a 1.3B-parameter short-clip model to minute-long generation in just 32 GPU-days. At inference, LongLive sustains 20.7 FPS on a single NVIDIA H100, achieves strong performance on VBench in both short- and long-video settings. LongLive supports up to 240-second videos on a single H100 GPU. With FP8 quantization, LongLive boosts inference to 24.8 FPS with marginal quality loss."},"_bibtex":{"value":"@inproceedings{\nyang2026longlive,\ntitle={LongLive: Real-time Interactive Long Video Generation},\nauthor={Shuai Yang and Wei Huang and Ruihang Chu and Yicheng Xiao and Yuyang Zhao and Xianbang Wang and Muyang Li and Enze Xie and Ying-Cong Chen and Yao Lu and Song Han and Yukang Chen},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=nCAODkpsPJ}\n}"},"title":{"value":"LongLive: Real-time Interactive Long Video Generation"},"pdf":{"value":"/pdf/fc6c8a70c050feda94f14f7bbd7e13df1dae002c.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"yang|longlive_realtime_interactive_long_video_generation"},"authorids":{"value":["~Shuai_Yang7","~Wei_Huang36","~Ruihang_Chu1","~Yicheng_Xiao1","~Yuyang_Zhao1","~Xianbang_Wang1","~Muyang_Li2","~Enze_Xie1","~Ying-Cong_Chen1","~Yao_Lu13","~Song_Han5","~Yukang_Chen1"]},"authors":{"value":["Shuai Yang","Wei Huang","Ruihang Chu","Yicheng Xiao","Yuyang Zhao","Xianbang Wang","Muyang Li","Enze Xie","Ying-Cong Chen","Yao Lu","Song Han","Yukang Chen"]}},"version":2},{"content":{"venue":{"value":"ICLR 2026 Poster"},"TLDR":{"value":"We repurpose the KV cache—traditionally used only for speedup—as a free representation for sampling and reasoning, enabling output-free self-evaluation and adaptive fast/slow thinking with negligible overhead and strong empirical results"},"keywords":{"value":["Machine Learning","LLM"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"KV caches, typically used only to speed up autoregressive decoding, encode contextual information that can be reused for downstream tasks at no extra cost. We propose treating the KV cache as a lightweight representation, eliminating the need to recompute or store full hidden states. Despite being weaker than dedicated embeddings, KV-derived representations are shown to be sufficient for two key applications: (i) Chain-of-Embedding, where they achieve competitive or superior performance on Llama-3.1-8B-Instruct and Qwen2-7B-Instruct; and (ii) Fast/Slow Thinking Switching, where they enable adaptive reasoning on Qwen3-8B and DeepSeek-R1-Distil-Qwen-14B, reducing token generation by up to $5.7\\times$ with minimal accuracy loss. Our findings establish KV caches as a free, effective substrate for sampling and reasoning, opening new directions for representation reuse in LLM inference."},"_bibtex":{"value":"@inproceedings{\nxing2026beyond,\ntitle={Beyond Speedup - Utilizing {KV} Cache for Sampling and Reasoning},\nauthor={Zeyu XING and Xing Li and Hui-Ling Zhen and Mingxuan Yuan and Sinno Jialin Pan},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=GUhmiJaAzv}\n}"},"title":{"value":"Beyond Speedup - Utilizing KV Cache for Sampling and Reasoning"},"pdf":{"value":"/pdf/4913f2cf6ab8f2dd32867a27233c0a1ce387a6ea.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"xing|beyond_speedup_utilizing_kv_cache_for_sampling_and_reasoning"},"authorids":{"value":["~Zeyu_XING1","~Xing_Li6","~Hui-Ling_Zhen1","~Mingxuan_Yuan1","~Sinno_Jialin_Pan1"]},"authors":{"value":["Zeyu XING","Xing Li","Hui-Ling Zhen","Mingxuan Yuan","Sinno Jialin Pan"]}},"tmdate":1775877116828,"pdate":1769436107021,"tcdate":1758207462734,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission12382/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission12382/Authors"],"forum":"GUhmiJaAzv","license":"CC BY 4.0","number":12382,"cdate":1758207462734,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/-/Submission","ICLR.cc/2026/Conference/-/Post_Submission","ICLR.cc/2026/Conference/Submission12382/-/Full_Submission","ICLR.cc/2026/Conference/-/Edit","ICLR.cc/2026/Conference/Submission12382/-/Camera_Ready_Revision"],"mdate":1775877116828,"odate":1759896705795,"domain":"ICLR.cc/2026/Conference","id":"GUhmiJaAzv","version":2},{"content":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["LLM","KV cache compression","Long-context"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"The Key-Value (KV) cache is crucial for efficient Large Language Models (LLMs) inference, but excessively long contexts drastically increase KV cache memory footprint. Existing KV cache compression methods typically rely on input-side attention patterns within a prompt observation window to estimate token importance during the prefill stage. They fail to preserve critical tokens for future generation since these assessments are not derived from the decoding process. Intuitively, an effective observation window should mirror the decoding-stage queries to accurately reflect which tokens the generation process will attend to. However, ground-truth decoding queries are inherently unavailable during inference. For constructing pseudo-queries to approximate them, we find that positional information plays a more critical role than semantic content. Motivated by this insight, we propose decoding-aligned KV cache compression via position-aware pseudo-queries (DapQ), a novel and lightweight eviction framework that leverages position-aware pseudo-queries to simulate the output tokens, thereby establishing an effective observation window for importance assessment. It enables precise token eviction that aligns closely with the actual generation context. Extensive evaluation across multiple benchmarks and LLMs demonstrates that DapQ achieves superior performance, particularly under strict memory constraints (e.g., up to nearly lossless performance 99.5\\% on NIAH with 3\\% KV cache budgets)."},"_bibtex":{"value":"@misc{\ntian2025where,\ntitle={Where Matters More Than What: Decoding-aligned {KV} Cache Compression via Position-aware Pseudo-queries},\nauthor={Zhenxu Tian and Yi Su and Juntao Li and Min Zhang},\nyear={2025},\nurl={https://openreview.net/forum?id=ZpgyPxdxiL}\n}"},"title":{"value":"Where Matters More Than What: Decoding-aligned KV Cache Compression via Position-aware Pseudo-queries"},"pdf":{"value":"/pdf/ad512fcca70f8b5e5368f1ae98d812868865ee6c.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"tian|where_matters_more_than_what_decodingaligned_kv_cache_compression_via_positionaware_pseudoqueries"},"authorids":{"value":["~Zhenxu_Tian1","~Yi_Su3","~Juntao_Li2","~Min_Zhang9"]},"authors":{"value":["Zhenxu Tian","Yi Su","Juntao Li","Min Zhang"]}},"tmdate":1766995573760,"tcdate":1758265255637,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission16503/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission16503/Authors"],"forum":"ZpgyPxdxiL","license":"CC BY 4.0","number":16503,"cdate":1758265255637,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/-/Submission","ICLR.cc/2026/Conference/-/Post_Submission","ICLR.cc/2026/Conference/Submission16503/-/Full_Submission","ICLR.cc/2026/Conference/-/Withdrawn_Submission"],"mdate":1766995573760,"odate":1759896705795,"domain":"ICLR.cc/2026/Conference","id":"ZpgyPxdxiL","version":2},{"content":{"venue":{"value":"NeurIPS 2023 poster"},"keywords":{"value":["Large language model; KV Cache Compression"]},"supplementary_material":{"value":"/attachment/23b721958b5b0609724f41bd8eaec4e19f6df17a.pdf"},"_bibtex":{"value":"@inproceedings{\nliu2023scissorhands,\ntitle={Scissorhands: Exploiting the Persistence of Importance Hypothesis for {LLM} {KV} Cache Compression at Test Time},\nauthor={Zichang Liu and Aditya Desai and Fangshuo Liao and Weitao Wang and Victor Xie and Zhaozhuo Xu and Anastasios Kyrillidis and Anshumali Shrivastava},\nbooktitle={Thirty-seventh Conference on Neural Information Processing Systems},\nyear={2023},\nurl={https://openreview.net/forum?id=JZfg6wGi6g}\n}"},"title":{"value":"Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time"},"paperhash":{"value":"liu|scissorhands_exploiting_the_persistence_of_importance_hypothesis_for_llm_kv_cache_compression_at_test_time"},"abstract":{"value":"Large language models(LLMs) have sparked a new wave of exciting AI applications. Hosting these models at scale requires significant memory resources. One crucial memory bottleneck for the deployment stems from the context window. It is commonly recognized that model weights are memory hungry; however, the size of key-value embedding stored during the generation process (KV cache) can easily surpass the model size. The enormous size of the KV cache puts constraints on the inference batch size, which is crucial for high throughput inference workload. Inspired by an interesting observation of the attention scores, we hypothesize the persistence of importance: only pivotal tokens, which had a substantial influence at one step, will significantly influence future generations. Based on our empirical verification and theoretical analysis around this hypothesis, we propose scissorhands, a system that maintains the memory usage of the KV cache at a fixed budget without finetuning the model. In essence, Scissorhands manages the KV cache by storing the pivotal tokens with a higher probability. We validate that scissorhands reduces the inference memory usage of the KV cache by up to 5$\\times$ without compromising model quality. We further demonstrate that scissorhands can be combined with 4-bit quantization, traditionally used to compress model weights, to achieve up to 20$\\times$ compression."},"pdf":{"value":"/pdf/1fc97e3481eec514e1bd9ae2bb5b23d5d29b3c27.pdf"},"venueid":{"value":"NeurIPS.cc/2023/Conference"},"authorids":{"value":["~Zichang_Liu1","~Aditya_Desai1","~Fangshuo_Liao1","wtwang@rice.edu","vyx2@rice.edu","~Zhaozhuo_Xu1","~Anastasios_Kyrillidis2","~Anshumali_Shrivastava1"]},"authors":{"value":["Zichang Liu","Aditya Desai","Fangshuo Liao","Weitao Wang","Victor Xie","Zhaozhuo Xu","Anastasios Kyrillidis","Anshumali Shrivastava"]}},"tmdate":1698949760922,"pdate":1695326034724,"tcdate":1683777061516,"writers":["NeurIPS.cc/2023/Conference","NeurIPS.cc/2023/Conference/Submission10014/Authors"],"signatures":["NeurIPS.cc/2023/Conference/Submission10014/Authors"],"forum":"JZfg6wGi6g","number":10014,"cdate":1683777061516,"mdate":1698949760922,"readers":["everyone"],"invitations":["NeurIPS.cc/2023/Conference/-/Submission","NeurIPS.cc/2023/Conference/-/Post_Submission","NeurIPS.cc/2023/Conference/Submission10014/-/Revision","NeurIPS.cc/2023/Conference/Submission10014/-/Supplementary_Material_Revision","NeurIPS.cc/2023/Conference/-/Edit","NeurIPS.cc/2023/Conference/Submission10014/-/Camera_Ready_Revision"],"odate":1698949760844,"domain":"NeurIPS.cc/2023/Conference","id":"JZfg6wGi6g","version":2},{"content":{"summary":{"value":"The paper addresses the recall gap of linear-attention and recurrent LMs by re-introducing targeted access to past tokens without abandoning efficiency. It studies two hybrid designs: (1) laNSA: interleaves linear attention with Native Sparse Attention (NSA) that combines query-aware block probing, a compressed branch, and SWA-style gating. This improves retrieval but keeps an O(N) KV cache. (2) laLTE: interleaves linear attention with Learnable Token Eviction (LTE). A tiny per-token, per-head 1D-CNN predicts whether to retain a KV as it leaves a large SWA window; combined with an attention sink and SWA, this aims for O(1) time and space per step while preserving long-range evidence. Experiments at roughly 0.4B and 1.4B parameters trained on 10B/30B tokens (FineWeb-Edu) evaluate short-context language tasks and long-context retrieval (RULER S-NIAH, EVAPORATE). Both laLTE and laNSA outperform strong linear baselines; laNSA is strongest among linear-time models on EVAPORATE, while laLTE is the best constant-space option and approaches hybrid full-attention variants in some settings."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"1. Can you report measured GPU memory (GB), effective KV size, tokens/sec, and latency for GDN, GDN+SWA, laLTE, laNSA, and a GDN+full-attention hybrid at 4K and at least 8K contexts on the same hardware?\n\n2. Since laLTE/laNSA are trained modules, could you also include an inference-only comparison on the same GPU (with matched context length and batch size) against training-free efficient attention approaches such as HiP [1]? Reporting retrieval accuracy (e.g., RULER/EVAPORATE), throughput (tokens/s), and peak memory would help clarify whether the additional training cost yields meaningful gains over training-free sparse attention methods.\n\n--- \n[1] Lee et al. A Training-free Sub-quadratic Cost Transformer Model Serving Framework With Hierarchically Pruned Attention. ICLR 2025."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"### Novelty: \nThis work proposes two complementary mixers interleaved with linear attention—NSA for query-aware sparse access over the full past and LTE for learned keep/evict under a strict cache budget; introduces per-token, per-head retention via a tiny 1D-CNN with SWA-enabled look-ahead and an attention sink to maintain near-constant KV memory; provides deployment-minded decoding/KV design (two-segment cache, lazy batched scoring) and frames a clear accuracy–efficiency Pareto frontier (laLTE for constant-space, laNSA for higher accuracy under linear time).\n\n### Differentiation from prior works: \nThis work moves beyond fixed windows and global/uniform/time-decay heuristics by making head-aware, context-conditioned retention decisions; regains direct token-level access to salient long-range evidence without reverting to O(N²) attention, contrasting with state-space/recurrence approaches that compress history into fixed states; offers a stronger NSA baseline (query-aware probing + compressive branch + SWA gating) that sharpens comparisons."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"### Scale and generality: \nResults are limited to 0.4B/1.4B. It is unclear whether the trends hold for larger, modern LLM families (e.g., Qwen2.5/3, DeepSeek) or for multilingual/code models.\n### Benchmark breadth: \nThe evaluation focuses on long-context retrieval. Broader benchmarks commonly used today (e.g., instruction following, math, and code such as AlpacaEval, GSM8K, HumanEval) are absent, making it hard to gauge side effects beyond retrieval.\n\n### Efficiency reporting: \nThe paper argues constant time/space for laLTE by design, but it does not provide systematic, measured wall-clock throughput and GPU memory usage across mixers (GDN, +SWA, laLTE, laNSA, +full-attention).\n\n### NSA dependency: \nlaNSA still requires O(N) KV. The trade-off versus laLTE is discussed conceptually; clearer guidance on when laNSA is preferable in practice would help."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762919463798,"tcdate":1761924011765,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7341/Reviewer_uoix"],"signatures":["ICLR.cc/2026/Conference/Submission7341/Reviewer_uoix"],"forum":"ioeOm5tFDM","number":3,"license":"CC BY 4.0","cdate":1761924011765,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7341/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762919463798,"domain":"ICLR.cc/2026/Conference","replyto":"ioeOm5tFDM","id":"yXirROwlqd","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"TLDR":{"value":"Enhanced retrieval performance of subquadratic time/constant space models by the hybrid of linear attention and query-aware sparse attention or our novel learnable token eviction network that evicts KV cache based on past and future local contexts."},"keywords":{"value":["linear attention","language modeling","sparse attention","token eviction"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Linear-attention models that compress the entire input sequence into a fixed-size recurrent state offer an efficient alternative to Transformers, but their finite memory induces forgetfulness that harms retrieval-intensive tasks.\nTo mitigate the issue, we explore a series of hybrid models that restore direct access to past tokens. We interleave token mixers of intermediate time and space complexity between linear attention and full attention, including the query-aware native sparse attention, and sparse attention with token eviction. We further propose a novel learnable token eviction module. Combined with sliding-window attention, an end-to-end trainable lightweight CNN-based eviction module aggregates information from both past and future adjacent tokens to adaptively retain a limited set of critical KV-pairs per head, maintaining linear attention's constant time and space complexity. Empirical evaluations on retrieval-intensive benchmarks support the effectiveness of our approaches."},"_bibtex":{"value":"@misc{\nhe2026alleviating,\ntitle={Alleviating Forgetfulness of Linear Attention by Hybrid Sparse Attention and Contextualized Learnable Token Eviction},\nauthor={Mutian He and Philip N. Garner},\nyear={2026},\nurl={https://openreview.net/forum?id=ioeOm5tFDM}\n}"},"title":{"value":"Alleviating Forgetfulness of Linear Attention by Hybrid Sparse Attention and Contextualized Learnable Token Eviction"},"pdf":{"value":"/pdf/e0b760b947ed9eecba7680732b0f302cc2502105.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"he|alleviating_forgetfulness_of_linear_attention_by_hybrid_sparse_attention_and_contextualized_learnable_token_eviction"},"authorids":{"value":["~Mutian_He1","~Philip_N._Garner1"]},"authors":{"value":["Mutian He","Philip N. Garner"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2406.17808v1"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"willette|trainingfree_exponential_extension_of_sliding_window_context_with_cascading_kv_cache"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Jeffrey_Willette:","","https://dblp.org/search/pid/api?q=author:Youngwan_Lee:","~Myeongjae_Jeon3","~Sung_Ju_Hwang1"]},"html":{"value":"https://doi.org/10.48550/arXiv.2406.17808"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2406-17808,\n publtype={informal},\n author={Jeffrey Willette and Heejun Lee and Youngwan Lee and Myeongjae Jeon and Sung Ju Hwang},\n title={Training-Free Exponential Extension of Sliding Window Context with Cascading KV Cache},\n year={2024},\n cdate={1704067200000},\n journal={CoRR},\n volume={abs/2406.17808},\n url={https://doi.org/10.48550/arXiv.2406.17808}\n}\n"},"abstract":{"value":"The context window within a transformer provides a form of active memory for the current task, which can be useful for few-shot learning and conditional generation, both which depend heavily on previous context tokens. However, as the context length grows, the computational cost increases quadratically. Recent works have shown that saving a few initial tokens along with a fixed-sized sliding window leads to stable streaming generation with linear complexity in transformer-based Large Language Models (LLMs). However, they make suboptimal use of the fixed window by naively evicting all tokens unconditionally from the key-value (KV) cache once they reach the end of the window, resulting in tokens being forgotten and no longer able to affect subsequent predictions. To overcome this limitation, we propose a novel mechanism for storing longer sliding window contexts with the same total cache size by keeping separate cascading sub-cache buffers whereby each subsequent buffer conditionally accepts a fraction of the relatively more important tokens evicted from the previous buffer. Our method results in a dynamic KV cache that can store tokens from the more distant past than a fixed, static sliding window approach. Our experiments show improvements of 5.6% on long context generation (LongBench), 1.2% in streaming perplexity (PG19), and 0.6% in language understanding (MMLU STEM) using LLMs given the same fixed cache size. Additionally, we provide an efficient implementation that improves the KV cache latency from 1.33ms per caching operation to 0.54ms, a 59% speedup over previous work."},"title":{"value":"Training-Free Exponential Extension of Sliding Window Context with Cascading KV Cache"},"authors":{"value":["Jeffrey Willette","Heejun Lee","Youngwan Lee","Myeongjae Jeon","Sung Ju Hwang"]}},"tmdate":1747238916884,"pdate":1704067200000,"tcdate":1722957542731,"writers":["~"],"signatures":["~Myeongjae_Jeon3"],"forum":"2vgrJ2N9W0","license":"CC BY-SA 4.0","number":56474,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1747238916884,"domain":"DBLP.org","id":"2vgrJ2N9W0","version":2},{"content":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Large Language Models; Long Context; Efficient Inference"]},"supplementary_material":{"value":"/attachment/9c66953417e130ca2b91b1370086d2c1db18974f.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Large language models (LLMs) with extended context windows enable powerful applications but impose significant memory overhead, as caching all key–value (KV) states grows linearly with sequence length and batch size. Existing cache eviction methods address this by exploiting attention sparsity, yet they typically rank tokens heuristically using accumulated attention weights without considering their true impact on attention outputs. We propose Optimal Brain Cache (OBCache), a principled framework that formulates cache eviction as a layer-wise structured pruning problem. Building on Optimal Brain Damage (OBD) theory, OBCache quantifies token saliency by measuring the perturbation on attention outputs induced by pruning tokens, with closed-form scores derived for isolated keys, isolated values, and joint key–value pairs. Our scores account not only for attention weights but also for information from value states and attention outputs, thereby enhancing existing eviction strategies with output-aware signals. Experiments on LLaMA and Qwen models show that replacing the heuristic scores in existing works, which estimate token saliency across different query positions, with OBCache's output-aware scores consistently improves long-context accuracy."},"_bibtex":{"value":"@misc{\ngu2026obcache,\ntitle={{OBC}ache: Optimal Brain {KV} Cache Pruning for Efficient Long-Context {LLM} Inference},\nauthor={Yuzhe Gu and Xiyu Liang and Jiaojiao Zhao and Enmao Diao},\nyear={2026},\nurl={https://openreview.net/forum?id=JLfky7RakB}\n}"},"title":{"value":"OBCache: Optimal Brain KV Cache Pruning for Efficient Long-Context LLM Inference"},"pdf":{"value":"/pdf/0d93ba99551baa503e09d14f51a1fde9c899638f.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"gu|obcache_optimal_brain_kv_cache_pruning_for_efficient_longcontext_llm_inference"},"authorids":{"value":["~Yuzhe_Gu2","~Xiyu_Liang1","~Jiaojiao_Zhao1","~Enmao_Diao1"]},"authors":{"value":["Yuzhe Gu","Xiyu Liang","Jiaojiao Zhao","Enmao Diao"]}},"tmdate":1770804804195,"tcdate":1756843336939,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission1117/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission1117/Authors"],"forum":"JLfky7RakB","license":"CC BY 4.0","number":1117,"cdate":1756843336939,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/-/Submission","ICLR.cc/2026/Conference/-/Post_Submission","ICLR.cc/2026/Conference/Submission1117/-/Full_Submission","ICLR.cc/2026/Conference/Submission1117/-/Rebuttal_Revision","ICLR.cc/2026/Conference/-/Edit"],"mdate":1770804804195,"odate":1759896705795,"domain":"ICLR.cc/2026/Conference","id":"JLfky7RakB","version":2},{"content":{"venue":{"value":"CoRR 2026"},"pdf":{"value":"https://arxiv.org/pdf/2605.17757v1"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"zhou|oscar_offline_spectral_covarianceaware_rotation_for_2bit_kv_cache_quantization"},"html":{"value":"https://doi.org/10.48550/arXiv.2605.17757"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2605-17757,\n publtype={informal},\n author={Zhongzhu Zhou and Donglin Zhuang and Jisen Li and Ziyan Chen and Shuaiwen Leon Song and Ben Athiwaratkun and Xiaoxia Wu},\n title={OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization},\n year={2026},\n month={May},\n cdate={1777593600000},\n journal={CoRR},\n volume={abs/2605.17757},\n url={https://doi.org/10.48550/arXiv.2605.17757}\n}\n"},"abstract":{"value":"INT2 KV-cache quantization is attractive for long-context LLM serving, but it remains difficult to make both accurate and deployable. Simple rotations such as Hadamard transforms reduce outliers, but still degrade at INT2 because they are not aligned with downstream attention. We propose OSCAR, an Ultra-low-bit KV Cache quantization method that estimates attention-aware covariance structures offline and uses them to derive fixed rotations and clipping thresholds for quantization. In this way, it aligns KV quantization with the covariance structures that attention actually consumes. More importantly, we not only provide theoretical justification but also develop a fully deployable OSCAR system with a custom INT2 attention kernel that remains compatible with paged KV-cache serving and fused kernel pipelines, enabling seamless integration into modern LLM serving frameworks such as SGLang and vLLM. We evaluate our methods on recent reasoning models with reasoning traces of up to 32k tokens across 5 tasks. On Qwen3-4B-Thinking-2507 and Qwen3-8B, OSCAR reduces the BF16 accuracy gap to 3.78 and 1.42 points, respectively, while naive rotation INT2 collapses to nearly zero. We further scale OSCAR to Qwen3-32B and GLM-4.7 (358B params), where it remains effectively on par with BF16. On long context - RULER-NIAH up to 128K, OSCAR remains robust on both Qwen3 models, while naive rotation INT2 collapses. System-wise, OSCAR reduces KV-cache memory by approximately 8x, improves throughput by up to 7x at large batch sizes under the same memory budget, and accelerates batch-size-1 decoding by up to 3x over BF16 due to reduced memory bandwidth overhead."},"title":{"value":"OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization"},"authors":{"value":[{"fullname":"Zhongzhu Zhou","username":""},{"fullname":"Donglin Zhuang","username":""},{"fullname":"Jisen Li","username":"~Jisen_Li2"},{"fullname":"Ziyan Chen","username":""},{"fullname":"Shuaiwen Leon Song","username":""},{"fullname":"Ben Athiwaratkun","username":""},{"fullname":"Xiaoxia Wu","username":""}]}},"tmdate":1785886555023,"pdate":1798675200000,"externalIds":["dblp:journals/corr/abs-2605-17757"],"tcdate":1785886551643,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Jisen_Li2"],"forum":"2r0kDyUsPy","license":"CC BY-SA 4.0","number":119912,"cdate":1777593600000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1785886555023,"domain":"OpenReview.net/Public_Article","id":"2r0kDyUsPy","version":2},{"content":{"data_release":{"value":"We authorize the release of our submission and author names to the public in the event of acceptance."},"TLDR":{"value":"OSCAR learns fixed INT2 KV rotations from attention-aware Q/K/V covariances, keeps near-BF16 accuracy on Qwen3 and GLM-4.7, and runs in paged KV serving via a fused Triton mixed-precision attention kernel."},"venue":{"value":"ACL-SELVA 2026 Poster"},"email_sharing":{"value":"We authorize the sharing of all author emails with Program Chairs."},"keywords":{"value":["KV cache quantization","INT2","rotation","spectral analysis"]},"abstract":{"value":"INT2 KV-cache quantization is attractive for long-context LLM serving, but it remains difficult to make both accurate and deployable. Simple rotations such as Hadamard transforms reduce outliers, but still degrade at INT2 because they are not aligned with downstream attention. We propose OSCAR, an ultra-low-bit KV-cache quantization method that estimates *attention-aware covariance* structures offline and uses them to derive fixed rotations and clipping thresholds for quantization. In this way, it aligns KV quantization with the covariance structures that attention actually consumes. More importantly, we not only provide theoretical justification but also develop a fully deployable OSCAR system with a custom INT2 attention kernel that remains compatible with paged KV-cache serving and fused kernel pipelines, enabling seamless integration into modern LLM serving frameworks such as SGLang and vLLM.\n\nWe evaluate OSCAR on recent reasoning models with reasoning traces of up to 32k tokens across five tasks. On Qwen3-4B-Thinking-2507 and Qwen3-8B, OSCAR reduces the BF16 accuracy gap to 3.78 and 1.42 points, respectively, while naive rotation INT2 collapses to nearly zero. We further scale OSCAR to Qwen3-32B and GLM-4.7 with 358B parameters, where it remains effectively on par with BF16. On long-context RULER-NIAH up to 128K, OSCAR remains robust on both Qwen3 models, while naive rotation INT2 collapses. System-wise, OSCAR reduces KV-cache memory by approximately 8×, improves throughput by up to 7× at large batch sizes under the same memory budget, and accelerates batch-size-1 decoding by up to 3× over BF16 due to reduced memory-bandwidth overhead."},"_bibtex":{"value":"@inproceedings{\nzhou2026oscar,\ntitle={{OSCAR}: Offline Spectral Covariance-Aware Rotation for 2-bit {KV} Cache Quantization},\nauthor={Zhongzhu Zhou and Donglin Zhuang and Jisen Li and Ziyan Chen and Shuaiwen Leon Song and Ben Athiwaratkun and Xiaoxia Wu},\nbooktitle={The 1st International Workshop on Sustainable and Efficient Language, Vision, and Action Models (SELVA) at ACL 2026},\nyear={2026},\nurl={https://openreview.net/forum?id=Y55s76ccZT}\n}"},"title":{"value":"OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization"},"pdf":{"value":"/pdf/3a5fc6393a9a42d0cd323ef92c9aa2fdc49c7290.pdf"},"venueid":{"value":"aclweb.org/ACL/2026/Workshop/SELVA"},"paperhash":{"value":"zhou|oscar_offline_spectral_covarianceaware_rotation_for_2bit_kv_cache_quantization"},"authorids":{"value":["~Zhongzhu_Zhou1","~Donglin_Zhuang1","~Jisen_Li2","~Ziyan_Chen2","~Shuaiwen_Leon_Song1","~Ben_Athiwaratkun1","~Xiaoxia_Wu1"]},"authors":{"value":["Zhongzhu Zhou","Donglin Zhuang","Jisen Li","Ziyan Chen","Shuaiwen Leon Song","Ben Athiwaratkun","Xiaoxia Wu"]}},"tmdate":1782025883892,"pdate":1782025882784,"tcdate":1779164695820,"writers":["aclweb.org/ACL/2026/Workshop/SELVA","aclweb.org/ACL/2026/Workshop/SELVA/Submission10/Authors"],"signatures":["aclweb.org/ACL/2026/Workshop/SELVA/Submission10/Authors"],"forum":"Y55s76ccZT","license":"CC BY 4.0","number":10,"cdate":1779164695820,"readers":["everyone"],"invitations":["aclweb.org/ACL/2026/Workshop/SELVA/-/Submission","aclweb.org/ACL/2026/Workshop/SELVA/-/Submission_Change_Before_Bidding","aclweb.org/ACL/2026/Workshop/SELVA/-/Submission_Change_Before_Reviewing","aclweb.org/ACL/2026/Workshop/SELVA/Submission10/-/Camera_Ready_Revision","aclweb.org/ACL/2026/Workshop/SELVA/-/Edit","aclweb.org/ACL/2026/Workshop/SELVA/-/Submission_Release"],"mdate":1782025883892,"odate":1782025882784,"domain":"aclweb.org/ACL/2026/Workshop/SELVA","id":"Y55s76ccZT","version":2},{"content":{"venue":{"value":"ICONIP 2025"},"pdf":{"value":"/pdf/9c0c73ecfad267490c6235c4b559fc20374ae9dc.pdf"},"venueid":{"value":"OpenReview.net/Archive"},"paperhash":{"value":"li|auv_efficient_kv_cache_eviction_for_llms_via_attention_score_aggregation_and_usage_count"},"authorids":{"value":["~Quanli_Li1","~Qinglin_Wang1","~Hongyang_Hu2","~Jie_Liu11"]},"html":{"value":"https://openreview.net/pdf?id=M9MtxFW30V"},"abstract":{"value":"As transformer-based large language models (LLMs) evolve, the need for efficient inference, particularly in managing the KV cache during the decoding phase, has intensified. However, the increasing sequence lengths in LLMs lead to substantial memory usage, which hinders their practical deployment. Existing cache eviction strategies, primarily based on accumulated attention scores, tend to prioritize early tokens, causing unstable performance and neglecting the integration of key metrics for dynamic cache management. To address these challenges, we propose AUV, an innovative cache eviction framework. AUV introduces a novel attention score aggregation method to mitigate the uneven eviction. It combines two complementary metrics: Total Attention Level and Strong Attention Frequency, by integrating the aggregated attention score with the usage count. Furthermore, AUV adopts a multi-step eviction strategy along with an eviction compensation mechanism to optimize both efficiency and accuracy. Finally, AUV improves KV cache management to avoid fragmentation. Through extensive experiments with OPT models, we demonstrate that AUV outperforms existing methods, including H2O, NACL, and full cache, under low cache budgets, achieving greater accuracy while maintaining throughput. Notably, AUV achieves a 10% point improvement over H2O in the BERTScore-F1 metric when retaining only 2% KV cache budget. These results highlight the potential of AUV to reduce memory consumption without sacrificing performance."},"title":{"value":"AUV: Efficient KV Cache Eviction for LLMs via Attention Score Aggregation and Usage Count"},"authors":{"value":["Quanli Li","Qinglin Wang","Hongyang Hu","Jie Liu"]}},"tmdate":1770024137375,"pdate":1761926400000,"tcdate":1770024137375,"writers":["~Quanli_Li1","~Qinglin_Wang1","~Hongyang_Hu2","~Jie_Liu11"],"signatures":["~Hongyang_Hu2"],"forum":"opuk5yOrW9","license":"CC BY 4.0","number":44617,"cdate":1770024137375,"readers":["everyone"],"invitations":["OpenReview.net/Archive/-/Direct_Upload"],"mdate":1770024137375,"domain":"OpenReview.net/Archive","id":"opuk5yOrW9","version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2511.04686v1"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"poudel|stateful_kv_cache_management_for_llms_balancing_space_time_accuracy_and_positional_fidelity"},"html":{"value":"https://doi.org/10.48550/arXiv.2511.04686"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2511-04686,\n publtype={informal},\n author={Pratik Poudel},\n title={Stateful KV Cache Management for LLMs: Balancing Space, Time, Accuracy, and Positional Fidelity},\n year={2025},\n month={November},\n cdate={1761955200000},\n journal={CoRR},\n volume={abs/2511.04686},\n url={https://doi.org/10.48550/arXiv.2511.04686}\n}\n"},"abstract":{"value":"The Key-Value (KV) cache is integral to efficient autoregressive inference in large language models (LLMs), yet its unbounded growth in stateful multi-turn scenarios presents major challenges. This paper examines the interplay between KV cache management strategies, the architectural context limits of models like meta-llama/Meta-Llama-3-8b-instruct, and the often-overlooked integrity of positional encodings. Through empirical analysis using a stateful benchmarking framework, we show that LLM generation quality degrades sharply when the accumulated KV cache approaches or exceeds the model's trained context window (e.g., 8192 tokens for Llama 3), a failure mode distinct from GPU memory exhaustion. Common eviction strategies, even high-retention ones (e.g., 99% via AttentionTop), can worsen performance if they disrupt positional coherence. Because LLMs rely on consistent positional signals (e.g., RoPE), compacting a cache by removing non-contiguous tokens can scramble these signals and lead to degenerative outputs. We further show that simple strategies preserving contiguous context blocks (e.g., keeping an initial \"gist\") can yield more coherent generations than complex or positionally disruptive ones. We advocate for eviction techniques that respect architectural limits, preserve positional structure, and view \"cache health\" holistically beyond mere size."},"title":{"value":"Stateful KV Cache Management for LLMs: Balancing Space, Time, Accuracy, and Positional Fidelity"},"authors":{"value":[{"fullname":"Pratik Poudel","username":"~Pratik_Poudel1"}]}},"tmdate":1778098594924,"pdate":1767139200000,"externalIds":["dblp:journals/corr/abs-2511-04686"],"tcdate":1778098590052,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Pratik_Poudel1"],"forum":"zNNiQwhr8F","license":"CC BY-SA 4.0","number":10715,"cdate":1761955200000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1778098594924,"domain":"OpenReview.net/Public_Article","id":"zNNiQwhr8F","version":2},{"content":{"venue":{"value":"INFOCOM 2025"},"pdf":{"value":"https://ieeexplore.ieee.org/iel8/11044418/11044439/11044533.pdf"},"venueid":{"value":"dblp.org/conf/INFOCOM/2025"},"paperhash":{"value":"liu|mell_memoryefficient_large_language_model_serving_via_multigpu_kv_cache_management"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Qianli_Liu:","https://dblp.org/search/pid/api?q=author:Zicong_Hong:","https://dblp.org/search/pid/api?q=author:Peng_Li_0017:","~Fahao_Chen1","https://dblp.org/search/pid/api?q=author:Song_Guo_0001:"]},"html":{"value":"https://doi.org/10.1109/INFOCOM55648.2025.11044533"},"_bibtex":{"value":"@inproceedings{DBLP:conf/infocom/LiuHLC025,\n author={Qianli Liu and Zicong Hong and Peng Li and Fahao Chen and Song Guo},\n title={Mell: Memory-Efficient Large Language Model Serving via Multi-GPU KV Cache Management},\n year={2025},\n cdate={1735689600000},\n pages={1-10},\n url={https://doi.org/10.1109/INFOCOM55648.2025.11044533},\n booktitle={INFOCOM},\n crossref={conf/infocom/2025}\n}\n"},"abstract":{"value":"Serving large language models (LLMs) for massive users is challenged by the significant memory footprint of the transient state, known as the key-value (KV) cache, which scales with sequence length and number of requests. Instead of renting or buying more expensive GPUs, the load imbalance of the KV cache across GPUs, coupled with recent advances in inter-GPU communication, provides an opportunity to serve more requests via request migration. However, high migration overhead and unpredictable request patterns make it challenging. Therefore, this paper proposes Mell, a memory-efficient LLM serving system via multi-GPU KV cache management. It saves the number of GPUs needed in the system by considering the dynamic KV cache load and the costly request migration. Specifically, we first develop an adaptive request migration mechanism to balance the computational and communication overheads and adapt to diverse resource conditions. Then, we design an online algorithm tailored to a multi-LLM request and multi-GPU scheduling problem with migration enabled. It aims to minimise the required GPUs while limiting the number of migrations. Finally, we implement a prototype of Mell and demonstrate that it reduces the number of GPUs by 31% and increases the GPU utilization by 43% at most compared to existing LLM serving systems."},"title":{"value":"Mell: Memory-Efficient Large Language Model Serving via Multi-GPU KV Cache Management"},"authors":{"value":["Qianli Liu","Zicong Hong","Peng Li","Fahao Chen","Song Guo"]}},"tmdate":1769341993635,"pdate":1767139200000,"externalIds":["dblp:conf/infocom/LiuHLC025"],"tcdate":1769341979224,"writers":["~"],"signatures":["~Fahao_Chen1"],"forum":"4X1lYqian4","license":"CC BY-SA 4.0","number":796843,"cdate":1735689600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1769341993635,"domain":"DBLP.org","id":"4X1lYqian4","version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2501.06709v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"qianli|mell_memoryefficient_large_language_model_serving_via_multigpu_kv_cache_management"},"authorids":{"value":["https://dblp.org/search/pid/api?q=author:Liu_Qianli:","https://dblp.org/search/pid/api?q=author:Zicong_Hong:","~Fahao_Chen1","https://dblp.org/search/pid/api?q=author:Peng_Li_0017:","https://dblp.org/search/pid/api?q=author:Song_Guo_0001:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2501.06709"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2501-06709,\n publtype={informal},\n author={Liu Qianli and Zicong Hong and Fahao Chen and Peng Li and Song Guo},\n title={Mell: Memory-Efficient Large Language Model Serving via Multi-GPU KV Cache Management},\n year={2025},\n month={January},\n cdate={1735689600000},\n journal={CoRR},\n volume={abs/2501.06709},\n url={https://doi.org/10.48550/arXiv.2501.06709}\n}\n"},"abstract":{"value":"Serving large language models (LLMs) for massive users is challenged by the significant memory footprint of the transient state, known as the key-value (KV) cache, which scales with sequence length and number of requests. Instead of renting or buying more expensive GPUs, the load imbalance of the KV cache across GPUs, coupled with recent advances in inter-GPU communication, provides an opportunity to serve more requests via request migration. However, high migration overhead and unpredictable request patterns make it challenging. Therefore, this paper proposes MELL, a memory-efficient LLM serving system via multi-GPU KV cache management. It saves the number of GPUs needed in the system by considering the dynamic KV cache load and the costly request migration. Specifically, we first develop an adaptive request migration mechanism to balance the computational and communication overheads and adapt to diverse resource conditions. Then, we design an online algorithm tailored to a multi-LLM request and multi-GPU scheduling problem with migration enabled. It aims to minimise the required GPUs while limiting the number of migrations. Finally, we implement a prototype of MELL and demonstrate that it reduces the number of GPUs by 31% and increases the GPU utilization by 43% at most compared to existing LLM serving systems."},"title":{"value":"Mell: Memory-Efficient Large Language Model Serving via Multi-GPU KV Cache Management"},"authors":{"value":["Liu Qianli","Zicong Hong","Fahao Chen","Peng Li","Song Guo"]}},"tmdate":1769341984142,"pdate":1767139200000,"externalIds":["dblp:journals/corr/abs-2501-06709"],"tcdate":1769341977664,"writers":["~"],"signatures":["~Fahao_Chen1"],"forum":"8zAuO4KWtS","license":"CC BY-SA 4.0","number":796831,"cdate":1735689600000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1769341984142,"domain":"DBLP.org","id":"8zAuO4KWtS","version":2},{"content":{"TLDR":{"value":"Instead of asking which cached tokens mattered while reading the prompt, \\textbf{LORE-KV} samples plausible futures to estimate which tokens will matter while generating the answer, enabling training-free, future-query-aware KV-cache eviction."},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["KV Cache Compression; KV Cache Eviction; Efficient LLM Inference; Long-Context LLMs; Future-Query Estimation; Monte-Carlo Estimation; Test-Time Computation; Memory-Efficient Transformers; Attention; Large Language Models."]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Most KV-cache eviction methods ask, in effect, which memory appeared important while reading the prompt? We instead ask, which memory will matter while answering? Since decoding queries are unavailable at eviction time, prior future-aware methods rely on pseudo-responses or synthetic future-query estimates. We cast fixed-budget future-aware eviction as distributional estimation over plausible model-conditional query trajectories and introduce LORE-KV (Lookahead Output-perturbation with Reliability-weighted Ensembles for Key-Value caches), a training-free method that samples short autoregressive continuations from the frozen target model and uses their response-side query states to estimate prompt-token utility. Tokens are scored by projected leave-one-out attention-output deletion cost and aggregated across sampled futures with optional trajectory weighting. The temporary continuations are discarded before final decoding, requiring no auxiliary model or training. Ablations isolate the mechanism: at B=128, a single response-side continuation recovers about 89% of the gain over the prompt-window control, while additional futures provide smaller improvements. At B=128, LORE-KV raises the LongBench average on Qwen2.5-14B from 45.49 to 48.24 (+2.75) and the 16K RULER average on Mistral-7B from 45.20 to 51.05 (+5.85). Gains diminish at larger cache budgets and coexist with task-level regressions. LORE-KV incurs 1.46–2.77× AnDPro's per-sample wall-clock time as a one-time compression overhead across six dense and hybrid-attention backbones."},"_bibtex":{"value":"@inproceedings{\nanonymous2026monte,\ntitle={Monte Carlo Future-Query Estimation for {KV} Cache Eviction},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=n6gVwCPzso},\nnote={under review}\n}"},"title":{"value":"Monte Carlo Future-Query Estimation for KV Cache Eviction"},"pdf":{"value":"/pdf/ffdd51ba70b9a35de929853c6618153f8986dc64.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791233828824,"tcdate":1789751495075,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission46428/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission46428/Authors"],"forum":"n6gVwCPzso","license":"CC BY 4.0","number":46428,"cdate":1789751495075,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission46428/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791233828824,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"n6gVwCPzso","version":2},{"content":{"venue":{"value":"ICML 2026 regular"},"keywords":{"value":["Clustering","K-Means","Vector Quantization","KV-Cache Quantization"]},"_bibtex":{"value":"@inproceedings{\nkim2026gsrq,\ntitle={{GSRQ}: Gain-Shape Residual Quantization for Sub-1-bit {KV} Cache},\nauthor={Soosung Kim and Minjae Park and Eui-Young Chung and Jaeyong Chung},\nbooktitle={Forty-third International Conference on Machine Learning},\nyear={2026},\nurl={https://openreview.net/forum?id=HoRGEhSUdN}\n}"},"title":{"value":"GSRQ: Gain-Shape Residual Quantization for Sub-1-bit KV Cache"},"paperhash":{"value":"kim|gsrq_gainshape_residual_quantization_for_sub1bit_kv_cache"},"originally_submitted_PDF":{"value":"/pdf/b9f32508202f04c69dc34bbef47aff6946907e1a.pdf"},"TLDR":{"value":"We propose Gain–Shape Residual Quantization (GSRQ) that improves directional fidelity, enabling substantial accuracy gains in sub-1-bit KV cache quantization."},"primary_area":{"value":"deep_learning->large_language_models"},"abstract":{"value":"The deployment of Large Language Models (LLMs) with extended context windows is increasingly constrained by the linear growth of Key-Value (KV) cache memory.\nVector Quantization (VQ), particularly Residual Quantization (RQ), is a promising approach for pushing KV cache storage toward the sub-1-bit regime by progressively encoding residuals with small codebooks.\nHowever, most VQ methods still rely on standard $\\ell_2$ $K$-means as the core codebook-learning primitive.\nWe identify a subtle high-dimensional issue of this primitive: Euclidean centroid averaging can induce centroid shrinkage, which weakens the angular alignment term in the $\\ell_2$ distortion and makes directional preservation harder.\nTo address this issue, we propose Gain-Shape $K$-means (GSKM), a drop-in replacement for $K$-means that improves directional fidelity while matching, and in some regimes improving, $\\ell_2$ distortion.\nWe then build Gain-Shape Residual Quantization (GSRQ) by incorporating a weighted extension of GSKM into an RQ pipeline.\nOn LLaMA-3-8B, GSRQ substantially improves over strong KV cache quantization baselines across bit rates.\nAt 1-bit, it improves the average accuracy across LongBench tasks from 11.34 to 33.54, a gain of 22.20 percentage points over VQLLM."},"pdf":{"value":"/pdf/1422253d8df69d2fd83cc0072584155b2d42b446.pdf"},"lay_summary":{"value":"Large language models can read and generate long documents, but doing so requires them to remember information from all previous words. This memory, called the key-value cache, grows linearly with the length of the input and can quickly become too large for efficient use on GPUs.\n\nA common way to reduce this memory is to compress groups of numbers into a small set of representative patterns. However, existing methods often learn these representatives by simple averaging. In high-dimensional data, this averaging can shrink the representatives and make them less aligned to the original directions of the data, which is important for preserving model behavior.\n\nWe propose a new compression method that separates each group representative into two parts-its size and its direction-and refines these two parts separately. This helps preserve directional information while still keeping the memory footprint very small. Our approach improves the quality of compressed key-value caches, especially in extremely low-bit settings.\n\nAs a result, large language models can handle longer contexts with much lower memory cost while maintaining stronger performance on reasoning and long-document tasks."},"venueid":{"value":"ICML.cc/2026/Conference"},"authorids":{"value":["~Soosung_Kim2","~Minjae_Park2","~Eui-Young_Chung1","~Jaeyong_Chung2"]},"authors":{"value":["Soosung Kim","Minjae Park","Eui-Young Chung","Jaeyong Chung"]}},"tmdate":1790067354209,"pdate":1777576440185,"tcdate":1769059967633,"writers":["ICML.cc/2026/Conference","ICML.cc/2026/Conference/Submission13966/Authors"],"signatures":["ICML.cc/2026/Conference/Submission13966/Authors"],"forum":"HoRGEhSUdN","license":"CC BY 4.0","number":13966,"cdate":1769059967633,"readers":["everyone"],"invitations":["ICML.cc/2026/Conference/-/Submission","ICML.cc/2026/Conference/-/Post_Submission","ICML.cc/2026/Conference/Submission13966/-/Full_Submission","ICML.cc/2026/Conference/-/Edit","ICML.cc/2026/Conference/Submission13966/-/Camera_Ready_Revision"],"mdate":1790067354209,"odate":1782341951867,"domain":"ICML.cc/2026/Conference","id":"HoRGEhSUdN","version":2},{"content":{"venue":{"value":"ICLR 2027 Conference Submission"},"TLDR":{"value":"A learnable KV eviction gate lives inside the softmax normalizer, and that same normalizer makes dense attention a no-op its own loss prefers; remove that preference and no extra machinery is needed."},"keywords":{"value":["KV cache eviction"]},"supplementary_material":{"value":"/attachment/4ce3bb37e52b94dd61d1fd761c344eba3d105351.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"KV cache eviction must decide which entries carry information that the rest of the cache can no longer supply. A natural way to estimate this is a learnable gate: give each cached entry a gate on its attention weight and train the gate on the model's own objective. The gate belongs inside the normalizing softmax, where suppressing one entry hands its mass to the others rather than discarding it. That same normalizer absorbs any per-query constant, so a constant gate cancels and reproduces the frozen model exactly. Trained on the language-modeling loss the gate stays near a constant score while its loss falls throughout: optimization succeeds, but a frozen backbone was fit for dense attention, so its loss prefers the no-op, and a near-constant score ranks nothing. The no-op is not a floor the gate improves from but what its objective asks for. A second obstacle survives once the gate moves, since the same gradient charges the gate for errors the frozen model already makes, which no reweighting of positions separates out. Much of the machinery in learnable eviction, from sparsity penalties to relaxations of the discrete decision, works around the first obstacle rather than removing it; both are removed by putting dense attention beyond the gate's reach and making the dense model's predictions its target. Under these two conditions capacity is never the missing ingredient: neither a larger scorer, nor explicit access to the past, nor a trainable backbone helps. The same holds for what eviction unavoidably discards, which is recovered by parameter-free centroids read inside the same softmax far better than by any memory read outside it, since the mass they must return is a share of that normalizer and available nowhere else. At 75\\% KV cache compression, this gate reaches 90.14 on RULER at 8K and 46.03 on LongBench with Qwen3-8B, 3.72 and 1.36 points below full-cache inference."},"_bibtex":{"value":"@inproceedings{\nanonymous2026the,\ntitle={The Gating Mechanism of {KV} Cache Eviction},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=PwxthHRt7s},\nnote={under review}\n}"},"title":{"value":"The Gating Mechanism of KV Cache Eviction"},"pdf":{"value":"/pdf/b565e061a27cbead9f9c0e1ec61ac730434d54d6.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791231504204,"tcdate":1789655280978,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission32212/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission32212/Authors"],"forum":"PwxthHRt7s","license":"CC BY 4.0","number":32212,"cdate":1789655280978,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission32212/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791231504204,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"PwxthHRt7s","version":2},{"content":{"venue":{"value":"ACL ARR 2026 October Submission"},"research_area_keywords":{"value":"LLM efficiency"},"keywords":{"value":["KV compression","KV eviction","Query-aware cache eviction"]},"_bibtex":{"value":"@inproceedings{\nanonymous2026afterglow,\ntitle={Afterglow: Propagating Importance Forward to Retain Answer Spans in Query-Aware {KV} Cache Eviction},\nauthor={Anonymous},\nbooktitle={Submitted to ACL Rolling Review - October 2026},\nyear={2026},\nurl={https://openreview.net/forum?id=w2RJU9nP89},\nnote={under review}\n}"},"title":{"value":"Afterglow: Propagating Importance Forward to Retain Answer Spans in Query-Aware KV Cache Eviction"},"contribution_types":{"value":"Approaches to low-resource settings"},"abstract":{"value":"The key--value (KV) cache bottlenecks long-context inference in LLMs through its memory footprint and attention cost. Query-aware eviction removes KV-cache entries that attention-based scores deem unimportant to the query. We find that these scores locate where a relevant span starts but not where it ends: attention concentrates on the span's first tokens and fades over the rest, so eviction retains later tokens in only a few KV heads. These sparsely retained tokens are often copied incorrectly, leading to truncated or corrupted answers.\n\nWe propose Afterglow, a training-free plug-in that augments the scoring mechanisms of existing KV-cache eviction methods.\nAfterglow sets each token's score to the maximum of its own score and the decayed scores of preceding tokens.\nOn RULER-4K needle retrieval at 87.5\\% eviction, Afterglow improves four methods on Qwen3-8B and Llama-3.1-8B from 49--74 to 98--100, within 1.8 points of the full cache.\nThis holds from 4K to 64K tokens and from 4B to 14B parameters.\nAt the same eviction rate, Afterglow raises the four methods by 17--30 points over all 13 RULER-4K tasks, closing 70--91\\% of the gap to the full cache, and matches or improves LongBench.\nIt also improves retrieval from real tool outputs."},"paper_type":{"value":"Short"},"pdf":{"value":"/pdf/e01bd5e07a3b22993df161faf0c1c5202190988f.pdf"},"research_area":{"value":"Efficient Methods for NLP"},"venueid":{"value":"aclweb.org/ACL/ARR/2026/October/Submission"}},"tmdate":1792185019827,"tcdate":1791840594466,"writers":["aclweb.org/ACL/ARR/2026/October","aclweb.org/ACL/ARR/2026/October/Submission7676/Authors"],"signatures":["aclweb.org/ACL/ARR/2026/October/Submission7676/Authors"],"forum":"w2RJU9nP89","license":"CC BY 4.0","number":7676,"cdate":1791840594466,"readers":["everyone"],"invitations":["aclweb.org/ACL/ARR/2026/October/-/Submission","aclweb.org/ACL/ARR/2026/October/-/Responsible_NLP_Checklist_Status","aclweb.org/ACL/ARR/2026/October/-/Edit","aclweb.org/ACL/ARR/2026/October/-/Preprint_Post_Submission"],"mdate":1792185019827,"odate":1791894541772,"domain":"aclweb.org/ACL/ARR/2026/October","id":"w2RJU9nP89","version":2},{"content":{"TLDR":{"value":"High-Throughput Long-Context LLM Inference System"},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Long-Context LLM Inference","KV Cache Optimization"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"With the widespread deployment of long-context large language models (LLMs), there has been a growing demand for efficient support of high-throughput inference. However, as the key-value (KV) cache expands with the sequence length, the increasing memory footprint and the need to access it for each token generation both result in low throughput when serving long-context LLMs. While various dynamic sparse attention methods have been proposed to speed up inference while maintaining generation quality, they either fail to sufficiently reduce GPU memory consumption or introduce significant decoding latency by offloading the KV cache to the CPU. We present ShadowKV, a high-throughput long-context LLM inference system that stores the low-rank key cache and offloads the value cache to reduce the memory footprint for larger batch sizes and longer sequences. To minimize decoding latency, ShadowKV employs an accurate KV selection strategy that reconstructs minimal sparse KV pairs on-the-fly. By evaluating ShadowKV on a broad range of benchmarks, including RULER, LongBench, and Needle In A Haystack, and models like Llama-3.1-8B, Llama-3-8B-1M, GLM-4-9B-1M, Yi-9B-200K, Phi-3-Mini-128K, and Qwen2-7B-128K, we demonstrate that it can support up to 6$\\times$ larger batch sizes and boost throughput by up to 3.04$\\times$ on an A100 GPU without sacrificing accuracy, even surpassing the performance achievable with infinite batch size under the assumption of infinite GPU memory."},"_bibtex":{"value":"@misc{\nsun2025shadowkv,\ntitle={Shadow{KV}: {KV} Cache in Shadows for High-Throughput Long-Context {LLM} Inference},\nauthor={Hanshi Sun and Li-Wen Chang and Wenlei Bao and Size Zheng and Ningxin Zheng and Harry Dong and Yuejie Chi and Beidi Chen},\nyear={2025},\nurl={https://openreview.net/forum?id=vHO9mU87dc}\n}"},"title":{"value":"ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference"},"pdf":{"value":"/pdf/6b1d95e6b28072cedf40a305fa97ca33b223a94f.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"sun|shadowkv_kv_cache_in_shadows_for_highthroughput_longcontext_llm_inference"},"authorids":{"value":["~Hanshi_Sun1","~Li-Wen_Chang1","~Wenlei_Bao1","~Size_Zheng1","~Ningxin_Zheng1","~Harry_Dong1","~Yuejie_Chi1","~Beidi_Chen1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Hanshi Sun","Li-Wen Chang","Wenlei Bao","Size Zheng","Ningxin Zheng","Harry Dong","Yuejie Chi","Beidi Chen"]}},"tmdate":1738735649660,"tcdate":1726959742865,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission2463/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission2463/Authors"],"forum":"vHO9mU87dc","license":"CC BY 4.0","number":2463,"cdate":1726959742865,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/-/Submission","ICLR.cc/2025/Conference/-/Post_Submission","ICLR.cc/2025/Conference/Submission2463/-/Full_Submission","ICLR.cc/2025/Conference/Submission2463/-/Rebuttal_Revision","ICLR.cc/2025/Conference/-/Edit"],"mdate":1738735649660,"odate":1728008565725,"domain":"ICLR.cc/2025/Conference","id":"vHO9mU87dc","version":2},{"content":{"venue":{"value":"ACL ARR 2026 October Submission"},"research_area_keywords":{"value":"quantization, LLM efficiency"},"keywords":{"value":["Model Quantization","KV Cache Compression"]},"_bibtex":{"value":"@inproceedings{\nanonymous2026smoothkv,\ntitle={Smooth{KV}: Fine-Grained Distribution Smoothing for Accurate {KV} Cache Quantization},\nauthor={Anonymous},\nbooktitle={Submitted to ACL Rolling Review - October 2026},\nyear={2026},\nurl={https://openreview.net/forum?id=9RHaDXADt8},\nnote={under review}\n}"},"title":{"value":"SmoothKV: Fine-Grained Distribution Smoothing for Accurate KV Cache Quantization"},"contribution_types":{"value":"Approaches to low-compute settings (efficiency)"},"abstract":{"value":"The Key-Value (KV) cache has become a major memory bottleneck for long-context inference in large language models (LLMs), yet its aggressive low-bit quantization remains challenging due to highly non-uniform distributions and extreme values. In this paper, we propose SmoothKV, a distribution-aware smoothing framework for accurate low-bit KV cache quantization through three complementary techniques. First, we observe that pre-RoPE keys exhibit channel-wise distributions concentrated around distinct non-zero centers, whereas RoPE substantially mixes these distributions. SmoothKV therefore applies channel-wise mean shifting to pre-RoPE keys, yielding more compact and quantization-friendly distributions. Second, we introduce fine-grained rotation and reordering to progressively smooth channels with comparable scales and balance extreme values across attention heads. Third, we develop rank-based adaptive clipping to accommodate heterogeneous group-wise distributions and prevent isolated extreme values from unnecessarily expanding the quantization range. To enable efficient inference, we further develop dedicated kernels with Kronecker-structured transformations to reduce the overhead of pre-quantization processing. Comprehensive experiments show that SmoothKV maintains performance close to full precision on both long-context understanding and mathematical reasoning tasks, while substantially outperforming existing KV cache quantization methods, including KIVI, RotateKV, and VecInfer, at 2 bits. Moreover, SmoothKV achieves up to $2.4\\times$ decoding speedup and reduces peak memory usage by 81% over FP16."},"paper_type":{"value":"Long"},"pdf":{"value":"/pdf/4eea1a8f0d6ef25573ac62b0d09d6aa1a008d279.pdf"},"research_area":{"value":"Efficient Methods for NLP"},"venueid":{"value":"aclweb.org/ACL/ARR/2026/October/Submission"}},"tmdate":1792185024180,"tcdate":1791859122274,"writers":["aclweb.org/ACL/ARR/2026/October","aclweb.org/ACL/ARR/2026/October/Submission8452/Authors"],"signatures":["aclweb.org/ACL/ARR/2026/October/Submission8452/Authors"],"forum":"9RHaDXADt8","license":"CC BY 4.0","number":8452,"cdate":1791859122274,"readers":["everyone"],"invitations":["aclweb.org/ACL/ARR/2026/October/-/Submission","aclweb.org/ACL/ARR/2026/October/-/Responsible_NLP_Checklist_Status","aclweb.org/ACL/ARR/2026/October/-/Edit","aclweb.org/ACL/ARR/2026/October/-/Preprint_Post_Submission"],"mdate":1792185024180,"odate":1791894541772,"domain":"aclweb.org/ACL/ARR/2026/October","id":"9RHaDXADt8","version":2},{"content":{"summary":{"value":"This paper introduces LightKV, a training-free approach for reducing the KV cache size in Large Vision-Language Models (LVLMs). The study identifies that vision tokens are the primary source of GPU memory overhead during the prefill stage. To mitigate this issue, LightKV employs a prompt-guided cross-modal compression strategy, which aggregates redundant vision tokens via graph-based message passing guided by text-prompt attention. The method performs hierarchical compression across layers and within local spatial windows, progressively minimizing KV cache size while retaining visually and semantically important information. Extensive experiments on eight LVLMs (including LLaVA-v1.5, LLaVA-NeXT, InternVL2, and Qwen2.5-VL) demonstrate that LightKV can halve KV cache memory usage and reduce computation by up to 40%, all while maintaining—or even slightly improving—accuracy on diverse benchmarks such as COCO, MME, SeedBench, and VizWiz."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"1. In Line 203, it is unclear what FD stands for. It would be clearer and more reader-friendly to spell out the full term (e.g., Feature Divergence) the first time it appears, rather than introducing it directly as an abbreviation. \n\n2. Many implementation details of the proposed method are unclear. For example, during graph construction, it is not specified how the set of nodes is divided into two subsets. Is there any criterion or heuristic guiding this split, or does the method simply follow the same strategy as ToMe (Bolya et al., 2023)? Providing clarification on this point would improve the method’s reproducibility and clarity. \n\n3. The proposed method adopts a bipartite matching strategy, similar to ToMe (Bolya et al., 2023), and demonstrates improved performance. However, the paper does not clearly discuss the computational complexity of the proposed approach relative to ToMe. Providing a detailed comparison or analysis of the computational cost would help clarify whether the performance gains come at the expense of higher complexity. \n\n4. The paper only evaluates on image-based benchmarks. It would be valuable to assess the method on video understanding datasets (e.g., Video-MME [1], MVBench [2]) to verify whether LightKV can handle temporal redundancy and long-context multimodal inputs effectively. \n\n5. While the paper claims notable efficiency improvements, it lacks a detailed runtime or wall-clock latency analysis, particularly comparing the overhead introduced by graph construction and message passing with that of baseline token-pruning methods. Providing such a breakdown would offer a clearer picture of the true computational trade-offs and strengthen the paper’s efficiency claims. \n\n \n\nReference: \n\n[1] Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. CVPR 2025. \n\n[2] Mvbench: A comprehensive multi-modal video understanding benchmark. CVPR 2024."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"This paper addresses a practical and important efficiency bottleneck in LVLMs—the excessive KV cache size from vision token. The proposed LightKV is training-free and can be seamlessly applied to various existing LVLM architectures without retraining, demonstrating strong generality."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The paper is somewhat difficult to follow, and the motivation behind the proposed method is not clearly articulated. It remains unclear why graph message passing is selected as the core mechanism for vision token compression. The authors should provide clearer intuition or justification for this design choice, explaining how it specifically contributes to improving efficiency or preserving cross-modal information compared with alternative strategies. In addition, the paper introduces an excessive number of notations, which overcomplicates the presentation and makes the method harder to understand. \n\n2. The main limitation of this paper is that it relies on attention weights to estimate token importance, which is incompatible with I/O-optimized implementations such as FlashAttention (Dao et al., 2022) that are widely adopted in long-context LVLMs. Although the authors note that attention scores are computed only in a small subset of layers, this still incurs additional computational overhead and may limit the scalability and practicality of the approach, especially for long-context inference or large-scale deployments where memory efficiency is critical."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762923047076,"tcdate":1761963315154,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission12077/Reviewer_FEHa"],"signatures":["ICLR.cc/2026/Conference/Submission12077/Reviewer_FEHa"],"forum":"U9OKWwkxuz","number":3,"license":"CC BY 4.0","cdate":1761963315154,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission12077/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762923047076,"domain":"ICLR.cc/2026/Conference","replyto":"U9OKWwkxuz","id":"bejGYXLsJZ","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["LVLM","efficiency","KV cache"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"Key-Value (**KV**) cache has become a _de facto_ component of modern Large Vision-Language Models (**LVLM**s) for inference.\nWhile it enhances decoding efficiency in Large Language Models (**LLMs**), its direct adoption in LVLMs introduces substantial GPU memory overhead due to the large number of vision tokens processed during the prefill stage.\nTo tackle this problem, we propose LightKV, a novel approach that reduces KV cache size by exploiting the redundancy among vision-token embeddings.\nGuided by text prompts, LightKV employs cross-modality message passing to aggregate informative messages across vision tokens and progressively compress them during prefill.\nThis prompt-aware guidance distinguishes our method from prior vision-only compression strategies.\nWe evaluate LightKV on eight open-source LVLMs across eight public benchmarks, such as MME and SeedBench.\nExperimental results demonstrate that with only 50% of the original vision tokens, LightKV (i) halves KV cache size, (ii) reduces computation by up to 40%, and (iii) preserves general-purpose performance while significantly outperforming existing baselines."},"_bibtex":{"value":"@misc{\nchen2025make,\ntitle={Make Your {LVLM} {KV} Cache More Lightweight},\nauthor={Xihao Chen and Yangyang Guo and Roger Zimmermann},\nyear={2025},\nurl={https://openreview.net/forum?id=U9OKWwkxuz}\n}"},"title":{"value":"Make Your LVLM KV Cache More Lightweight"},"pdf":{"value":"/pdf/e6c2d59862f67ffebc2c17372c9e9806d4961b4c.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"chen|make_your_lvlm_kv_cache_more_lightweight"},"authorids":{"value":["~Xihao_Chen2","~Yangyang_Guo1","~Roger_Zimmermann1"]},"authors":{"value":["Xihao Chen","Yangyang Guo","Roger Zimmermann"]}},"version":2},{"content":{"TLDR":{"value":"CASAK-V: A learned approach for dynamic sparse attention and KV-cache compression, enabling efficient long-context LLM inference in memory-constrained environments with minimal performance loss."},"venue":{"value":"ICLR 2025 Conference Desk Rejected Submission"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Large Language Models","Sparse Attention","KV-cache Compression","Long-context Processing","Meta-learning","Adaptive Algorithms","Memory Efficiency","Inference Optimization","On-device Deployment","Context-aware Models","Dynamic Attention","Transformer Architectures","Efficient Natural Language Processing","Machine Learning Systems","Attention Mechanisms","Sparse Computation","Benchmarking","Model Compression","Resource-constrained Computing","Edge AI","Computational Complexity","Information Retrieval","Self-attention","Transfer Learning","Deep Learning","Artificial Intelligence","Chunk-wise Compression","Pattern Recognition"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"reciprocal_reviewing":{"value":"I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6."},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The emergence of long-context Large Language Models (LLMs) has triggered a rapid expansion of applications across various domains. However, these models remain inaccessible for on-device or on-premises deployments due to significant computational and memory challenges. The quadratic complexity of attention mechanisms and the substantial memory requirements of KV-caches, hinder adoption in resource-constrained environments. Current solutions, such as sparse attention mechanisms and KV-cache compression techniques, often rely on pre-observed patterns or context-independent, head-specific profiling strategies, which can compromise model accuracy, especially in long-context processing. This paper introduces Context-Aware adaptive Sparse Attention with Key-Value cache compression (CASAK-V), an inference-time approach that dynamically generates and applies head-specific sparse attention patterns. CASAK-V leverages a meta-learning framework to fine-tune a compact pre-trained vision-language encoder-decoder transformer for sparse pattern identification from per-layer attention scores. These patterns include fixed local windows, dynamic column stripes, block-sparse, and various other learned hybrid configurations. The technique additionally implements adaptive chunk-wise KV-cache compression using policies adapted from these layer-wise sparse configurations. To retain context-awareness, these configuration are dynamically adjusted during token generation, based on an attention map reconstruction heuristic. Our evaluations show that CASAK-V achieves minimal performance degradation on long-context benchmarks (LongBench), while reducing memory usage by 40% and delivering near-linear runtime complexity compared to full attention and caching. In summary, CASAK-V enables efficient long-context processing in memory-limited environments, extending the applicability of LLMs and facilitating their deployment in on-premises and on-device scenarios."},"_bibtex":{"value":"@misc{\nmohammed2024casakv,\ntitle={{CASAK}-V: Dynamic Sparse Attention and Adaptive {KV}-Cache Compression for Memory-Efficient Long-Context {LLM} Inference},\nauthor={Hamza Mohammed and Sai Chand Boyapati and Hang Yin},\nyear={2024},\nurl={https://openreview.net/forum?id=n7RqgqbxP7}\n}"},"title":{"value":"CASAK-V: Dynamic Sparse Attention and Adaptive KV-Cache Compression for Memory-Efficient Long-Context LLM Inference"},"pdf":{"value":"/pdf/482e6b0a8f32d4f7720a0916085a4197fc9efb57.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Desk_Rejected_Submission"},"paperhash":{"value":"mohammed|casakv_dynamic_sparse_attention_and_adaptive_kvcache_compression_for_memoryefficient_longcontext_llm_inference"},"authorids":{"value":["~Hamza_Mohammed1","~Sai_Chand_Boyapati2","~Hang_Yin9"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Hamza Mohammed","Sai Chand Boyapati","Hang Yin"]}},"tmdate":1729484358932,"tcdate":1727522729267,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission14192/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission14192/Authors"],"forum":"n7RqgqbxP7","license":"CC BY 4.0","number":14192,"cdate":1727522729267,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/-/Submission","ICLR.cc/2025/Conference/-/Post_Submission","ICLR.cc/2025/Conference/Submission14192/-/Full_Submission","ICLR.cc/2025/Conference/-/Desk_Rejected_Submission"],"mdate":1729484358932,"odate":1728008565725,"domain":"ICLR.cc/2025/Conference","id":"n7RqgqbxP7","version":2},{"content":{"TLDR":{"value":"GeoKV compresses KV caches by selecting structurally valuable tokens and preserving their global geometry through lightweight key rectification."},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["KV Cache Compression;Vision-Language Models;Geometric Representation;Token Selection;Key Rectification"]},"supplementary_material":{"value":"/attachment/0f494f6b9f73ea5ce6db26301c5bac2405e7da3f.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Vision-Language Models (VLMs) have demonstrated remarkable capabilities, yet the rapidly growing memory consumption of Key-Value (KV) caches poses a major challenge for efficient inference. Existing KV cache compression methods predominantly characterize token importance from a query-dependent functional perspective, such as attention or local relevance, while overlooking the global structural role of cached representations. Consequently, tokens with low current-query relevance may still be important for preserving underrepresented directions and the global geometry of the cached-key space. To address this limitation, we propose GeoKV, a global-structure-aware framework for KV cache compression. Specifically, we introduce Global Structural Contribution (GSC), a query-independent criterion derived from the marginal gain of a regularized log-determinant objective, and perform greedy selection to preserve structurally valuable tokens. We further introduce a lightweight geometry-aware key rectification module based on a low-rank feature-space transformation to mitigate compression-induced geometric distortion. Extensive experiments on VLMs demonstrate that GeoKV effectively reduces KV cache size while maintaining model performance. Comprehensive analyses further show that structural contribution captures complementary information beyond query-dependent functional relevance."},"_bibtex":{"value":"@inproceedings{\nanonymous2026geokv,\ntitle={Geo{KV}: Geometry-Aware {KV} Cache Compression via Global Structural Contribution and Key Rectification},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=yQqanj3O42},\nnote={under review}\n}"},"title":{"value":"GeoKV: Geometry-Aware KV Cache Compression via Global Structural Contribution and Key Rectification"},"pdf":{"value":"/pdf/7b61d0fcff33112d1ec36cca5e6b502de06f6023.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791234975620,"tcdate":1789796098060,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission54955/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission54955/Authors"],"forum":"yQqanj3O42","license":"CC BY 4.0","number":54955,"cdate":1789796098060,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission54955/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791234975620,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"yQqanj3O42","version":2},{"content":{"venue":{"value":"CoRR 2026"},"pdf":{"value":"https://arxiv.org/pdf/2604.26557v1"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"jeong|dualblade_dualpath_nvmedirect_kvcache_offloading_for_edge_llm_inference"},"html":{"value":"https://doi.org/10.48550/arXiv.2604.26557"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2604-26557,\n publtype={informal},\n author={Bodon Jeong and Hongsu Byun and Youngjae Kim and Weikuan Yu and Kyungkeun Lee and Jihoon Yang and Sungyong Park},\n title={DUAL-BLADE: Dual-Path NVMe-Direct KV-Cache Offloading for Edge LLM Inference},\n year={2026},\n month={April},\n cdate={1775001600000},\n journal={CoRR},\n volume={abs/2604.26557},\n url={https://doi.org/10.48550/arXiv.2604.26557}\n}\n"},"abstract":{"value":"The increasing deployment of Large Language Model (LLM) inference on edge AI systems demands efficient execution under tight memory budgets. A key challenge arises from Key-Value (KV) caches, which often exceed available device memory. Although NVMe-based offloading offers scalable capacity, existing file-based designs rely heavily on the kernel page cache, leading to cache thrashing, unpredictable latency, and high software overhead under memory pressure. We present DUAL-BLADE, a dual-path KV residency framework that dynamically assigns KV tensors to either a page-cache path or an NVMe-direct path based on runtime memory availability. The NVMe-direct path bypasses the filesystem by mapping KV tensors to contiguous logical block address (LBA) regions, enabling low-overhead direct storage access. DUAL-BLADE further incorporates adaptive pipeline parallelism to overlap storage I/O with GPU DMA, improving inference throughput. Our evaluation shows that DUAL-BLADE substantially mitigates I/O bottlenecks, reducing prefill and decode latency by up to 33.1% and 42.4%, respectively, while improving SSD utilization by 2.2x across diverse memory budgets."},"title":{"value":"DUAL-BLADE: Dual-Path NVMe-Direct KV-Cache Offloading for Edge LLM Inference"},"authors":{"value":[{"fullname":"Bodon Jeong","username":""},{"fullname":"Hongsu Byun","username":""},{"fullname":"Youngjae Kim","username":"~Youngjae_Kim3"},{"fullname":"Weikuan Yu","username":""},{"fullname":"Kyungkeun Lee","username":""},{"fullname":"Jihoon Yang","username":"~Jihoon_Yang1"},{"fullname":"Sungyong Park","username":""}]}},"tmdate":1787632769955,"pdate":1798675200000,"externalIds":["dblp:journals/corr/abs-2604-26557"],"tcdate":1785943981252,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Youngjae_Kim3"],"forum":"Dx39bbBuOj","license":"CC BY-SA 4.0","number":123218,"cdate":1775001600000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit","OpenReview.net/Public_Article/-/Authorship_Claim"],"mdate":1787632769955,"domain":"OpenReview.net/Public_Article","id":"Dx39bbBuOj","version":2},{"content":{"TLDR":{"value":"We derive a training free, value aware KV page representaiton that directly optimizes attention output fidelity and consistently outperforms key only compression methods at matched cache budgets."},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["KV cache compression","LLM Inference","Sparse Attention"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Long context inference with large language models becomes increasingly expensive as attention must operate over an ever growing KV cache. Page sparse attention reduces this cost by representing each KV page compactly and retrieving only a subset for each query. Existing retrieval methods are designed to estimate attention scores or page relevance, but their objectives do not directly account for how approximation errors affect the resulting value weighted attention output. We introduce OVAL, an output aware page encoding derived from the joint structure of keys and values while preserving the key information needed for accurate retrieval. OVAL is training free and requires no additional value dependent statistics at inference time. Once constructed, its stored representation has the same size and decode time scoring cost as a key only spectral representation. Across long reasoning, long context understanding, and long generation benchmarks, OVAL consistently improves over the key only spectral baseline and performs competitively with recent KV cache compression and retrieval methods. On long reasoning benchmarks, it achieves strong avg@\\(k\\) performance across model benchmark pairs, while matching or surpassing leading baselines on several long context understanding and generation settings with modest decoding overhead. Code is available at \\url{https://anonymous.4open.science/r/oval-kv-4424/}."},"_bibtex":{"value":"@inproceedings{\nanonymous2026oval,\ntitle={{OVAL}: Output-Aware Local Page Bases for {KV} Cache Retrieval},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=YSOeAm9KT6},\nnote={under review}\n}"},"title":{"value":"OVAL: Output-Aware Local Page Bases for KV Cache Retrieval"},"pdf":{"value":"/pdf/aed0426cebe65445d9cf23230e0e49b971477cd3.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791227060871,"tcdate":1789228889961,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission15274/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission15274/Authors"],"forum":"YSOeAm9KT6","license":"CC BY 4.0","number":15274,"cdate":1789228889961,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission15274/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791227060871,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"YSOeAm9KT6","version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"http://arxiv.org/pdf/2502.15075v1"},"venueid":{"value":"dblp.org/journals/CORR/2025"},"paperhash":{"value":"hariri|more_for_keys_less_for_values_adaptive_kv_cache_quantization"},"authorids":{"value":["~Mohsen_Hariri1","https://dblp.org/search/pid/api?q=author:Lam_Nguyen:","https://dblp.org/search/pid/api?q=author:Sixu_Chen:","https://dblp.org/search/pid/api?q=author:Shaochen_Zhong:","~Qifan_Wang1","https://dblp.org/search/pid/api?q=author:Xia_Ben_Hu:","~Xiaotian_Han1","https://dblp.org/search/pid/api?q=author:Vipin_Chaudhary:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2502.15075"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2502-15075,\n publtype={informal},\n author={Mohsen Hariri and Lam Nguyen and Sixu Chen and Shaochen Zhong and Qifan Wang and Xia Ben Hu and Xiaotian Han and Vipin Chaudhary},\n title={More for Keys, Less for Values: Adaptive KV Cache Quantization},\n year={2025},\n month={February},\n cdate={1738368000000},\n journal={CoRR},\n volume={abs/2502.15075},\n url={https://doi.org/10.48550/arXiv.2502.15075}\n}\n"},"abstract":{"value":"This paper introduces an information-aware quantization framework that adaptively compresses the key-value (KV) cache in large language models (LLMs). Although prior work has underscored the distinct roles of key and value cache during inference, our systematic analysis -- examining singular value distributions, spectral norms, and Frobenius norms -- reveals, for the first time, that key matrices consistently exhibit higher norm values and are more sensitive to quantization than value matrices. Furthermore, our theoretical analysis shows that matrices with higher spectral norms amplify quantization errors more significantly. Motivated by these insights, we propose a mixed-precision quantization strategy, KV-AdaQuant, which allocates more bit-width for keys and fewer for values since key matrices have higher norm values. With the same total KV bit budget, this approach effectively mitigates error propagation across transformer layers while achieving significant memory savings. Our extensive experiments on multiple LLMs (1B--70B) demonstrate that our mixed-precision quantization scheme maintains high model accuracy even under aggressive compression. For instance, using 4-bit for Key and 2-bit for Value achieves an accuracy of 75.2%, whereas reversing the assignment (2-bit for Key and 4-bit for Value) yields only 54.7% accuracy. The code is available at https://tinyurl.com/kv-adaquant"},"title":{"value":"More for Keys, Less for Values: Adaptive KV Cache Quantization"},"authors":{"value":["Mohsen Hariri","Lam Nguyen","Sixu Chen","Shaochen Zhong","Qifan Wang","Xia Ben Hu","Xiaotian Han","Vipin Chaudhary"]}},"tmdate":1758317104288,"pdate":1735689600000,"tcdate":1747579179261,"writers":["~"],"signatures":["~Mohsen_Hariri1"],"forum":"fy9fd83XrT","license":"CC BY-SA 4.0","number":521919,"cdate":1738368000000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1758317104288,"domain":"DBLP.org","id":"fy9fd83XrT","version":2},{"content":{"data_release":{"value":"We authorize the release of our submission and author names to the public in the event of acceptance."},"venue":{"value":"ICML MemFM 2026 Workshop Oral"},"email_sharing":{"value":"We authorize the sharing of all author emails with Program Chairs."},"keywords":{"value":["large language models","KV cache","retrieval-augmented generation","prompt injection","long-context reasoning"]},"abstract":{"value":"Context erasing for KV cache is challenging because a local edit has a global consequence: once a span has been processed, its influence propagates into the cached states of all subsequent tokens. Exact erasing must recompute all tokens after the deleted span. We introduce KVEraser, a learned KV-cache editing method for efficient localized context erasing. Experiments show that KVEraser matches full recomputation's performance on in-domain tasks across 1K--32K context lengths, while its latency increases by only 24\\% compared with a 17.6$\\times$ increase for full recomputation. KVEraser also generalizes to unseen QA tasks with harmful factual distractors, achieving a 3--4$\\times$ speedup over full recomputation."},"_bibtex":{"value":"@inproceedings{\nli2026kveraser,\ntitle={{KVE}raser: Learning to Steer {KV} Cache for Efficient Localized Context Erasing},\nauthor={Mufei Li and Shikun Liu and Dongqi Fu and Haoyu Peter Wang and Yinglong Xia and Hong Li and Hong Yan and Pan Li},\nbooktitle={The Second Workshop on the Impact of Memorization on Trustworthy Foundation Models at ICML},\nyear={2026},\nurl={https://openreview.net/forum?id=dOrmkrvRVM}\n}"},"title":{"value":"KVEraser: Learning to Steer KV Cache for Efficient Localized Context Erasing"},"pdf":{"value":"/pdf/837b08ff729d8857793588263295c043724617ea.pdf"},"venueid":{"value":"ICML.cc/2026/Workshop/MemFM"},"paperhash":{"value":"li|kveraser_learning_to_steer_kv_cache_for_efficient_localized_context_erasing"},"authorids":{"value":["~Mufei_Li2","~Shikun_Liu3","~Dongqi_Fu1","~Haoyu_Peter_Wang1","~Yinglong_Xia1","~Hong_Li15","~Hong_Yan1","~Pan_Li2"]},"authors":{"value":["Mufei Li","Shikun Liu","Dongqi Fu","Haoyu Peter Wang","Yinglong Xia","Hong Li","Hong Yan","Pan Li"]}},"tmdate":1780548615349,"pdate":1780548613635,"tcdate":1778281901898,"writers":["ICML.cc/2026/Workshop/MemFM","ICML.cc/2026/Workshop/MemFM/Submission44/Authors"],"signatures":["ICML.cc/2026/Workshop/MemFM/Submission44/Authors"],"forum":"dOrmkrvRVM","license":"CC BY 4.0","number":44,"cdate":1778281901898,"readers":["everyone"],"invitations":["ICML.cc/2026/Workshop/MemFM/-/Submission","ICML.cc/2026/Workshop/MemFM/-/Submission_Change_Before_Reviewing","ICML.cc/2026/Workshop/MemFM/-/Submission_Release"],"mdate":1780548615349,"odate":1780548613635,"domain":"ICML.cc/2026/Workshop/MemFM","id":"dOrmkrvRVM","version":2},{"content":{"summary":{"value":"The paper proposes Learn2PD, a learnable parallel decoding framework for dLLMs that replaces fixed heuristics with a lightweight filter model that predicts, per token, whether its current prediction already matches the final output. \nThis adaptive filter enables tokens to be “unmasked” dynamically and in parallel, improving the speed–quality tradeoff. They also introduce an EoTP mechanism to detect completion and prevent redundant decoding, and experiments show significant speedups."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"Please see the weaknesses above."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"1. The problem this paper tries to address is an important one: current dLLM decoding is mainly heuristic-based. A hyperparameter needs to be used (like confidence) for unmasking. The neural predictor is indeed a plausible solution. \n\n2. The speedup achieved is impressive. \n\n3. The evaluation is qutie solid, including compatibility with KV cache methods."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Using a KV cache during inference can cause a misalignment with the training process, potentially altering the model's generated outputs from the training ground truth.\n\n2. As shown in Table 2, there is a big acc drop with dual cache (5.83). Is this caused by Learn2PD? \n\n3. The evaluation is mainly on LLaDA. How does the method generalize to other models like Dream?\n\n4. How much is the inference overhead is the additional predictor? \n\n5. How does the performance change with respect to the size of the predictor?"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762999988304,"tcdate":1760477796409,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission18792/Reviewer_vfNL"],"signatures":["ICLR.cc/2026/Conference/Submission18792/Reviewer_vfNL"],"forum":"bFJ8Sdr224","number":1,"license":"CC BY 4.0","cdate":1760477796409,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission18792/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762999988304,"domain":"ICLR.cc/2026/Conference","replyto":"bFJ8Sdr224","id":"VBnoumvW7V","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Diffusion LLM"]},"supplementary_material":{"value":"/attachment/78556696d259c24c49eaaa3e9ca83840a6fbbc7e.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Autoregressive decoding in large language models (LLMs) requires $\\mathcal{O}(n)$ sequential steps for $n$ tokens, fundamentally limiting inference throughput. Recent diffusion-based LLMs (dLLMs) enable parallel token generation through iterative denoising. However, current parallel decoding strategies rely on fixed, input-agnostic heuristics (e.g., confidence thresholds), which fail to adapt to input-specific characteristics, resulting in suboptimal speed-quality trade-offs across diverse NLP tasks. In this work, we explore a more flexible and dynamic approach to parallel decoding. We propose **Learning to Parallel Decode (Learn2PD)**, a framework that trains a lightweight and adaptive filter model to predict, for each token position, whether the current prediction matches the final output. This learned filter approximates an oracle parallel decoding strategy that unmasks tokens only when correctly predicted. Importantly, the filter model is learned in a post-training manner, requiring only a small amount of computation to optimize it (minute-level GPU time). Additionally, we introduce **End-of-Text Prediction (EoTP)** to detect decoding completion at the end of sequence, avoiding redundant decoding of padding tokens. Experiments on the LLaDA benchmark demonstrate that our method achieves up to **22.58×** speedup without any performance drop, and up to **57.51×** when combined with KV-Cache."},"_bibtex":{"value":"@inproceedings{\nbao2026learning,\ntitle={Learning to Parallel: Accelerating Diffusion Large Language Models via Learnable Parallel Decoding},\nauthor={Wenrui Bao and Zhiben Chen and Dan Xu and Yuzhang Shang},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=bFJ8Sdr224}\n}"},"title":{"value":"Learning to Parallel: Accelerating Diffusion Large Language Models via Learnable Parallel Decoding"},"pdf":{"value":"/pdf/56e6c080c699a8d88e370d5a31acc90e6dc79f02.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"bao|learning_to_parallel_accelerating_diffusion_large_language_models_via_learnable_parallel_decoding"},"authorids":{"value":["~Wenrui_Bao1","~Zhiben_Chen1","~Dan_Xu4","~Yuzhang_Shang1"]},"authors":{"value":["Wenrui Bao","Zhiben Chen","Dan Xu","Yuzhang Shang"]}},"version":2},{"content":{"TLDR":{"value":"PANORAMA reuses approximate cross-model KV caches as context for selective recomputation, enabling near-native quality while making model handoffs substantially faster."},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["large language model serving","KV cache transfer","prefill reuse","cross-model serving","selective recomputation"]},"primary_area":{"value":"infrastructure, software libraries, hardware, systems, etc."},"abstract":{"value":"Real-world LLM workloads, such as multi-agent systems and model cascades that\nescalate requests to larger models, frequently switch a request between models\nwhile retaining the same context. Because KV caches are tied to model parame-\nters and cannot be shared across models, each switch forces the target model to\nre-prefill the entire context that the source model has already processed, incur-\nring substantial computation cost and increasingly delaying time to first token as\ncontext length grows. Existing cross-model KV translation methods avoid this\nre-prefill by mapping the source KV cache into an approximate target KV cache,\nbut translation alone can substantially degrade accuracy. Selective KV recomputa-\ntion has been shown to repair context-inconsistent KV cache entries produced by\nthe same model, but whether selective recomputation remains effective for a KV\ncache translated from a different model is unknown. Our key observation is that\neven when a translated KV cache is insufficient for direct decoding, the translated\ncache can still provide useful context for target-model recomputation. Based on\nthis observation, we propose PANORAMA, a cross-model KV sharing method that\nrecovers high-quality target inference from approximate cross-model KV with-\nout full target re-prefill. PANORAMA contextually recomputes a small subset of\nimportant target positions over the full KV cache produced by the cross-model\nmapper, allowing target-model computation to exploit the translated context. To\nprovide the full translated context efficiently, PANORAMA factorizes the mapper\ninto a shared source projection and layer-specific target reconstruction, transfer-\nring only a compact latent per prompt position. PANORAMA further refines the\nmapper under the repaired mixed cache used at inference, removing the training–\ninference discrepancy between translation-only mapper fitting and mixed-cache\ndecoding. Across model pairs and tasks, PANORAMA achieves the highest ac-\ncuracy among prior cross-model handoff methods on nearly all benchmarks at\ncomparable handoff cost. Moreover, in the Qwen3 8B-to-32B switching scenario,\nour vLLM implementation of PANORAMA reduces handoff latency by 5.84×over\nnative re-prefill at 16K tokens with minimal accuracy loss."},"_bibtex":{"value":"@inproceedings{\nanonymous2026panorama,\ntitle={{PANORAMA}: {CROSS}-{MODEL} {KV} {SHARING} {VIA} {CONTEXTUAL} {RECOMPUTATION}},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=4a3EoxMBfA},\nnote={under review}\n}"},"title":{"value":"PANORAMA: CROSS-MODEL KV SHARING VIA CONTEXTUAL RECOMPUTATION"},"pdf":{"value":"/pdf/bd8d15b2e731959b8ae293fefeaf924fe0b572d2.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791231987770,"tcdate":1789696159884,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission35657/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission35657/Authors"],"forum":"4a3EoxMBfA","license":"CC BY 4.0","number":35657,"cdate":1789696159884,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission35657/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791231987770,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"4a3EoxMBfA","version":2}],"count":10000}