{"notes":[{"content":{"summary":{"value":"Large language models (LLMs) excel at a wide range of tasks, but choosing the right model often involves balancing performance and cost. This paper proposes a routing approach, RouteLLM, to dynamically select between a stronger and weaker LLM during inference. Experiments on three real-world benchmarks demonstrate the effectiveness of RouteLLM."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"Q1. In Sec 3.1, authors introduced the cost threshold \\lambda. It is unclear that how to choose the right cost threshold when user have specific cost budgets.\n\nQ2. Some details in overhead analysis are unclear. Both SW ranking and matrix factorization rely on embeddings generated by text-embedding-3-small. Are the costs incurred by embedding extraction included in the overhead analysis? Also, given that the model size ratio between BERT-base (110M) and causal LLM (8B) is 110M / 8B ~= 1%, it is surprising to see the cost overhead of BERT-base is ~60% of the causal LLM, and the achieved throughput is only 60% higher, according to Table 7. More details on how the overheads are estimated could be very helpful."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"S1. The proposed approach, RouteLLM, is able to achieve over 2x cost savings on popular benchmarks with\nminimal impact on response quality.\n\nS2. Authors demonstrate that RouteLLM enables routers to generalize to unseen data while maintaining strong performance across multiple LLMs."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"W1. Unclear novelty and limited technical contribution. Training a router to harness the respective strengths of different LLMs has been widely studied [1,2,3,4]. Specifically, how to generalize LLM routing to OOD data has been studied in [5], how to use LLMs to generate more training data to help improve routing performance has been explored in [6], which authors did not compare to. Moreover, some technology (SW ranking) proposed in this paper shares unignorable similarity to prior work [7]. \n\nW2. Weak baselines. Provided the rich literature on this topic as aforementioned, considering a random router as the only baseline is insufficient in this work. The effectiveness of RouteLLM could be further demonstrated if authors could compare it to more advanced baselines (e.g., a subset from [1-6]).\n\nW3. In Sec 5.5, authors provided the overhead analysis. Notably, SW ranking is both expensive ($37 / 1M requests) and slow (2.9 requests / second), which makes it hard to use in practice.\n\n\nReferences:  \n[1] Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Models, https://arxiv.org/pdf/2311.08692  \n[2] Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing, https://arxiv.org/abs/2404.14618  \n[3] Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modeling, https://arxiv.org/pdf/2308.06077  \n[4] ROUTERBENCH: A Benchmark for Multi-LLM Routing System, https://arxiv.org/pdf/2403.12031  \n[5] Large Language Model Routing with Benchmark Datasets, https://arxiv.org/pdf/2309.15789  \n[6] Routoo: Learning to Route to Large Language Models Effectively, https://arxiv.org/abs/2401.13979  \n[7] Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference, https://arxiv.org/abs/2403.04132"}},"nonreaders":[],"tmdate":1733195764608,"tcdate":1730679883997,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission13083/Reviewer_GuJg"],"signatures":["ICLR.cc/2025/Conference/Submission13083/Reviewer_GuJg"],"forum":"8sSqNntaMr","number":2,"license":"CC BY 4.0","cdate":1730679883997,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission13083/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1733195764608,"domain":"ICLR.cc/2025/Conference","replyto":"8sSqNntaMr","id":"R4qducnFBL","forumContent":{"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Large language models","query routing"]},"supplementary_material":{"value":"/attachment/21a6352b733fa8b803857c0be57176fe75cfc5f6.zip"},"primary_area":{"value":"generative models"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Large language models (LLMs) excel at a wide range of tasks, but choosing the right model often involves balancing performance and cost. Powerful models offer better results but are expensive, while smaller models are more cost-effective but less capable. To address this trade-off, we introduce a training framework for learning efficient router models that dynamically select between a stronger and weaker LLM during inference. Our framework leverages human preference data and employs data augmentation techniques to enhance performance. Evaluations on public benchmarks show that our approach can reduce costs by over 2 times without sacrificing response quality. Moreover, our routers exhibit strong generalization capabilities, maintaining performance even when routing between LLMs not included in training. This highlights the potential of our framework to deliver cost-effective, high-performance LLM solutions."},"_bibtex":{"value":"@inproceedings{\nong2025routellm,\ntitle={Route{LLM}: Learning to Route {LLM}s from Preference Data},\nauthor={Isaac Ong and Amjad Almahairi and Vincent Wu and Wei-Lin Chiang and Tianhao Wu and Joseph E. Gonzalez and M Waleed Kadous and Ion Stoica},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=8sSqNntaMr}\n}"},"title":{"value":"RouteLLM: Learning to Route LLMs from Preference Data"},"pdf":{"value":"/pdf/900c2f5b71fd9d4dc34d8f299cce9a11336d30b4.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"ong|routellm_learning_to_route_llms_from_preference_data"},"authorids":{"value":["~Isaac_Ong1","~Amjad_Almahairi1","~Vincent_Wu1","~Wei-Lin_Chiang1","~Tianhao_Wu1","~Joseph_E._Gonzalez1","~M_Waleed_Kadous1","~Ion_Stoica1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Isaac Ong","Amjad Almahairi","Vincent Wu","Wei-Lin Chiang","Tianhao Wu","Joseph E. Gonzalez","M Waleed Kadous","Ion Stoica"]}},"version":2},{"content":{"comment":{"value":"## **W4: The evidence supporting the superiority of \"intrinsic assessment\" is marginal. In Table 5, the purpose-trained 7B DiSRouter (0.80 F1) barely outperforms a general-purpose 8B external classifier (0.77 F1). This negligible gap fails to provide a strong justification for the added complexity of the distributed training approach over a simpler, well-trained external router.**\n\nWe respectfully disagree with the assessment that the performance gap is marginal. While the raw difference in F1 score on a single agent appears small, our new experiments demonstrate that this \"marginal\" advantage translates into a **substantial gain in system-level utility** due to the cumulative nature of the routing cascade.\n\n### **1. Theoretical Clarification: Error Propagation in Cascades**\n\nComparing single-node F1 scores in isolation underestimates the impact of precision in a distributed system. In a cascading architecture, errors are **cumulative**: a slight degradation in the rejection precision of early-stage models can lead to significant downstream inefficiency or accuracy loss. \n\n- **False Rejections:** A \"marginal\" increase in false rejections at early stages (small models) forces a disproportionately large number of simple queries to expensive downstream models, causing increased total cost.\n- **False Acceptances:** A slight drop in rejection precision leads to immediate performance loss (incorrect answers).\n\nTherefore, what appears to be a 0.04 F1 gap in a single node can result in **a significant degradation of the global Utility metric**.\n\n### **2. New Comparative Experiments**\n\nTo rigorously validate this and isolate the benefit of \"Intrinsic Assessment\" from \"Model Size\" or \"Architecture,\" we implemented two new strong baselines for the *Balance* scenario:\n\n- **`RouteLLM_large` (Centralized 8B):** A centralized router using the standard RouteLLM architecture (N-classification) but upgraded to use a powerful **Llama3-8B-Instruct** backbone. This controls for model size against our 7B agents.\n- **`DiSRouter_external` (Cascade 8B):** A cascading system structurally identical to DiSRouter, but instead of using self-awareness, each agent is paired with a dedicated **external binary classifier** (also Llama3-8B-Instruct) to make the accept/reject decision. This isolates the \"Intrinsic vs. External\" variable.\n\nThe results are summarized in **Table R6** below.\n\n*Table R6: Comparative ablation study isolating the impact of Routing Architecture (Centralized vs. Cascade) and Assessment Mechanism (External vs. Intrinsic) on System Utility.*\n\n| Index | Method | Architecture | Backbone Model for Router | Accuracy | Cost | Utility |\n| --- | --- | --- | --- | --- | --- | --- |\n| 1 | RouteLLM | centralized | RoBERTa-base | 0.44 | 0.16 | 0.36 |\n| 2 | RouteLLM_large | centralized | Llama3-8B-Instruct | 0.57 | 0.37 | 0.39 |\n| 3 | DiSRouter_external | cascading | Llama3-8B-Instruct | 0.57 | 0.29 | 0.42 |\n| 4 | DiSRouter | cascading | LLM itself | 0.77 | 0.32 | 0.61 |\n\n### **3. Analysis of Results**\n\nThe comparison yields three critical insights:\n\n1. **Impact of Router Size:** Comparing `RouteLLM_large` with the original `RouteLLM` (**Result 1 vs. 2**), we observe that while increasing the router size improves centralized performance, the gain is relatively limited.\n2. **Cascade vs. Centralized (External):** `DiSRouter_external` outperforms `RouteLLM_large` (**Result 2 vs. 3**). This confirms that a cascading architecture—where classifiers focus on the simpler task of modeling a single agent's binary boundary—is easier to optimize than a centralized router attempting to rank all agents simultaneously.\n3. **Intrinsic vs. External (The Critical Gap):** Most importantly, `DiSRouter (Intrinsic)` significantly outperforms `DiSRouter_external` (**Result 3 vs. 4**, Utility **0.61 vs. 0.42**).\n\n**Conclusion:** This substantial gap in Utility (**+45% relative improvement**) decisively validates our approach. It proves that an external judge—even a powerful 8B model—cannot perfectly align with the target model's internal knowledge boundary. **Intrinsic self-awareness** eliminates this alignment mismatch, providing a superior routing signal that justifies the training design."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764156832515,"tcdate":1764156832515,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7635/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission7635/Authors"],"forum":"KDcwXKr0NU","number":11,"license":"CC BY 4.0","cdate":1764156832515,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7635/-/Official_Comment"],"mdate":1764156832515,"domain":"ICLR.cc/2026/Conference","replyto":"A5YaHYKdc6","id":"gsKHwa7M2o","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"TLDR":{"value":"We introduce DiSRouter, a distributed system where LLMs leverage \"self-awareness\" to route queries among themselves, outperforming conventional centralized routers."},"keywords":{"value":["query routing","model selection","distributed system","self-awareness of LLM"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"The proliferation of Large Language Models (LLMs) has created a diverse ecosystem of models with highly varying performance and costs, necessitating effective query routing to balance performance and expense. Current routing systems often rely on a centralized external router trained on a fixed set of LLMs, making them inflexible and prone to poor performance since the small router can not fully understand the knowledge boundaries of different LLMs. We introduce DiSRouter (Distributed Self-Router), a novel paradigm that shifts from centralized control to distributed routing. In DiSRouter, a query traverses a network of LLM agents, each independently deciding whether to answer or route to other agents based on its own self-awareness—its ability to judge its competence. This distributed design offers superior flexibility, scalability, and generalizability. To enable this, we propose a two-stage Self-Awareness Training pipeline that enhances each LLM's self-awareness. Extensive experiments demonstrate that DiSRouter significantly outperforms existing routing methods in utility across various scenarios, effectively distinguishes between easy and hard queries, and shows strong generalization to out-of-domain tasks. Our work validates that leveraging an LLM's intrinsic self-awareness is more effective than external assessment, paving the way for more modular and efficient multi-agent systems."},"_bibtex":{"value":"@inproceedings{\nzheng2026disrouter,\ntitle={Di{SR}outer: Distributed Self-Routing for {LLM} Selections},\nauthor={Hang Zheng and Hongshen Xu and Yongkai.lin and Shuai Fan and Lu Chen and Kai Yu},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=KDcwXKr0NU}\n}"},"title":{"value":"DiSRouter: Distributed Self-Routing for LLM Selections"},"pdf":{"value":"/pdf/0190f98d94890b700d03febdee678e22a2a5e380.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"zheng|disrouter_distributed_selfrouting_for_llm_selections"},"authorids":{"value":["~Hang_Zheng3","~Hongshen_Xu1","~Yongkai.lin1","~Shuai_Fan1","~Lu_Chen3","~Kai_Yu3"]},"authors":{"value":["Hang Zheng","Hongshen Xu","Yongkai.lin","Shuai Fan","Lu Chen","Kai Yu"]}},"version":2},{"content":{"venue":{"value":"CoRR 2024"},"pdf":{"value":"http://arxiv.org/pdf/2406.18665v3"},"venueid":{"value":"dblp.org/journals/CORR/2024"},"paperhash":{"value":"ong|routellm_learning_to_route_llms_with_preference_data"},"authorids":{"value":["~Isaac_Ong1","https://dblp.org/search/pid/api?q=author:Amjad_Almahairi:","https://dblp.org/search/pid/api?q=author:Vincent_Wu:","https://dblp.org/search/pid/api?q=author:Wei-Lin_Chiang:","https://dblp.org/search/pid/api?q=author:Tianhao_Wu_0002:","~Joseph_E._Gonzalez1","https://dblp.org/search/pid/api?q=author:M._Waleed_Kadous:","https://dblp.org/search/pid/api?q=author:Ion_Stoica:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2406.18665"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2406-18665,\n  publtype={informal},\n  author={Isaac Ong and Amjad Almahairi and Vincent Wu and Wei-Lin Chiang and Tianhao Wu and Joseph E. Gonzalez and M. Waleed Kadous and Ion Stoica},\n  title={RouteLLM: Learning to Route LLMs with Preference Data},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2406.18665},\n  url={https://doi.org/10.48550/arXiv.2406.18665}\n}\n"},"abstract":{"value":"Large language models (LLMs) exhibit impressive capabilities across a wide range of tasks, yet the choice of which model to use often involves a trade-off between performance and cost. More powerful models, though effective, come with higher expenses, while less capable models are more cost-effective. To address this dilemma, we propose several efficient router models that dynamically select between a stronger and a weaker LLM during inference, aiming to optimize the balance between cost and response quality. We develop a training framework for these routers leveraging human preference data and data augmentation techniques to enhance performance. Our evaluation on widely-recognized benchmarks shows that our approach significantly reduces costs-by over 2 times in certain cases-without compromising the quality of responses. Interestingly, our router models also demonstrate significant transfer learning capabilities, maintaining their performance even when the strong and weak models are changed at test time. This highlights the potential of these routers to provide a cost-effective yet high-performance solution for deploying LLMs."},"title":{"value":"RouteLLM: Learning to Route LLMs with Preference Data"},"authors":{"value":["Isaac Ong","Amjad Almahairi","Vincent Wu","Wei-Lin Chiang","Tianhao Wu","Joseph E. Gonzalez","M. Waleed Kadous","Ion Stoica"]}},"tmdate":1731476158374,"pdate":1704067200000,"tcdate":1728270974512,"writers":["~"],"signatures":["~Isaac_Ong1"],"forum":"V1QSnecgHO","license":"CC BY-SA 4.0","number":140645,"cdate":1704067200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit","DBLP.org/-/Author_Coreference"],"mdate":1731476158374,"domain":"DBLP.org","id":"V1QSnecgHO","version":2},{"content":{"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Large language models","query routing"]},"supplementary_material":{"value":"/attachment/21a6352b733fa8b803857c0be57176fe75cfc5f6.zip"},"primary_area":{"value":"generative models"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Large language models (LLMs) excel at a wide range of tasks, but choosing the right model often involves balancing performance and cost. Powerful models offer better results but are expensive, while smaller models are more cost-effective but less capable. To address this trade-off, we introduce a training framework for learning efficient router models that dynamically select between a stronger and weaker LLM during inference. Our framework leverages human preference data and employs data augmentation techniques to enhance performance. Evaluations on public benchmarks show that our approach can reduce costs by over 2 times without sacrificing response quality. Moreover, our routers exhibit strong generalization capabilities, maintaining performance even when routing between LLMs not included in training. This highlights the potential of our framework to deliver cost-effective, high-performance LLM solutions."},"_bibtex":{"value":"@inproceedings{\nong2025routellm,\ntitle={Route{LLM}: Learning to Route {LLM}s from Preference Data},\nauthor={Isaac Ong and Amjad Almahairi and Vincent Wu and Wei-Lin Chiang and Tianhao Wu and Joseph E. Gonzalez and M Waleed Kadous and Ion Stoica},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=8sSqNntaMr}\n}"},"title":{"value":"RouteLLM: Learning to Route LLMs from Preference Data"},"pdf":{"value":"/pdf/900c2f5b71fd9d4dc34d8f299cce9a11336d30b4.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"ong|routellm_learning_to_route_llms_from_preference_data"},"authorids":{"value":["~Isaac_Ong1","~Amjad_Almahairi1","~Vincent_Wu1","~Wei-Lin_Chiang1","~Tianhao_Wu1","~Joseph_E._Gonzalez1","~M_Waleed_Kadous1","~Ion_Stoica1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Isaac Ong","Amjad Almahairi","Vincent Wu","Wei-Lin Chiang","Tianhao Wu","Joseph E. Gonzalez","M Waleed Kadous","Ion Stoica"]}},"tmdate":1740299879943,"pdate":1737562955798,"tcdate":1727492428126,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission13083/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission13083/Authors"],"forum":"8sSqNntaMr","license":"CC BY 4.0","number":13083,"cdate":1727492428126,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/-/Submission","ICLR.cc/2025/Conference/-/Post_Submission","ICLR.cc/2025/Conference/Submission13083/-/Full_Submission","ICLR.cc/2025/Conference/-/Edit","ICLR.cc/2025/Conference/Submission13083/-/Camera_Ready_Revision"],"mdate":1740299879943,"odate":1728008565725,"domain":"ICLR.cc/2025/Conference","id":"8sSqNntaMr","version":2},{"content":{"venue":{"value":"Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2025"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"hallgarten|routellm_a_large_language_model_with_native_route_context_understanding_to_enable_contextaware_reasoning"},"html":{"value":"https://doi.org/10.1145/3749552"},"_bibtex":{"value":"@article{DBLP:journals/imwut/HallgartenHKBG25,\n  author={Philipp Hallgarten and Verena Jasmin Hallitschke and Enkelejda Kasneci and Michael Beigl and Tobias Grosse-Puppendahl},\n  title={RouteLLM: A Large Language Model with Native Route Context Understanding to Enable Context-Aware Reasoning},\n  year={2025},\n  month={September},\n  cdate={1756684800000},\n  journal={Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.},\n  volume={9},\n  number={3},\n  pages={83:1-83:34},\n  url={https://doi.org/10.1145/3749552}\n}\n"},"title":{"value":"RouteLLM: A Large Language Model with Native Route Context Understanding to Enable Context-Aware Reasoning"},"authors":{"value":[{"fullname":"Philipp Hallgarten","username":""},{"fullname":"Verena Jasmin Hallitschke","username":"~Verena_Jasmin_Hallitschke1"},{"fullname":"Enkelejda Kasneci","username":""},{"fullname":"Michael Beigl","username":""},{"fullname":"Tobias Grosse-Puppendahl","username":""}]}},"tmdate":1787301746533,"pdate":1767139200000,"externalIds":["dblp:journals/imwut/HallgartenHKBG25"],"tcdate":1787301746413,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Verena_Jasmin_Hallitschke1"],"forum":"cLMu0kiv7C","license":"CC BY-SA 4.0","number":134866,"cdate":1756684800000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1787301746533,"domain":"OpenReview.net/Public_Article","id":"cLMu0kiv7C","version":2},{"content":{"comment":{"value":"Thank you for your very thorough additional experiments. I think you were able to address most of my concerns. I just have two points to make:\nRegarding the preference based comparison: My argument regarding RouteLLM was less that this specific model would be used, but rather that the mathematical model for preference that they assume seems to be more fitting than the one proposed here. RouteLLM uses the framework of preference modelling to produce the distribution over \"winning\" LLMs in (in their case) a price vs performance setting. This makes sense due to the decision theoretic underpinnings of preference optimization (indeed maximizing a utility and encoding preferences is equivalent via the von Neumann–Morgenstern utility theorem). RouteLLM's modelling essentially introduces a utility function (in their case the log-likelihood), which implicitly encodes a preference for one vs the other model. The reason they do not capture user preferences is because they do not condition their distribution on a \"user embedding\" (a very concrete example of this happening is in RLHF reward models which optimizes e.g. a bradley-terry model over the preferences. Route LLM doesn't need this since it has ground-truth binary preferences, but from a decision theory point of view this is identical, you just assume absolute rather than relative feedback).\n\nI'm unsure whether a link-prediction task is really ideal for the problem as described: The method used by e.g. RouteLLM also introduces a preference graph, but in an implicit fashion. However, I do want to stress that I don't see this as a blocker for this paper's acceptance: The fact that the link-prediction problem might not be ideal does not mean this paper is unpublishable!\n\nThe second remark is regarding the \"1000 User setting\": How did you define the users in that setting? I see that you have a table in the appendix for 15 users, but how are you generating the personas for the 1000 user setting?\n\nOther than that, your updates seem to have covered my concerns."}},"parentInvitations":"TMLR/-/Official_Comment","tmdate":1759338707937,"tcdate":1759338707937,"writers":["TMLR","TMLR/Paper5500/Reviewer_h7mj"],"signatures":["TMLR/Paper5500/Reviewer_h7mj"],"forum":"W80eE3ArAl","number":10,"license":"CC BY 4.0","cdate":1759338707937,"readers":["everyone"],"invitations":["TMLR/Paper5500/-/Official_Comment"],"mdate":1759338707937,"domain":"TMLR","replyto":"KX1BcZex34","id":"WUGAaKvDp9","forumContent":{"submission_length":{"value":"Regular submission (no more than 12 pages of main content)"},"venue":{"value":"Accepted by TMLR"},"code":{"value":"https://github.com/ulab-uiuc/PersonalizedRouter"},"supplementary_material":{"value":"/attachment/e4bbeecf2accf23bec60f721de3508b2cf488096.zip"},"abstract":{"value":"The growing number of Large Language Models (LLMs) with diverse capabilities and response styles provides users with a wider range of choices, which presents challenges in selecting appropriate LLMs, as user preferences vary in terms of performance, cost, and response style. Current LLM selection methods typically optimize for a single fixed objective, such as performance, cost, or a trade-off between them, and fail to learn individual user preferences from interaction data. To address these limitations, we propose PersonalizedRouter, a graph-based framework that models diverse user profiles and performs personalized LLM selection by leveraging interaction data that includes task context, queries, candidate LLMs, and user decisions. To capture contextual information between user queries and optimal LLMs, PersonalizedRouter converts the interaction data into a heterogeneous graph, where the relationships between different types of nodes are represented by edges. To evaluate adaptability across users, we design two strategies: the multi-cost-efficiency simulation strategy and the LLM-as-a-Judge strategy. In addition, we construct PersonaRoute-Bench, a large-scale benchmark with 1,000 simulated users and 10 LLMs. Experimental results show that PersonalizedRouter significantly outperforms existing LLM selection methods and surpasses the strongest methods by a large margin of 15.38% and 9.83% under two simulation strategies. On the PersonaRoute-Bench with 1,000 users, it further surpasses the best methods by 16.19% and 59.69% while maintaining higher efficiency. Moreover, PersonalizedRouter demonstrates strong few-shot generalization, achieving 64.81% and 85.80% of the fully trained model’s performance when adapting to new users and new LLMs."},"_bibtex":{"value":"@article{\ndai2025personalizedrouter,\ntitle={PersonalizedRouter: Personalized {LLM} Routing via Graph-based User Preference Modeling},\nauthor={Zhongjie Dai and Tao Feng and Jiaxuan You},\njournal={Transactions on Machine Learning Research},\nissn={2835-8856},\nyear={2025},\nurl={https://openreview.net/forum?id=W80eE3ArAl},\nnote={}\n}"},"title":{"value":"PersonalizedRouter: Personalized LLM Routing via Graph-based User Preference Modeling"},"pdf":{"value":"/pdf/1c7d7c4a7a37c3cd71672b102988c3788f67c8e7.pdf"},"venueid":{"value":"TMLR"},"paperhash":{"value":"dai|personalizedrouter_personalized_llm_routing_via_graphbased_user_preference_modeling"},"authorids":{"value":["~Zhongjie_Dai1","~Tao_Feng5","~Jiaxuan_You2"]},"assigned_action_editor":{"value":"~Yen-Chang_Hsu1"},"authors":{"value":["Zhongjie Dai","Tao Feng","Jiaxuan You"]}},"version":2},{"content":{"research_area_keywords":{"value":"Multi-Agent, Route Recommendation, Information Retrieval and Text Mining,"},"languages_studied":{"value":"English"},"venue":{"value":"ACL ARR 2025 October Submission"},"_bibtex":{"value":"@inproceedings{\nanonymous2026constraintaware,\ntitle={Constraint-Aware Route Recommendation from Natural Language via Hierarchical {LLM} Agents},\nauthor={Anonymous},\nbooktitle={Submitted to ACL Rolling Review - October 2025},\nyear={2026},\nurl={https://openreview.net/forum?id=vYIDQTXT2U},\nnote={under review}\n}"},"title":{"value":"Constraint-Aware Route Recommendation from Natural Language via Hierarchical LLM Agents"},"contribution_types":{"value":["NLP engineering experiment"]},"abstract":{"value":"Route recommendation aims to provide users with optimal travel plans that satisfy diverse and complex requirements.\nClassical routing algorithms (e.g., shortest-path and constraint-aware search) are efficient but assume structured inputs and fixed objectives, limiting adaptability to natural-language queries. \nRecent LLM-based approaches enhance flexibility but struggle with spatial reasoning and the joint modeling of route-level and POI-level preferences.\nTo address these limitations, we propose RouteLLM, a hierarchical multi-agent framework that grounds natural-language intents into constraint-aware routes.\nIt first parses user queries into structured intents including POIs, paths, and constraints. A manager agent then coordinates specialized sub-agents: a constraint agent that resolves and formally check constraints, a POI agent that retrieves and ranks candidate POIs, and a path refinement agent that refines routes via a routing engine with preference-conditioned costs. \nA final verifier agent ensures constraint satisfaction and produces the final route with an interpretable rationale.\nThis design bridges linguistic flexibility and spatial structure, enabling reasoning over route feasibility and user preferences.\nExperiments show that our method reliably grounds textual preferences into constraint-aware routes, improving route quality and preference satisfaction over classical methods.\nOur code and data are publicly available: https://anonymous.4open.science/r/RouteLLM_ACL-DE52."},"paper_type":{"value":"Long"},"pdf":{"value":"/pdf/3cd4da2becaf74d59a0fd57b702c9ccbc2ca6cb1.pdf"},"research_area":{"value":"NLP Applications"},"venueid":{"value":"aclweb.org/ACL/ARR/2025/October/Submission"}},"tmdate":1782961402061,"tcdate":1759568982620,"writers":["aclweb.org/ACL/ARR/2025/October","aclweb.org/ACL/ARR/2025/October/Submission677/Authors"],"signatures":["aclweb.org/ACL/ARR/2025/October/Submission677/Authors"],"forum":"vYIDQTXT2U","license":"CC BY 4.0","number":677,"cdate":1759568982620,"readers":["everyone"],"invitations":["aclweb.org/ACL/ARR/2025/October/-/Submission","aclweb.org/ACL/ARR/2025/October/-/Edit","aclweb.org/ACL/ARR/2025/October/-/Post_Submission","aclweb.org/ACL/ARR/2025/October/-/Preprint_Post_Submission"],"mdate":1782961402061,"odate":1782961377851,"domain":"aclweb.org/ACL/ARR/2025/October","id":"vYIDQTXT2U","version":2},{"content":{"summary":{"value":"This paper introduces RouteLLM, a framework for training router models that direct queries between stronger and weaker LLMs to optimise cost-performance tradeoffs. It employs preference data from Chatbot Arena. Empirical results show that data augmentation is important to letting the routers outperform a random baseline on MMLU and GSM8K. It also demonstrates that the routers generalise across domains, and do not need to be retrained."},"soundness":{"value":4},"confidence":{"value":4},"questions":{"value":"1. How well do the routers do at separating models which are much closer in ability e.g. two different 7B models?\n2. How well do the routers do if the two models are not strictly \"stronger\" or \"weaker\", but rather have been finetuned to do different tasks?"},"rating":{"value":8},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":3},"strengths":{"value":"1. The paper presents a novel framework of directly using human preference data, as opposed to reward models, to route between a pair of LLMs.\n2. The empirical evaluation is comprehensive, touching multiple architectures, LLMs and common benchmarks.\n3. Both metrics of \"average performance gap recovered\" and \"call-performance threshold\" are clear reflections of real world considerations: the general ability for the router to close the performance gap between the better and worse model, as well as the cost to doing so for a minimum quality bar.\n4. The increasing variation in LLM quality and cost makes it more important to be able to efficiently tradeoff between cost and performance. On MT Bench, RouteLLM is able to achieve comparable performance to GPT-4 with a cost saving of ~3.7x."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The paper focuses on a binary routing of \"strong\" versus \"weak\" model, but doesn't consider other binary differences between models.\n2. The paper does not discuss in depth why certain architectures perform better or worse, especially with respect to the improvement in the causal LLM's performance when faced with data augmentation."}},"nonreaders":[],"tmdate":1731429536210,"tcdate":1730719727937,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission13083/Reviewer_V25N"],"signatures":["ICLR.cc/2025/Conference/Submission13083/Reviewer_V25N"],"forum":"8sSqNntaMr","number":3,"license":"CC BY 4.0","cdate":1730719727937,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission13083/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731429536210,"domain":"ICLR.cc/2025/Conference","replyto":"8sSqNntaMr","id":"Cndn7h17jH","forumContent":{"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Large language models","query routing"]},"supplementary_material":{"value":"/attachment/21a6352b733fa8b803857c0be57176fe75cfc5f6.zip"},"primary_area":{"value":"generative models"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Large language models (LLMs) excel at a wide range of tasks, but choosing the right model often involves balancing performance and cost. Powerful models offer better results but are expensive, while smaller models are more cost-effective but less capable. To address this trade-off, we introduce a training framework for learning efficient router models that dynamically select between a stronger and weaker LLM during inference. Our framework leverages human preference data and employs data augmentation techniques to enhance performance. Evaluations on public benchmarks show that our approach can reduce costs by over 2 times without sacrificing response quality. Moreover, our routers exhibit strong generalization capabilities, maintaining performance even when routing between LLMs not included in training. This highlights the potential of our framework to deliver cost-effective, high-performance LLM solutions."},"_bibtex":{"value":"@inproceedings{\nong2025routellm,\ntitle={Route{LLM}: Learning to Route {LLM}s from Preference Data},\nauthor={Isaac Ong and Amjad Almahairi and Vincent Wu and Wei-Lin Chiang and Tianhao Wu and Joseph E. Gonzalez and M Waleed Kadous and Ion Stoica},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=8sSqNntaMr}\n}"},"title":{"value":"RouteLLM: Learning to Route LLMs from Preference Data"},"pdf":{"value":"/pdf/900c2f5b71fd9d4dc34d8f299cce9a11336d30b4.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"ong|routellm_learning_to_route_llms_from_preference_data"},"authorids":{"value":["~Isaac_Ong1","~Amjad_Almahairi1","~Vincent_Wu1","~Wei-Lin_Chiang1","~Tianhao_Wu1","~Joseph_E._Gonzalez1","~M_Waleed_Kadous1","~Ion_Stoica1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Isaac Ong","Amjad Almahairi","Vincent Wu","Wei-Lin Chiang","Tianhao Wu","Joseph E. Gonzalez","M Waleed Kadous","Ion Stoica"]}},"version":2},{"content":{"comment":{"value":"## RouterBench (Hu et al., 2024)\n\n| Method | Baseline(s) compared | Performance / Cost | Evaluation conditions |\n|---|---|---|---|\n| Cascade Routing (Dekoninck et al., 2025) | Linear interpolation, Routing, Cascade (baseline) | 87.24% AUC vs. 84.48% for the strongest baseline (11-model, low-noise) | 3/5/11-model subsets tested; cost basis not stated |\n| GreenServ (Ziller et al., 2026) | Contextual ε-greedy, Thompson sampling | Best accuracy (71.7% avg.) but not best AIQ (0.607 vs. 0.637 for ε-greedy) | ~36k queries, 9 tasks; pool composition not stated |\n| IRT-Router (Song et al., 2025) | Small/Large LLM only, HybridLLM (Ding et al., 2024), RouteLLM (Ong et al., 2025), RouterBench's Predictive Router | 80.69% accuracy at USD 0.55/query vs. 80.01% at USD 1.15 for RouterBench's Predictive Router | Small=Ministral-8B, Large=GPT-4o; cost = USD 0.2/M output tokens |\n| MixLLM (Wang et al., 2025) | AutoMix (Aggarwal et al., 2024), RouteLLM (Ong et al., 2025), Zooter (Lu et al., 2024), FORC, OptLLM, MetaLLM (Nguyen et al., 2025), Oracle | 97.25% of GPT-4 quality at 24.18% cost vs. 96.39% at 32.94% for best baseline (OptLLM) | RouterBench + Llama 3.1 8B/70B; 80/20 split |\n| PILOT (Panda et al., 2025) | All-to-one, LinUCB (Li et al., 2010), Epoch-Greedy, HybridLLM (Ding et al., 2024) | 93% of GPT-4 performance at 25% of its cost, multi-task RouterBench setting (a distinct single-task MMLU-only result reaches 86% at 27% of cost) | 11 LLMs (6 open-source, 5 proprietary) |\n| UniRoute (Jitkrittum et al., 2026) | ZeroRouter, K-NN, MLP/Matrix Factorization (oracle) | Two UniRoute variants reported: LearnedMap (0.711, the paper's primary method) and K-Means (0.712, a simpler variant) — both vs. 0.707 (K-NN), 0.689 (ZeroRouter); oracle baselines higher (0.720–0.723) | Random 50/50 train/test split of 11 LLMs |\n\n---\n## MT-Bench (Zheng et al., 2023)\n\n| Method | Baseline(s) compared | Performance / Cost | Evaluation conditions |\n|---|---|---|---|\n| BEST-Route (Ding et al., 2025) | N-label, N-class, clustering-based routing (all from Srivatsa et al., 2024) | 1.59% quality drop at 60% cost reduction vs. 5.89% for N-label; N-class/clustering never reduce cost | 8-LLM pool; GPT-4o reference; out-of-distribution evaluation |\n| Confidence-Driven Router (Zhang et al., 2025) | Random router, TO-Router, RouteLLM (Ong et al., 2025) | CPT(80%)=55.61 vs. 78.55 for random; USD 3.74 vs. USD 4.06 at that operating point | Strong=GPT-4, Weak=Mixtral-8x7B; cost = OpenAI API USD |\n| RouteLLM (Ong et al., 2025) | Random router, BERT, Causal LLM, SW Ranking, Matrix Factorization | Best variant (Matrix Factorization, Arena+Judge-augmented training) needs only 13.40% of queries routed to GPT-4 to recover 50% of the GPT-4-vs-Mixtral quality gap (CPT(50%)=13.40%) vs. 49.03% for random routing, a 60.4% APGR improvement over random | Strong=gpt-4-1106-preview, Weak=Mixtral 8x7B; cost = % of GPT-4 calls |\n| Zooter (Lu et al., 2024) | BMA (best single model on average), RMR (reward-model ranking; 6 reward-model variants) | 7.11/10 vs. 6.72 for BMA; infers 1 of 6 candidates vs. RMR's 6 | 6×13B LLAMA-based pool; GPT-4 judge |\n\n---\n\n### Metrics used (addition to the Metrics subsection)\n\n- **AUC** (Cascade Routing): area under the accuracy-cost trade-off curve, computed across multiple operating points; higher indicates a better trade-off.\n- **AIQ** (GreenServ): RouterBench's own cost-performance trade-off metric, averaged across willingness-to-pay operating points.\n- **APGR** (RouteLLM): Average Performance Gap Recovered, like AUC, an integral over cost budgets, but of a normalized weak-to-strong gap rather than raw quality.\n- **CPT(x%)** (Confidence-Driven Router): minimum fraction of queries that must be routed to the strong model to recover x% of the accuracy gap over the weak model alone; lower is more cost-effective.\n- **Area-under-deferral-curve** (UniRoute): the area under the curve relating the fraction of queries deferred to a larger model against resulting accuracy; higher indicates a better accuracy-deferral trade-off.\n\n---\n\n### Takeaways\n\nAcross both tables, every method outperforms at least one baseline under its own paper's protocol, but no metric, model-pool size, or cost basis is shared across rows. On RouterBench: AUC (Cascade Routing), AIQ (GreenServ), accuracy-at-a-cost-point (IRT-Router, MixLLM, PILOT), and area-under-deferral-curve (UniRoute) are different ways of scoring \"performance\" over pools ranging from an unspecified size up to all 11 RouterBench models. On MT-Bench: the four methods express their gains in different units: a quality-drop percentage at a chosen cost-reduction level (BEST-Route), calls-to-target-accuracy as a % of queries (RouteLLM) or converted to USD (Confidence-Driven Router, which adopted RouteLLM's own metric to compare against it directly), and a raw judge score (Zooter). Hence, drawing from the same dataset does not imply a shared evaluation protocol, which reveals a gap and opportunity for future work."},"title":{"value":"[Continued] Evaluation table"}},"parentInvitations":"TMLR/-/Official_Comment","tmdate":1782557854727,"tcdate":1782130094720,"writers":["TMLR","TMLR/Paper8645/Authors"],"signatures":["TMLR/Paper8645/Authors"],"forum":"ypRg1TvQaM","number":3,"license":"CC BY 4.0","cdate":1782130094720,"readers":["everyone"],"invitations":["TMLR/Paper8645/-/Official_Comment"],"mdate":1782557854727,"domain":"TMLR","replyto":"fUrBslKmYM","id":"etFpAhWQYC","forumContent":{"submission_length":{"value":"Long submission (more than 12 pages of main content)"},"venue":{"value":"Accepted by TMLR"},"code":{"value":"https://github.com/ymoslem/awesome-llm-routing-cascading"},"abstract":{"value":"The rapid growth of large language models (LLMs) with diverse capabilities, costs, and domains has created a critical need for intelligent model selection at inference time. While smaller models suffice for routine queries, complex tasks demand more capable models. However, static model deployment does not account for the complexity and domain of incoming queries, leading to suboptimal performance and increased costs. Dynamic routing systems that adaptively select models based on query characteristics have emerged as a solution to this challenge.\n\nThis survey provides a systematic analysis of multi-LLM routing and cascading approaches, focusing on systems that route queries across a pool of independently trained LLMs at inference time. We cover diverse routing paradigms, including query difficulty, human preferences, clustering, uncertainty quantification, reinforcement learning, multimodality, and cascading. For each paradigm, we analyze representative methods and examine key trade-offs. Beyond taxonomy, we introduce a conceptual framework that characterizes routing systems along three dimensions: when decisions are made, what information is used, and how they are computed. This perspective highlights that practical systems are often compositional, integrating multiple paradigms under operational constraints.\n\nOur analysis demonstrates that effective multi-LLM routing requires balancing competing objectives. Choosing the optimal routing strategy depends on deployment and computational constraints. Well-designed routing systems can outperform even the most powerful individual models by strategically leveraging specialized capabilities across models while maximizing efficiency gains. Meanwhile, open challenges remain in developing and evaluating routing mechanisms that generalize across diverse architectures, modalities, and applications."},"_bibtex":{"value":"@article{\nmoslem2026dynamic,\ntitle={Dynamic Model Routing and Cascading for Efficient {LLM} Inference: A Survey},\nauthor={Yasmin Moslem and John Kelleher},\njournal={Transactions on Machine Learning Research},\nissn={2835-8856},\nyear={2026},\nurl={https://openreview.net/forum?id=ypRg1TvQaM},\nnote={Survey Certification}\n}"},"title":{"value":"Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey"},"certifications":{"value":["Survey Certification"]},"pdf":{"value":"/pdf/45284c4cebe7a84da0d3496a3f9473496d786a9c.pdf"},"venueid":{"value":"TMLR"},"paperhash":{"value":"moslem|dynamic_model_routing_and_cascading_for_efficient_llm_inference_a_survey"},"authorids":{"value":["~Yasmin_Moslem1","~John_Kelleher1"]},"assigned_action_editor":{"value":"~Aaron_Klein1"},"authors":{"value":["Yasmin Moslem","John Kelleher"]}},"version":2},{"content":{"comment":{"value":"**Q1. Connections with GraphRouter are not well discussed: This work seems to use an identical (or near identical) network and graph setup compared to GraphRouter, but does not disclose this. Further, GraphRouter is not used as a baseline in the experiments.** \n\n**Response:** Thanks for your valuable questions and insightful feedback. We answer your questions in two parts.\n\n**[Connections with GraphRouter]:** PersonalizedRouter and GraphRouter differ fundamentally in their application scenarios. GraphRouter performs LLM routing using fixed cost/performance weight pairs (details are provided in Section 4.2), which restricts it to static user settings. As stated in our paper (Section 1, Instruction, Paragraph 2):\n\n>“GraphRouter constructs a heterogeneous graph from user interaction data and leverages a GNN to predict the most suitable LLM. However, a fundamental limitation is its inability to adequately model user preferences (see Table 1).”\n\nWhen extended to multi-user scenarios, GraphRouter would require training a separate model for each individual user, which is computationally inefficient and impractical. In contrast, PersonalizedRouter is explicitly designed for multi-user and personalized settings. It introduces user nodes into the graph (details are provided in Section 3.2), and edges are used to represent different types of preferences (e.g., performance-oriented, style-oriented). This enables PersonalizedRouter to learn latent user preferences from interaction data and achieve personalized LLM selection across multiple users, which is not considered in prior work, to the best of our knowledge.\n\n**[GraphRouter is not used as a baseline in the experiments.]:** In the revised version of the paper, we have also included GraphRouter as a baseline and conducted additional experiments under both the multi-cost-efficiency simulation strategy and the LLM-as-a-Judge strategy. \n\n**Q2. a. This paper claims \"[Current routers] fail to learn user preferences from interaction data\" in the abstract: However, if this is a core motivation, then a preference based method such as RouteLLM would be more appropriate.**  \n\n**b. discuss why direct preference methods (such as RouteLLM) \"fail to learn user preferences from interaction data\"**\n\n**Response:** We thank the reviewer for pointing this out and have revised the statement in the latest version to: \n\n>“[Current routers] fail to learn individual user preferences from interaction data.”\n\n It is important to note that our motivation is to enable LLM selection in multi-user routing scenarios by learning each user’s preferences over cost/performance trade-offs or response styles. RouteLLM is trained on Chatbot Arena preference data, which consists of votes from all users, but it does not model individual preferences at inference time. Moreover, RouteLLM’s inference relies on the performance gap between a strong and a weak model together with the cost threshold α, which corresponds to the first part of the original sentence.\n\n>“Current LLM selection methods typically optimize for a single fixed objective, such as performance, cost, or a trade-off between them, and fail to learn user preferences from interaction data.”\n\n In addition, RouteLLM is only applicable to binary routing between one strong and one weak LLM, and therefore cannot address multi-user, multi-LLM routing scenarios.\n\n**Q3. a. The experiments are really small scale: Just using 10 LLMs may arguably be fine, but just 9 users is too small by many orders of magnitude. In reality one might have thousands of users to route between the 10 LLMs (in the open router setting you might even have millions), while I do understand that training in the million user regime is intractable, one should still look at at least 1000 users (probably more) to showcase the router can potentially scale to such instances. Looking at the work right now, my fear is that the setup essentially designs dense tripartite graphs which might quickly reach their scaling limits.**  \n\n**b. The LLM-as-a-Judge method may introduce arbitrary bias into model evaluation due to the prompt design and the LLM judge model used. Arguably the \"correct\" way to use such a judge might be to check multiple LLMs and multiple prompts and combine their rankings.**   \n\n**c. Larger scale experiments (more users)**\n\n**Response:** Thanks for raising the above concerns. Following your suggestion, we conducted new large-scale experiments under both the multi-cost-efficiency simulation strategy and the LLM-as-a-Judge strategy. Under the multi-cost-efficiency simulation strategy, we evaluated 10 LLMs across 1,000 users. We also carried out new LLM-as-a-Judge experiments. To further mitigate the bias of relying on a single LLM as judge, we employed 3 different LLMs, each combined with two types of prompts, yielding 6 judge settings. The experimental results are presented in Table1."},"title":{"value":"Response to Reviewer h7mj (Part 1)"}},"parentInvitations":"TMLR/-/Official_Comment","tmdate":1758312024388,"tcdate":1758312024388,"writers":["TMLR","TMLR/Paper5500/Authors"],"signatures":["TMLR/Paper5500/Authors"],"forum":"W80eE3ArAl","number":7,"license":"CC BY 4.0","cdate":1758312024388,"readers":["everyone"],"invitations":["TMLR/Paper5500/-/Official_Comment"],"mdate":1758312024388,"domain":"TMLR","replyto":"WfA7D74GQL","id":"5UwgKdLNFi","forumContent":{"submission_length":{"value":"Regular submission (no more than 12 pages of main content)"},"venue":{"value":"Accepted by TMLR"},"code":{"value":"https://github.com/ulab-uiuc/PersonalizedRouter"},"supplementary_material":{"value":"/attachment/e4bbeecf2accf23bec60f721de3508b2cf488096.zip"},"abstract":{"value":"The growing number of Large Language Models (LLMs) with diverse capabilities and response styles provides users with a wider range of choices, which presents challenges in selecting appropriate LLMs, as user preferences vary in terms of performance, cost, and response style. Current LLM selection methods typically optimize for a single fixed objective, such as performance, cost, or a trade-off between them, and fail to learn individual user preferences from interaction data. To address these limitations, we propose PersonalizedRouter, a graph-based framework that models diverse user profiles and performs personalized LLM selection by leveraging interaction data that includes task context, queries, candidate LLMs, and user decisions. To capture contextual information between user queries and optimal LLMs, PersonalizedRouter converts the interaction data into a heterogeneous graph, where the relationships between different types of nodes are represented by edges. To evaluate adaptability across users, we design two strategies: the multi-cost-efficiency simulation strategy and the LLM-as-a-Judge strategy. In addition, we construct PersonaRoute-Bench, a large-scale benchmark with 1,000 simulated users and 10 LLMs. Experimental results show that PersonalizedRouter significantly outperforms existing LLM selection methods and surpasses the strongest methods by a large margin of 15.38% and 9.83% under two simulation strategies. On the PersonaRoute-Bench with 1,000 users, it further surpasses the best methods by 16.19% and 59.69% while maintaining higher efficiency. Moreover, PersonalizedRouter demonstrates strong few-shot generalization, achieving 64.81% and 85.80% of the fully trained model’s performance when adapting to new users and new LLMs."},"_bibtex":{"value":"@article{\ndai2025personalizedrouter,\ntitle={PersonalizedRouter: Personalized {LLM} Routing via Graph-based User Preference Modeling},\nauthor={Zhongjie Dai and Tao Feng and Jiaxuan You},\njournal={Transactions on Machine Learning Research},\nissn={2835-8856},\nyear={2025},\nurl={https://openreview.net/forum?id=W80eE3ArAl},\nnote={}\n}"},"title":{"value":"PersonalizedRouter: Personalized LLM Routing via Graph-based User Preference Modeling"},"pdf":{"value":"/pdf/1c7d7c4a7a37c3cd71672b102988c3788f67c8e7.pdf"},"venueid":{"value":"TMLR"},"paperhash":{"value":"dai|personalizedrouter_personalized_llm_routing_via_graphbased_user_preference_modeling"},"authorids":{"value":["~Zhongjie_Dai1","~Tao_Feng5","~Jiaxuan_You2"]},"assigned_action_editor":{"value":"~Yen-Chang_Hsu1"},"authors":{"value":["Zhongjie Dai","Tao Feng","Jiaxuan You"]}},"version":2},{"content":{"comment":{"value":"> W3. The baselines are insufficient. BERT-based router that directly routes prompt to multiple LLMs can be compared.\n\n**Additional experiments with RouteLLM.** We conducted additional experiments comparing Cascadia against RouteLLM with BERT-based router (https://github.com/lm-sys/RouteLLM) on Llama cascades (Llama3-8B → Llama3-70B) following the setup in Section 4.1 with average quality requirement of 80 on Traces 1 and 2. For fair comparison, we tune the deployment for each model for RouteLLM. Results show that Cascadia achieves on average **21.3%** lower SLO scale in achieving 95% SLO attainment (4.6, 2.8 vs. 5.8, 3.6) and **18.8%** higher throughput (2.2, 3.5 vs. 1.9, 2.9) compared to RouteLLM.\n\nCascadia's performance advantage stems from its **system-algorithm co-design** (Section 3): While RouteLLM focuses solely on routing optimization and fails to consider how system-side optimization (e.g., resource allocation, parallelism) impacts routing decisions and latency-quality trade-offs, Cascadia jointly optimizes both aspects for better end-to-end performance.\n\n\nThank you for your valuable suggestion; we have added this additional experiment to the paper (see Section 4.2 in page 8 and Appendix I in the revised manuscript)."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763641193026,"tcdate":1763641193026,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission12568/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission12568/Authors"],"forum":"wkrWbrHTJQ","number":5,"license":"CC BY 4.0","cdate":1763641193026,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission12568/-/Official_Comment"],"mdate":1763641193026,"domain":"ICLR.cc/2026/Conference","replyto":"32VpwIWmgt","id":"v4EBHeK2SD","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Distributed","Parallel","and Cluster Computing"]},"primary_area":{"value":"infrastructure, software libraries, hardware, systems, etc."},"abstract":{"value":"Recent advances in large language models (LLMs) have intensified the need to deliver both rapid responses and high-quality outputs. More powerful models yield better results but incur higher inference latency, whereas smaller models are faster yet less capable. Recent work proposes balancing this latency–quality trade-off using model cascades, which route simpler queries to smaller models and more complex ones to larger models. However, enabling efficient cascade serving remains challenging. Current frameworks lack effective mechanisms for handling (i) the huge and varying resource demands of different LLMs, (ii) the inherent heterogeneity of LLM workloads, and (iii) the co-optimization of system deployment and routing strategy.\n\nMotivated by these observations, we introduce Cascadia, a novel cascade serving framework designed explicitly to schedule request routing and deploy model cascades for fast, quality-preserving LLM serving. Cascadia employs a bi-level optimization method: at the deployment level, it uses a mixed-integer linear program to select resource allocations and parallelism strategies based on LLM information and workload characteristics; at the routing level, it applies a Chebyshev-guided method to iteratively co-optimize the routing strategy and the system deployment produced by the deployment level. Our extensive evaluation on diverse workload traces and different model cascades (DeepSeek and the Llama series) demonstrates that Cascadia significantly outperforms both single-model deployments and the state-of-the-art cascade serving baseline, achieving up to 4$\\times$ (2.3$\\times$ on average) tighter latency SLOs and up to 5$\\times$ (2.4$\\times$ on average) higher throughput while maintaining target answer quality."},"_bibtex":{"value":"@inproceedings{\njiang2026cascadia,\ntitle={Cascadia: An Efficient Cascade Serving System for Large Language Models},\nauthor={Youhe Jiang and Fangcheng Fu and Wanru Zhao and Stephan Rabanser and Jintao Zhang and Nicholas D. Lane and Binhang Yuan},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=wkrWbrHTJQ}\n}"},"title":{"value":"Cascadia: An Efficient Cascade Serving System for Large Language Models"},"pdf":{"value":"/pdf/a1ec00676c2c3a4e0e9b432a8676721778d8cd3d.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"jiang|cascadia_an_efficient_cascade_serving_system_for_large_language_models"},"authorids":{"value":["~Youhe_Jiang3","~Fangcheng_Fu1","~Wanru_Zhao1","~Stephan_Rabanser1","~Jintao_Zhang2","~Nicholas_D._Lane2","~Binhang_Yuan1"]},"authors":{"value":["Youhe Jiang","Fangcheng Fu","Wanru Zhao","Stephan Rabanser","Jintao Zhang","Nicholas D. Lane","Binhang Yuan"]}},"version":2},{"content":{"data_release":{"value":"We authorize the release of our submission and author names to the public in the event of acceptance."},"TLDR":{"value":"An SLM panel’s monitoring cost can change which system looks better: on GSM8K and MMLU, median comparisons favor Pyramid at equal strong-model calls but RouteLLM at equal modeled cost, over their respective supported operating points."},"venue":{"value":"SLM-Agents Poster"},"email_sharing":{"value":"We authorize the sharing of all author emails with the Program Chairs."},"keywords":{"value":["small language models","LLM cascades","model routing","anytime computation","metareasoning","monitoring cost","cost–accuracy tradeoffs"]},"submission_requirements":{"value":["format","anonymity","authorship","originality","ethics_and_llm"]},"abstract":{"value":"A panel cascade runs several small language models (SLMs) before deciding whether to call a stronger model. The panel cost is therefore incurred on every query. We use a frozen three-SLM Pyramid MoA configuration to study how the choice of resource-matching criterion affects system comparisons. Equal-call matching holds the number of strong-model calls fixed; matched-cost evaluation instead holds fixed the mean token-priced cost of recorded model-generation calls. Over their respective supported sets of operating points (anchor supports), the median difference against RouteLLM favors Pyramid R3 at equal calls on GSM8K and MMLU but favors RouteLLM after cost matching; RouteLLM is favored under both views on a restricted MBPP cohort. At matched modeled cost, the median comparator-minus-R3 gap over each policy's supported R3 anchors is positive on every benchmark for four comparator methods and a seeded-random baseline. A strong-model price-sensitivity analysis tests whether and where these median gaps change sign. These descriptive results concern one frozen configuration and one dated set of token prices; they do not measure full deployment cost or rank intrinsic router quality."},"_bibtex":{"value":"@inproceedings{\nkhaled2026the,\ntitle={The Monitor Is Not Free: When Equal-Call and Equal-Cost Evaluations Disagree in {SLM}-to-{LLM} Cascades},\nauthor={Arindam Khaled},\nbooktitle={SLM-Agents: 1st NeurIPS Workshop on SLMs for Agentic Systems},\nyear={2026},\nurl={https://openreview.net/forum?id=lg33voDNWb}\n}"},"title":{"value":"The Monitor Is Not Free: When Equal-Call and Equal-Cost Evaluations Disagree in SLM-to-LLM Cascades"},"paper_type":{"value":"Long paper — up to 6 content pages"},"pdf":{"value":"/pdf/9a9e0228b57eac3371d9a4a9df6bb7152a9331da.pdf"},"venueid":{"value":"NeurIPS.cc/2026/Workshop/SLM-Agents"},"paperhash":{"value":"khaled|the_monitor_is_not_free_when_equalcall_and_equalcost_evaluations_disagree_in_slmtollm_cascades"},"primary_subject_area":{"value":"Inference systems, efficiency, and deployment"},"publication_policy":{"value":"We understand that the workshop is non-archival and authorize public release of the accepted submission, its metadata, and author names."},"authors":{"value":[{"username":"~Arindam_Khaled2","fullname":"Arindam Khaled","institutions":[{"name":"Seekr","domain":"seekr.com","country":"US"}]}]}},"tmdate":1791132911846,"pdate":1791132907674,"tcdate":1788465480201,"writers":["NeurIPS.cc/2026/Workshop/SLM-Agents","NeurIPS.cc/2026/Workshop/SLM-Agents/Submission104/Authors"],"signatures":["NeurIPS.cc/2026/Workshop/SLM-Agents/Submission104/Authors"],"forum":"lg33voDNWb","license":"CC BY 4.0","number":104,"cdate":1788465480201,"readers":["everyone"],"invitations":["NeurIPS.cc/2026/Workshop/SLM-Agents/-/Submission","NeurIPS.cc/2026/Workshop/SLM-Agents/-/Submission_Change_Before_Bidding","NeurIPS.cc/2026/Workshop/SLM-Agents/-/Submission_Change_Before_Reviewing","NeurIPS.cc/2026/Workshop/SLM-Agents/Submission104/-/Camera_Ready_Revision","NeurIPS.cc/2026/Workshop/SLM-Agents/-/Submission_Release"],"mdate":1791132911846,"odate":1791132907674,"domain":"NeurIPS.cc/2026/Workshop/SLM-Agents","id":"lg33voDNWb","version":2},{"content":{"summary":{"value":"This paper proposes Cognitive-Inspired Elastic Reasoning (CogER), a framework for dynamically allocating computational resources in LLM reasoning based on query complexity. The authors classify queries into four complexity levels (L1-L4) inspired by Bloom's Taxonomy, train a 7B CogER-Agent using reinforcement learning to route queries to appropriate processing modes (ranging from direct answering to tool-augmented reasoning), and introduce Cognitive Tool-Assisted Reasoning (CoTool) for L4 queries. Experiments on mathematical reasoning and commonsense QA tasks show improvements over test-time scaling baselines with reduced computational cost."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"## Q1: L_min Label Generation Methodology\n\n Please provide a complete, step-by-step specification of how L_min labels are generated for the 8,000 training samples:\n\n- What is the exact procedure? (Human annotation / Empirical testing / Heuristic rules?)\n\nL_min is the core supervision signal and the method cannot be reproduced without this information.\n\n## Q2: Routing Accuracy Validation\n\nTable 4 reports level assignment distributions (2% / 28.17% / 21.90% / 47.93%) but never validates routing correctness. Please provide:\n\n- **Confusion matrix**: Predicted level vs. ground truth L_min for test set\n- **Per-level metrics**: Precision, recall, and F1 for L1/L2/L3/L4 classification\n\nWithout these metrics, we cannot assess whether the system truly learns complexity classification or succeeds through other factors (ensemble effects, tool usage).\n\n## Q3: Router Capability Boundary Learning\n\nThe 7B router must predict whether queries require 32B/QwQ models, yet only receives binary success signals. Please address:\n\n- **Learning mechanism**: How does the 7B router learn the capability boundaries of stronger models it never observes reasoning? GRPO provides only correct/incorrect feedback—no reasoning traces, no intermediate steps.\n\n- **Table 4 explanation**: Without R_hierarchy, 88.46% of queries route to L4. Does this indicate:\n  - The router genuinely assesses most queries as requiring tools?\n  - Or risk-aversion (\"when uncertain, pick the strongest option\")?\n  \n  If the latter, how can we trust that R_hierarchy teaches genuine complexity understanding rather than merely forcing cheaper selections through penalty?\n\n- **Router size ablation**: Have you tested routing with different model sizes (1.8B / 7B / 14B / 32B)?\n  - If larger routers improve accuracy → confirms stronger models make better judges (but undermines cost savings)\n  - If accuracy plateaus at 7B → important empirical finding worth reporting\n  - If 1.8B performs comparably → suggests learned heuristics rather than deep understanding\n\n- **Comparison to RouteLLM**: RouteLLM uses preference data to teach routers about capability differences. Why is GRPO's binary feedback sufficient when RouteLLM requires richer training signals?\n\n## Q4: Baseline Comparisons and Fairness\n\n- **Missing routing baselines**: Why not compare to:\n  - RouteLLM (ICLR 2025) - directly comparable routing framework\n\n- Can you compare against DeepSeek-R1-Distill models (7B/14B/32B) for fairer scale comparison?\n\n## Q5: Generalization Beyond Training Distribution\n\n- **Out-of-distribution tasks**: Training uses only math + commonsense + medical QA. How does CogER perform on:\n  - Code generation (HumanEval, MBPP)\n  - Long-context reasoning (QuALITY, NarrativeQA)\n  - Multi-turn dialogue (MT-Bench)\n  - Factual QA with retrieval (Natural Questions)\n  \n  The current 2 OOD datasets (MAWPS, CollegeMath) are still mathematical—not true distribution shift.\n\n- **Training data scale**: You use 8K samples. Have you:\n  - Ablated training data size to show performance saturates at 8K?\n  - Tested whether more diverse training data improves routing generalization?\n\n- **Dataset artifact learning**: Training on GSM8K/MATH then testing on same benchmarks risks learning dataset-specific patterns (e.g., \"MATH → L3\" heuristic). How do you ensure the router learns generalizable complexity assessment rather than dataset shortcuts?\n\n## Q6: Model Upgrade and Long-term Viability\n\nThe 7B router is trained on specific model capabilities (Qwen2.5-7B/32B, QwQ-32B):\n\n- **Model upgrades**: If Qwen3.0-7B surpasses Qwen2.5-32B in capability, how should the system adapt?\n  - Does the router require complete retraining?\n  - Can L_min labels transfer across model versions?\n\n- **Cross-model generalization**: Can the trained router generalize to different model families (e.g., Llama, Mistral)?\n\n- **Failure modes**: When does CogER perform worse than baselines? Please provide:\n  - Case studies of routing failures\n  - Characterization of query types most prone to misclassification"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. **Well-Motivated Problem**: Dynamic resource allocation in test-time compute is an important practical challenge. The paper clearly articulates the inefficiency of one-size-fits-all reasoning strategies.\n\n2. **Comprehensive System Design**: The framework includes multiple well-integrated components: complexity classification, MDP formulation, specialized reward functions (particularly R_hierarchy for cost-awareness), and CoTool for tool integration. Algorithm 1 provides clear implementation guidance.\n\n3. **Solid Experimental Results**: Achieves competitive accuracy (89.28% on ID tasks) with significant efficiency gains (118.53s average latency vs. 506.19s for DeepSeek-R1). The OOD evaluation on MAWPS and CollegeMath demonstrates some generalization capability.\n\n4. **Detailed Reproducibility Information**: Appendix B provides complete prompts, Appendix D specifies RSTKit tools, and implementation details (hyperparameters, training procedures) are thoroughly documented.\n\n5. **Ablation Studies**: Tables 2-5 examine the contribution of different components (routing levels, reward terms, CoTool), providing insights into what drives performance."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"## 1. Inadequate Baseline Selection\n\n**Missing critical routing baselines**: The paper omits comparisons to directly relevant work:\n- **RouteLLM (ICLR 2025)**: Uses preference-based training for LLM routing with similar objectives\n\n**Unfair comparisons**:\n- DeepSeek-R1 is a closed-source 671B model tested under unknown conditions; should compare against open DeepSeek-R1-Distill (7B/14B/32B)\n- No iso-compute baseline: should compare \"always QwQ-32B\" with same average compute budget as CogER\n\n## 2. Narrow Dataset Selection\n\n**Limited task coverage**: Evaluation restricted to math reasoning and QA. Missing:\n- Code generation (HumanEval, MBPP)\n- Long-context reasoning (QuALITY, NarrativeQA)\n- Multi-turn dialogue (MT-Bench)\n- Factual QA with retrieval (Natural Questions, TriviaQA)\n- Multimodal tasks (ScienceQA, MMMU)\n\n**Train-test contamination risk**: Training uses samples from GSM8K/MATH/CommonsenseQA, then evaluates on same benchmark test sets. Router may learn dataset artifacts rather than complexity.\n\n**Weak OOD evaluation**: Only 2 OOD datasets (MAWPS, CollegeMath), both still mathematical. Needs different task types.\n\n**Small scale**: Only 8K training samples total; no ablation showing this suffices.\n\n## 3. The L_min Determination Black Box\n\n**Core methodology undefined**: Equation (6) depends on L_min(S) but never explains how these labels are obtained for 8,000 training samples. Possible methods, both problematic:\n\n- **Human annotation**: No guidelines, inter-annotator agreement, or subjectivity handling reported\n\n- **Empirical testing**: How is stochasticity handled? (Same query may succeed/fail across runs). What success threshold? (50%? 80%? 100%?). Circular dependency: need models to determine L_min for training those models.\n\n## 4. Router Capability Paradox\n\n**Weak-model-judging-strong-model**: 7B router must assess whether queries require 32B models or tools—but how can it judge capabilities beyond its own? Missing:\n- Analysis of how 7B learns 32B/QwQ capability boundaries from binary success signals alone\n- Routing accuracy stratified by true complexity (likely degrades for hard queries where accurate routing matters most)\n- Router size ablation (1.8B/7B/14B/32B)\n\n**Table 4 evidence**: Without R_hierarchy, 88.46% of queries route to L4. This reveals risk-aversion (\"when uncertain, pick strongest\") rather than genuine complexity understanding. R_hierarchy forces cheaper selections via penalty, not learned understanding.\n\n**Training signal insufficiency**: GRPO provides only correct/incorrect feedback, never showing how stronger models reason. Compare to RouteLLM's preference-based training that explicitly teaches capability differences.\n\n**No routing accuracy validation**: Table 4 shows level distributions but never reports:\n- Confusion matrix vs. ground truth L_min\n- Per-level precision/recall\n- Error decomposition (routing mistakes vs. execution failures)\n\nThis is the fundamental metric for any routing system—its absence is critical."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762915584849,"tcdate":1761923843848,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission705/Reviewer_opSX"],"signatures":["ICLR.cc/2026/Conference/Submission705/Reviewer_opSX"],"forum":"z9HUMon2Br","number":4,"license":"CC BY 4.0","cdate":1761923843848,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission705/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762915584849,"domain":"ICLR.cc/2026/Conference","replyto":"z9HUMon2Br","id":"M93xVmPWSg","forumContent":{"TLDR":{"value":"We propose the  Cognitive-Inspired Elastic Reasoning framework for efficient scaling of language model reasoning, which dynamically selects the most appropriate processing mode for each query."},"venue":{"value":"ICLR 2026 Conference Desk Rejected Submission"},"keywords":{"value":["Large language models; Reinforcement Learning; Markov decision process"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Large language models (LLMs) have demonstrated impressive performance across various language tasks. However, existing LLM reasoning strategies mainly rely on the LLM itself with fast or slow mode (like o1 thinking) and thus struggle to balance reasoning efficiency and accuracy across queries of varying difficulties. In this paper, we propose Cognitive-Inspired Elastic Reasoning (CogER), a framework inspired by human hierarchical reasoning that dynamically selects the most suitable reasoning strategy for each query. Specifically, CogER first assesses the complexity of incoming queries and assigns them to one of several predefined levels, each corresponding to a tailored processing strategy, thereby addressing the challenge of unobservable query difficulty. To achieve automatic strategy selection, we model the process as a Markov Decision Process and train a CogER-Agent using reinforcement learning. The agent is guided by a reward function that balances solution quality and computational cost, ensuring resource-efficient reasoning. Moreover, for queries requiring external tools, we introduce Cognitive Tool-Assisted Reasoning, which enables the LLM to autonomously invoke external tools within its chain-of-thought. Extensive experiments demonstrate that CogER outperforms state‑of‑the‑art Test‑Time scaling methods, achieving at least a 13% relative improvement in average exact match on In‑Domain tasks and an 8% relative gain on Out‑of‑Domain tasks."},"_bibtex":{"value":"@misc{\nanonymous2026beyond,\ntitle={Beyond Fast and Slow: Cognitive-Inspired Elastic Reasoning for Large Language Models},\nauthor={Anonymous},\nyear={2026},\nurl={https://openreview.net/forum?id=z9HUMon2Br}\n}"},"title":{"value":"Beyond Fast and Slow: Cognitive-Inspired Elastic Reasoning for Large Language Models"},"pdf":{"value":"/pdf/1252c0b740227fa4d6c59edd0735bd20fd07cbe8.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Desk_Rejected_Submission"},"paperhash":{"value":"hu|beyond_fast_and_slow_cognitiveinspired_elastic_reasoning_for_large_language_models"},"authorids":{"value":["~Jinwu_Hu1","~Dongjin_Yang1","~LangYu_Bian1","~Zhiquan_Wen1","~Yufeng_Wang5","~Yaofo_Chen1","~Bin_Xiao4","~Yuanqing_Li2","~Mingkui_Tan2"]},"authors":{"value":["Jinwu Hu","Dongjin Yang","LangYu Bian","Zhiquan Wen","Yufeng Wang","Yaofo Chen","Bin Xiao","Yuanqing Li","Mingkui Tan"]}},"version":2},{"content":{"comment":{"value":"***W1. Missing baselines. It seems that the paper is missing some baselines achieving the personalization through graph. For example, Knowledge Graph Tuning: Real-time Large Language Model Personalization based on Human Feedback.***\n\n**Author Response:**\n\nWe thank the reviewer for this helpful suggestion to include a graph-based personalization baseline. We have added a discussion of KGT in ***§ 6 Line 517-519***. However, since KGT is not open-sourced and its paper omits some implementation details, **we, instead, adopt another graph-based personalization method, MAGNN [1], as our new baseline *[Detailed explaination in Line 338-342]***. Results are shown in the table below and have been integrated into ***Table 2*** in the paper accordingly.\n\n***W2. This paper is claiming it is a LLM routing based model, however, it does not compare to baselines of routed LLMs. For example, RouteLLM: Learning to Route LLMs with Preference Data.***\n\n**Author Response:**\n\nThanks for this constructive suggestion to include a routed LLM baseline. We would first like to clarify that our method is **a graph-based framework for LLM routing**, rather than an LLM-routing-based model. Following the reviewer’s suggestion, **we added RouteLLM[2] as a new baseline and conducted the corresponding experiments *[Detailed explaination in Line 335-337]***. \n\nThe new results are presented in the table below and have been incorporated into ***Table 2*** of the paper.\n\n**Table of RouteLLM & MA-GNN & GMTRouter performance across datasets**\n| Model     | Chatbot-Arena | MT-Bench | GSM8K | MMLU | LaMP |\n|------------|:-------------:|:--------:|:-----:|:----:|:----:|\n|            | ACC / AUC     | ACC / AUC | ACC / AUC | ACC / AUC | ACC / AUC |\n| RouteLLM   | 0.492 / 0.485 | 0.480 / 0.475 | 0.499 / 0.498 | 0.532 / 0.513 | - |\n| MA-GNN   | 0.673 / 0.775 | 0.679 / 0.739 | 0.636 / 0.648 | 0.702 / 0.758 | 0.347 / 0.661 |\n| GMTRouter  | 0.774 / 0.875 | 0.784 / 0.859 | 0.773 / 0.859 | 0.771 / 0.870 | 0.349 / 0.662 |\n\n*Note that RouteLLM is designed for routing between two LLMs, whereas the LaMP dataset requires routing among five LLMs; therefore, RouteLLM was not evaluated on LaMP.*\n\nThese additions provide a more comprehensive comparison and demonstrate GMTRouter’s effectiveness across routing paradigms.\n\n**[1]** Ma, Chen, et al. \"Memory augmented graph neural networks for sequential recommendation.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 34. No. 04. 2020.\n\n**[2]** Ong, Isaac, et al. \"Routellm: Learning to route llms with preference data.\" arXiv preprint arXiv:2406.18665 (2024)."},"title":{"value":"Author response to Reviewer s59p (1/2)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764045812476,"tcdate":1764045478792,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission14657/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission14657/Authors"],"forum":"oRe9tHpSpn","number":4,"license":"CC BY 4.0","cdate":1764045478792,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission14657/-/Official_Comment"],"mdate":1764045812476,"domain":"ICLR.cc/2026/Conference","replyto":"P3WLMUKrsc","id":"ZeiecNpGO4","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Heterogeneous GNN","LLM routing","Preference Learning"]},"primary_area":{"value":"learning on graphs and other geometries & topologies"},"abstract":{"value":"Large Language Model (LLM) routing has demonstrated strong capability in balancing response quality with computational cost. As users exhibit diverse preferences, personalization has attracted increasing attention in LLM routing, since even identical queries may require different models to generate responses tailored to individual needs. However, existing approaches are not fully personalized and often fail to faithfully capture the complex interactions between specific users and LLMs. Moreover, user preference data is typically scarce, noisy, and inconsistent in format, which limits the effectiveness of methods that rely solely on user-specific data. To address these challenges, we propose GMTRouter, which represents multi-turn user–LLM interactions as a heterogeneous graph with four node types: user, LLM, query, and response, thereby maximally preserving the rich relational structure of the interaction. Through a tailored message-passing mechanism, GMTRouter learns to capture user preferences from few-shot data within a lightweight inductive graph learning framework, enabling effective personalization. Extensive experiments demonstrate that GMTRouter consistently outperforms the strongest baselines, achieving 0.9%–21.6% higher accuracy and 0.006–0.309 higher AUC across multiple datasets. More importantly, we further demonstrate that GMTRouter can adapt to new users and evolving preferences using only few-shot data, without extensive fine-tuning."},"_bibtex":{"value":"@misc{\nxie2026gmtrouter,\ntitle={{GMTR}outer: Personalized {LLM} Router over Multi-turn User Interactions},\nauthor={Encheng Xie and Yihang Sun and Tao Feng and Jiaxuan You},\nyear={2026},\nurl={https://openreview.net/forum?id=oRe9tHpSpn}\n}"},"title":{"value":"GMTRouter: Personalized LLM Router over Multi-turn User Interactions"},"pdf":{"value":"/pdf/b4f227b8ee59ab4f2a14d8fc028c0e06e9455c44.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"xie|gmtrouter_personalized_llm_router_over_multiturn_user_interactions"},"authorids":{"value":["~Encheng_Xie1","~Yihang_Sun2","~Tao_Feng5","~Jiaxuan_You2"]},"authors":{"value":["Encheng Xie","Yihang Sun","Tao Feng","Jiaxuan You"]}},"version":2},{"content":{"summary":{"value":"This paper uses a VLM to attempt to overcome image degradations due to environmental conditions, and correspondingly improve VSLAM performance in settings with poor observations"},"soundness":{"value":1},"confidence":{"value":5},"questions":{"value":"N/A"},"rating":{"value":0},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":1},"strengths":{"value":"It is hard to find a genuine strength among the review criteria set of originality, quality, clarity, or significance."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The main weakness with this approach is that its validity is highly questionable. Accurate VSLAM requires accurate feature tracking from the environment. An input image manipulated by a VLM is will have no guarantees on consistency or accuracy wrt real world geometry. At best, some simple image processing may help, such as gamma correction, but in such cases, the use of a VLM is overkill - autoexposure algorithms are a better bet. For anything non-trivial, such as distortion due to rain or fog, I would not trust the manipulated image from a VLM for performing VSLAM downstream.\n\nThe RouteLLMs claim is that per-frame choice of depth estimation algorithm should improve consistency, which does not make sense - if the algorithm changes per frame, the output depth would vary significantly between frames too."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1764358930270,"tcdate":1761956529858,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7039/Reviewer_TLuo"],"signatures":["ICLR.cc/2026/Conference/Submission7039/Reviewer_TLuo"],"forum":"duIZUIRKJB","number":4,"license":"CC BY 4.0","cdate":1761956529858,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7039/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1764358930270,"domain":"ICLR.cc/2026/Conference","replyto":"duIZUIRKJB","id":"YWnnBuogUr","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Deep SLAM; Multi-agent; Deep Estimation ;3DGS"]},"primary_area":{"value":"applications to robotics, autonomy, planning"},"abstract":{"value":"Visual Simultaneous Localization and Mapping (SLAM) reconstructs the metric structure of the physical world from sensor imagery, enabling precise robotic pose estimation. However, environmentally induced image degradation and varied image processing strategie significantly compromise localization accuracy. Intelligent SLAM systems address this challenge by autonomously perceiving dynamic perturbations and formulating adaptive processing strategies, further identifying and deploying optimal methodologies to achieve target localization objectives with enhanced metric precision. This paper introduces MAV-SLAM, a novel Multi-LLM-Agent-Orchestrated visual SLAM framework that proactively identifies and compensates for suboptimal image quality while autonomously selecting optimal depth estimation models. Specifically, we integrate a visual-language model that performs autonomous image restoration guided by image quality assessment, significantly enhancing SLAM localization performance. Furthermore, we implement a routing large language model for adaptive depth estimation, which consequently elevates the quality of 3D reconstruction via 3D Gaussian Splatting (3DGS). Rigorous evaluation across multiple benchmarks demonstrates that MAV-SLAM exhibits superior performance in both localization accuracy and 3DGS-based reconstruction fidelity, validating its effectiveness in real-world scenarios."},"_bibtex":{"value":"@misc{\nyao2026mavslam,\ntitle={{MAV}-{SLAM}: Multi-{LLM}-Agent Crew for Visual {SLAM} with 3D Gaussian Splatting},\nauthor={Jian Yao and Lisha Tang and Yang Li and Danping Zou},\nyear={2026},\nurl={https://openreview.net/forum?id=duIZUIRKJB}\n}"},"title":{"value":"MAV-SLAM: Multi-LLM-Agent Crew for Visual SLAM with 3D Gaussian Splatting"},"pdf":{"value":"/pdf/8fccc6924a5fddfae91e0c152e944d9a70c4b58f.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"yao|mavslam_multillmagent_crew_for_visual_slam_with_3d_gaussian_splatting"},"authorids":{"value":["~Jian_Yao9","~Lisha_Tang1","~Yang_Li26","~Danping_Zou1"]},"authors":{"value":["Jian Yao","Lisha Tang","Yang Li","Danping Zou"]}},"version":2},{"content":{"comment":{"value":"During our initial evaluation of RouteLLM, we allowed the query-level routers to fallback to the cheap model $M_w$ when the expensive model $M_s$ produced an incorrect answer, since such fallback does not increase expensive-token usage. However, to ensure consistency with standard practice in prior query-routing work, we have removed this fallback behavior. In the revised experiments, each query-level router of RouteLLM now commits to exactly one model for the entire generation and this change lowers the performance of baselines (performance of all TRIM methods remains unchanged). We have updated the results accordingly and expanded the baseline coverage in the revised version of the paper. Importantly, this modification does not affect any of our earlier conclusions: query-level routers such as those in RouteLLM still exhibit weak cross-dataset generalization, often fitting to dataset-specific characteristics rather than capturing transferable routing behavior."},"title":{"value":"Updated Baseline Evaluation"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763794685576,"tcdate":1763794685576,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission15697/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission15697/Authors"],"forum":"MoKrugWUfC","number":7,"license":"CC BY 4.0","cdate":1763794685576,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission15697/-/Official_Comment"],"mdate":1763794685576,"domain":"ICLR.cc/2026/Conference","replyto":"MoKrugWUfC","id":"wUcSeFTPGQ","forumContent":{"TLDR":{"value":"We introduce TRIM, a routing approach for multi-step reasoning that selectively routes only critical steps to larger LLMs based on uncertainty and budget."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Large Language Models","Model routing","Reinforcement Learning","Partially Observable MDPs (POMDPs)"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Multi-step reasoning tasks like mathematical problem solving are vulnerable to cascading failures, where a single incorrect step leads to complete solution breakdown. Current LLM routing methods assign entire queries to one model, treating all reasoning steps as equal. We propose TRIM (Targeted routing in multi-step reasoning tasks), which routes only critical steps$\\unicode{x2013}$those likely to derail the solution$\\unicode{x2013}$to larger models while letting smaller models handle routine continuations. Our key insight is that targeted step-level interventions can fundamentally transform inference efficiency by confining expensive calls to precisely those steps where stronger models prevent cascading errors. TRIM operates at the step-level: it uses process reward models to identify erroneous steps and makes routing decisions based on step-level uncertainty and budget constraints. We develop several routing strategies within TRIM, ranging from a simple threshold-based policy to more expressive policies that reason about long-horizon accuracy-cost trade-offs and uncertainty in step-level correctness estimates. On MATH-500, even the simplest thresholding strategy surpasses prior routing methods with 5x higher cost efficiency, while more advanced policies match the strong, expensive model's performance using 80% fewer expensive model tokens. On harder benchmarks such as AIME, TRIM achieves up to 6x higher cost efficiency. All methods generalize effectively across math reasoning tasks, demonstrating that step-level difficulty represents fundamental characteristics of reasoning."},"_bibtex":{"value":"@inproceedings{\nkapoor2026trim,\ntitle={{TRIM}: Hybrid Inference via Targeted Stepwise Routing in Multi-Step Reasoning Tasks},\nauthor={Vansh Kapoor and Aman Gupta and Hao Chen and Anurag Beniwal and Jing Huang and Aviral Kumar},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=MoKrugWUfC}\n}"},"title":{"value":"TRIM: Hybrid Inference via Targeted Stepwise Routing in Multi-Step Reasoning Tasks"},"pdf":{"value":"/pdf/dc645489055b4f58be6a0b3429b27c9e76a0fb36.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"kapoor|trim_hybrid_inference_via_targeted_stepwise_routing_in_multistep_reasoning_tasks"},"authorids":{"value":["~Vansh_Kapoor1","~Aman_Gupta5","~Hao_Chen119","~Anurag_Beniwal1","~Jing_Huang3","~Aviral_Kumar2"]},"authors":{"value":["Vansh Kapoor","Aman Gupta","Hao Chen","Anurag Beniwal","Jing Huang","Aviral Kumar"]}},"version":2},{"content":{"comment":{"value":"Many thanks to Reviewer 28jX for increasing the rating. We really appreciate it.\n\nFor the sake of completeness and just in case this comes up in later discussions, after discussing with and getting permission from the area chair, we have added the RouteLLM and RouterDC baseline results to the first two additional experiments that we provided in this rebuttal to Reviewer 28jX, where we were not able to gather the results during the one-week rebuttal period. The updated Tables I and II for our rebuttal to Reviewer 28jX are shown below. These more complete results underpin the effectiveness of MESS+ to minimize operating costs while maintaining SLA compliance over time. \n\nTable I extended: Mixed zoo with Llama 3 and Qwen 2 models (now including RouteLLM and RouterDC)\n| Category | Operating Cost | Request Satisfaction | Model Call Ratio (Q32B/L8B/L1B/Q0.5B) |\n|---|---|---|---|\n| Qwen2 0.5B | 0.15±0.01 | 54.12±45.15 | 0% / 0% / 0% / 100% |\n| Llama 3.2 1B | 0.20±0.01 | 61.13±4.79 | 0% / 0% / 100% / 0% |\n| Llama 3.1 8B | 0.50±0.01 | 67.07±4.61 | 0% / 100% / 0% / 0% |\n| Qwen2.5 32B | 1.99±0.01 | 70.91±4.48 | 100% / 0% / 0% / 0% |\n| Educated Guessing | 1.06±0.01 | 67.69±2.30 | 46% / 14% / 29% / 11% |\n| RouteLLM | 1.79±0.01 | 68.93±2.32 | 83% / 0% / 0% / 17% |\n| RouterDC | 1.31±0.01 | 69.17±2.25 | 72% / 11% / 10% / 7% |\n| **MESS+ (ours)** | **0.98±0.01** | 67.47±3.28 | 42% / 27% / 17% / 14% |\n\nTable II extended: Experiments on the Qwen 2 model zoo with a 2x squeezed cost ratio around the average inference cost per request (now including RouteLLM and RouterDC)\n| Category | Operating Cost | Request Satisfaction | Model Call Ratio (Q32B/Q7B/Q1.5B/Q0.5B) |\n|---|---|---|---|\n| Qwen2 0.5B | 0.15±0.01 | 54.12±45.15 | 0% / 0% / 0% / 100% |\n| Qwen2 1.5B | 0.20±0.01 | 61.13±4.79 | 0% / 0% / 100% / 0% |\n| Qwen2 7B | 0.50±0.01 | 67.07±4.61 | 0% / 100% / 0% / 0% |\n| Qwen2.5 32B | 1.99±0.01 | 70.91±4.48 | 100% / 0% / 0% / 0% |\n| Educated Guessing | 0.59±0.01 | 67.36±2.46 | 45% / 29% / 13% / 13% |\n| RouteLLM | 1.71±0.01 | 69.06±2.32 | 83% / 0% / 0% / 17% |\n| RouterDC | 1.05±0.01 | 70.73±2.30 | 71% / 17% / 11% / 0% |\n| **MESS+ (ours)** | **0.33±0.01** | 66.36±3.68 | 33% / 29% / 15% / 24% |\n\nWe will include all the additional results in the next version of this paper. Thank you again!"}},"parentInvitations":"NeurIPS.cc/2025/Conference/-/Official_Comment","tmdate":1761721870029,"tcdate":1754718382471,"writers":["NeurIPS.cc/2025/Conference","NeurIPS.cc/2025/Conference/Submission17761/Authors"],"signatures":["NeurIPS.cc/2025/Conference/Submission17761/Authors"],"forum":"wIM0y07NGX","number":9,"license":"CC BY 4.0","cdate":1754718382471,"readers":["everyone"],"invitations":["NeurIPS.cc/2025/Conference/Submission17761/-/Official_Comment","NeurIPS.cc/2025/Conference/-/Edit"],"mdate":1761721870029,"domain":"NeurIPS.cc/2025/Conference","replyto":"xPCIshkkIb","id":"CukIBTIWnh","forumContent":{"TLDR":{"value":"We propose a new algorithm that introduces guarantees for minimum user satisfaction rates in language model zoos while optimizing for operating cost, which can be practical for inference endpoint services."},"venue":{"value":"NeurIPS 2025 poster"},"keywords":{"value":["inference optimization","model routing","stochastic optimization algorithms","convergence analysis"]},"primary_area":{"value":"general_machine_learning"},"abstract":{"value":"Open-weight large language model (LLM) zoos provide access to numerous high-quality models, but selecting the appropriate model for specific tasks remains challenging and requires technical expertise. Most users simply want factually correct, safe, and satisfying responses without concerning themselves with model technicalities, while inference service providers prioritize minimizing operating costs. These competing interests are typically mediated through service level agreements (SLAs) that guarantee minimum service quality. \nWe introduce MESS+, a stochastic optimization algorithm for cost-optimal LLM request routing while providing rigorous SLA compliance guarantees. MESS+ learns request satisfaction probabilities of LLMs in real-time as users interact with the system, based on which model selection decisions are made by solving a per-request optimization problem. Our algorithm includes a novel combination of virtual queues and request satisfaction prediction, along with a theoretical analysis of cost optimality and constraint satisfaction.\nAcross a wide range of state-of-the-art LLM benchmarks, MESS+ achieves an average of $2\\times$ cost savings compared to existing LLM routing techniques."},"_bibtex":{"value":"@inproceedings{\nwoisetschlager2025mess,\ntitle={{MESS}+: Dynamically Learned Inference-Time {LLM} Routing in Model Zoos with Service Level Guarantees},\nauthor={Herbert Woisetschl{\\\"a}ger and Ryan Zhang and Shiqiang Wang and Hans Arno Jacobsen},\nbooktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},\nyear={2025},\nurl={https://openreview.net/forum?id=wIM0y07NGX}\n}"},"title":{"value":"MESS+: Dynamically Learned Inference-Time LLM Routing in Model Zoos with Service Level Guarantees"},"pdf":{"value":"/pdf/912c7e017b59dbbec1e0942b1b1bce704b7cd27f.pdf"},"venueid":{"value":"NeurIPS.cc/2025/Conference"},"paperhash":{"value":"woisetschläger|mess_dynamically_learned_inferencetime_llm_routing_in_model_zoos_with_service_level_guarantees"},"authorids":{"value":["~Herbert_Woisetschläger1","~Ryan_Zhang2","~Shiqiang_Wang1","~Hans_Arno_Jacobsen1"]},"authors":{"value":["Herbert Woisetschläger","Ryan Zhang","Shiqiang Wang","Hans Arno Jacobsen"]}},"version":2},{"content":{"comment":{"value":">Q9. Table 4 evidence: Without R_hierarchy, 88.46% of queries route to L4. This reveals risk-aversion (\"when uncertain, pick strongest\") rather than genuine complexity understanding. R_hierarchy forces cheaper selections via penalty, not learned understanding.\n\n**A9.** We thank the reviewer for this insightful comment, which directly relates to our core goal of achieving cognitively inspired elastic reasoning rather than a one-size-fits-all strategy.\n* **Table 4 highlights the overthinking problem that CogER is designed to solve.** Without $R_{\\text{hierarchy}}$, the policy indeed routes most queries to $L_4$. This behavior reflects the overthinking phenomenon that motivates our work, where optimizing purely for accuracy naturally drives the system to always choose the strongest and most expensive reasoning mode. Therefore, Table 4 supports our problem formulation and shows why an additional hierarchy-aware signal is necessary to realize elastic reasoning across the four levels.\n* **The hierarchy reward guides but does not hard-code complexity-aware routing.** $R_{\\text{hierarchy}}$ does not fix the level for any query. It is a soft cost term that trades off with the accuracy reward. If the router sends a query to a cheaper level that fails, the drop in accuracy makes the total reward low. To obtain a high return, the agent still has to learn which queries can be reliably solved at lower levels and when it is safer to stay at $L_4$. With $R_{\\text{hierarchy}}$, CogER uses all four levels and at the same time keeps or improves accuracy compared with the variant that almost always chooses $L_4$. This pattern indicates that the agent learns a non-trivial routing policy instead of only avoiding penalties.\n\n\n>Q10. RouteLLM uses preference data to teach routers about capability differences. Why is GRPO's binary feedback sufficient when RouteLLM requires richer training signals?\n\n**A10.** We thank the reviewer for raising this insightful comparison with RouteLLM. We clarify why GRPO with our composite reward is sufficient in our setting and how this differs from the preference based training used by RouteLLM, as follows.\n* **Composite reward and GRPO optimization.** The CogER Agent is trained with the combined reward $R_{\\text{format}} +R_{\\text{accuracy}} + R_{\\text{hierarchy}}$ under GRPO rather than a single binary signal. For each query, we generate multiple rollouts that use different reasoning levels and compute rewards within the same GRPO group. Trajectories that solve the query at a lower level obtain a higher total return because they receive a positive accuracy reward and a smaller hierarchical penalty. In contrast, trajectories that fail or rely on unnecessarily high levels receive a lower return. This group relative optimization induces a preference ordering among levels for each query, similar in spirit to preference data without explicitly constructing pairwise preference labels.\n* **Structured four-level hierarchy simplifies routing supervision.** RouteLLM learns routers from human preference data to distinguish capability differences between candidate models. In our CogER, the four routing options are not arbitrary experts but cognitively motivated reasoning modes L1–L4 that are defined by their behavior, including prompt answering, concise chain of thought, deep chain of thought, and tool-enhanced reasoning, with a monotonic increase in compute and typical capability. The estimated minimal sufficient level $L_{\\min}(S)$ and the hierarchical aware reward further encode the preference to use the simplest level that is usually sufficient for each query. This structured hierarchy and reward design provide a strong inductive bias for routing and make it possible to train an effective CogER Agent with GRPO without requiring additional human preference labels. At the same time, the concrete base models used to implement each level can be replaced or upgraded without changing this routing principle.\n* **Empirical evidence that GRPO-based feedback is effective.** To directly address the concern, we include RouteLLM as an additional routing baseline in the revised version (Table 1 in the paper). From Table A, under the same backbone and evaluation setup, **CogER outperforms RouteLLM** on both ID and OOD benchmarks. These results indicate that in our cognitive hierarchy setting, the GRPO-based correctness signal combined with the hierarchical aware reward is sufficient to learn an effective router in practice, even though RouteLLM uses richer human preference supervision."},"title":{"value":"Response to Reviewer opSX (Part 5)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763799482230,"tcdate":1763796580093,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission705/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission705/Authors"],"forum":"z9HUMon2Br","number":12,"license":"CC BY 4.0","cdate":1763796580093,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission705/-/Official_Comment"],"mdate":1763799482230,"domain":"ICLR.cc/2026/Conference","replyto":"M93xVmPWSg","id":"5EbtKHmu1R","forumContent":{"TLDR":{"value":"We propose the  Cognitive-Inspired Elastic Reasoning framework for efficient scaling of language model reasoning, which dynamically selects the most appropriate processing mode for each query."},"venue":{"value":"ICLR 2026 Conference Desk Rejected Submission"},"keywords":{"value":["Large language models; Reinforcement Learning; Markov decision process"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Large language models (LLMs) have demonstrated impressive performance across various language tasks. However, existing LLM reasoning strategies mainly rely on the LLM itself with fast or slow mode (like o1 thinking) and thus struggle to balance reasoning efficiency and accuracy across queries of varying difficulties. In this paper, we propose Cognitive-Inspired Elastic Reasoning (CogER), a framework inspired by human hierarchical reasoning that dynamically selects the most suitable reasoning strategy for each query. Specifically, CogER first assesses the complexity of incoming queries and assigns them to one of several predefined levels, each corresponding to a tailored processing strategy, thereby addressing the challenge of unobservable query difficulty. To achieve automatic strategy selection, we model the process as a Markov Decision Process and train a CogER-Agent using reinforcement learning. The agent is guided by a reward function that balances solution quality and computational cost, ensuring resource-efficient reasoning. Moreover, for queries requiring external tools, we introduce Cognitive Tool-Assisted Reasoning, which enables the LLM to autonomously invoke external tools within its chain-of-thought. Extensive experiments demonstrate that CogER outperforms state‑of‑the‑art Test‑Time scaling methods, achieving at least a 13% relative improvement in average exact match on In‑Domain tasks and an 8% relative gain on Out‑of‑Domain tasks."},"_bibtex":{"value":"@misc{\nanonymous2026beyond,\ntitle={Beyond Fast and Slow: Cognitive-Inspired Elastic Reasoning for Large Language Models},\nauthor={Anonymous},\nyear={2026},\nurl={https://openreview.net/forum?id=z9HUMon2Br}\n}"},"title":{"value":"Beyond Fast and Slow: Cognitive-Inspired Elastic Reasoning for Large Language Models"},"pdf":{"value":"/pdf/1252c0b740227fa4d6c59edd0735bd20fd07cbe8.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Desk_Rejected_Submission"},"paperhash":{"value":"hu|beyond_fast_and_slow_cognitiveinspired_elastic_reasoning_for_large_language_models"},"authorids":{"value":["~Jinwu_Hu1","~Dongjin_Yang1","~LangYu_Bian1","~Zhiquan_Wen1","~Yufeng_Wang5","~Yaofo_Chen1","~Bin_Xiao4","~Yuanqing_Li2","~Mingkui_Tan2"]},"authors":{"value":["Jinwu Hu","Dongjin Yang","LangYu Bian","Zhiquan Wen","Yufeng Wang","Yaofo Chen","Bin Xiao","Yuanqing Li","Mingkui Tan"]}},"version":2},{"content":{"comment":{"value":"We are deeply grateful to you for recognizing the strengths of our work, particularly the **well-motivated problem** of dynamic test-time resource allocation, the **comprehensive and well-integrated system design**, the **solid experimental results**, the **detailed reproducibility information**, and the ablation studies that help clarify the contribution of each component.\n\n>Q1. The paper omits comparisons to directly relevant work: RouteLLM (ICLR 2025): Uses preference-based training for LLM routing with similar objectives.\n\n**A1.** We thank the reviewer for highlighting the importance of comparing with RouteLLM, which is indeed a closely related routing-based approach. In the revised version, we include RouteLLM as an additional baseline and report the results in Table A (also added to Table 1 in the paper). From Table A, **CogER achieves better accuracy on both ID and OOD benchmarks.**  Specifically, CogER outperforms RouteLLM on all ID datasets and on the averaged EM score (89.28 vs. 86.58 EM), and also achieves higher average performance on OD datasets (93.56 vs. 92.60 EM). This indicates that even when compared against a strong routing-based method with similar efficiency objectives, CogER provides a better accuracy.\n\nTable A: Comparison between CogER and the LLM routing baseline RouteLLM on ID and OOD benchmarks.\n| Baseline           | GSM8K        | MATH         | Com-QA        | MedQA        | AVG (ID)      | MAWPS        | College      | AVG (OOD)    |\n|--------------------|-------------:|-------------:|--------------:|-------------:|--------------:|-------------:|-------------:|-------------:|\n| RouteLLM           | 95.80±0.05   | 87.29±0.12   | 83.99±0.19    | 79.22±0.32   | 86.58±0.20    | **97.90±0.00**   | 87.29±0.08   | 92.60±0.06   |\n| **CogER (Ours)**   | **96.18±0.05** | **95.20±0.20** | **84.52±0.30** | **81.23±0.00** | **89.28±0.18** | 97.87±0.01 | **89.24±0.14** | **93.56±0.10** |\n\n\n>Q2. Unfair comparisons:1) DeepSeek-R1 is a closed-source 671B model tested under unknown conditions; should compare against open DeepSeek-R1-Distill (7B/14B/32B). 2) No iso-compute baseline: should compare \"always QwQ-32B\" with same average compute budget as CogER.\n\n**A2.** We thank the reviewer for raising these important concerns about the fairness of our comparisons. We clarify the role of DeepSeek-R1, our use of DeepSeek-R1-Distill baselines, and the iso-compute comparison with “always QwQ-32B” as follows.\n* **Use of DeepSeek-R1 as a strong reference system.** We include DeepSeek-R1 (671B) because it is one of the most powerful publicly reported reasoning-centric LLMs and therefore serves as a meaningful reference point. From Table 1, **compared to DeepSeek-R1, CogER achieves a relative performance improvement of 9.48% in terms of average EM on ID tasks**, while using substantially lower average compute. \n* **Comparisons with open DeepSeek-R1-Distill models.** To provide fully reproducible and model-family-consistent baselines, we also compare against DeepSeek-R1-Distill 7B, 14B, and 32B. These results are reported in **Table 1** in the paper. Our **CogER consistently outperforms the DeepSeek-R1-Distill models** in terms of average EM.\n* **Iso-compute comparison with the “always QwQ-32B” baseline.** We agree that iso-compute baselines are important for a fair assessment. To address this, we explicitly include an **“always QwQ-32B”** baseline in **Table 2**, which uses the same QwQ-32B model as our higher levels and is evaluated under a comparable average compute budget. From Table 2, under a similar average compute in terms of latency and token usage, CogER achieves a higher average EM than “always QwQ-32B”."},"title":{"value":"Response to Reviewer opSX (Part 1)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763796273156,"tcdate":1763796273156,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission705/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission705/Authors"],"forum":"z9HUMon2Br","number":8,"license":"CC BY 4.0","cdate":1763796273156,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission705/-/Official_Comment"],"mdate":1763796273156,"domain":"ICLR.cc/2026/Conference","replyto":"M93xVmPWSg","id":"1td4jTxEqx","forumContent":{"TLDR":{"value":"We propose the  Cognitive-Inspired Elastic Reasoning framework for efficient scaling of language model reasoning, which dynamically selects the most appropriate processing mode for each query."},"venue":{"value":"ICLR 2026 Conference Desk Rejected Submission"},"keywords":{"value":["Large language models; Reinforcement Learning; Markov decision process"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Large language models (LLMs) have demonstrated impressive performance across various language tasks. However, existing LLM reasoning strategies mainly rely on the LLM itself with fast or slow mode (like o1 thinking) and thus struggle to balance reasoning efficiency and accuracy across queries of varying difficulties. In this paper, we propose Cognitive-Inspired Elastic Reasoning (CogER), a framework inspired by human hierarchical reasoning that dynamically selects the most suitable reasoning strategy for each query. Specifically, CogER first assesses the complexity of incoming queries and assigns them to one of several predefined levels, each corresponding to a tailored processing strategy, thereby addressing the challenge of unobservable query difficulty. To achieve automatic strategy selection, we model the process as a Markov Decision Process and train a CogER-Agent using reinforcement learning. The agent is guided by a reward function that balances solution quality and computational cost, ensuring resource-efficient reasoning. Moreover, for queries requiring external tools, we introduce Cognitive Tool-Assisted Reasoning, which enables the LLM to autonomously invoke external tools within its chain-of-thought. Extensive experiments demonstrate that CogER outperforms state‑of‑the‑art Test‑Time scaling methods, achieving at least a 13% relative improvement in average exact match on In‑Domain tasks and an 8% relative gain on Out‑of‑Domain tasks."},"_bibtex":{"value":"@misc{\nanonymous2026beyond,\ntitle={Beyond Fast and Slow: Cognitive-Inspired Elastic Reasoning for Large Language Models},\nauthor={Anonymous},\nyear={2026},\nurl={https://openreview.net/forum?id=z9HUMon2Br}\n}"},"title":{"value":"Beyond Fast and Slow: Cognitive-Inspired Elastic Reasoning for Large Language Models"},"pdf":{"value":"/pdf/1252c0b740227fa4d6c59edd0735bd20fd07cbe8.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Desk_Rejected_Submission"},"paperhash":{"value":"hu|beyond_fast_and_slow_cognitiveinspired_elastic_reasoning_for_large_language_models"},"authorids":{"value":["~Jinwu_Hu1","~Dongjin_Yang1","~LangYu_Bian1","~Zhiquan_Wen1","~Yufeng_Wang5","~Yaofo_Chen1","~Bin_Xiao4","~Yuanqing_Li2","~Mingkui_Tan2"]},"authors":{"value":["Jinwu Hu","Dongjin Yang","LangYu Bian","Zhiquan Wen","Yufeng Wang","Yaofo Chen","Bin Xiao","Yuanqing Li","Mingkui Tan"]}},"version":2},{"content":{"comment":{"value":"> W1. In Algorithm 1, Cascadia relies on iteratively optimizing both deployment and routing strategies. It remains unclear if the bi-level optimization is theoretically optimal or not. Such guarantees could be critical in practical scenarios.\n\n**Theoretical optimality.** Due to the NP-hardness of the problem and the mutual dependencies between deployment and routing, providing theoretical optimality guarantees is intractable. However, we can empirically validate our approach against **exhaustive search**, which enumerates all feasible resource allocations, parallelism strategies, and routing thresholds, serving as an empirical optimum.\n\nSpecifically, we conducted additional experiments comparing our bi-level optimization against exhaustive search on Llama cascades (Llama3-8B → Llama3-70B) following the setup in Section 4.1. To make exhaustive search computationally feasible, we applied the same deployment constraints from Section 3.2. Results show that our approach achieves near-optimal performance with only **2-6%** gap compared to exhaustive search, while reducing search time from >5 minutes to ~20 seconds—a **>15×** speedup. Notably, exhaustive search time grows **exponentially** with cluster size, making our bi-level approach (grows **linearly**) essential for practical deployment at scale.\n\n**Practical advantages.** While theoretical optimality is intractable, our method provides strong empirical performance with practical efficiency, making it suitable for real-world deployment scenarios where search overhead matters.\n\nThank you for mentioning this; we have added this discussion to the paper (see Appendix J in the revised manuscript).\n\n> W2. LLM routing is another well-studied technique aiming for efficient & effective LLM serving, which is under-discussed in this paper. Authors may want to discuss and compare to this line of work to better position the contribution of this paper. Several example references are as follows,\n\n**Additional experiments with RouteLLM.** We conducted additional experiments comparing Cascadia against RouteLLM (https://github.com/lm-sys/RouteLLM) on Llama cascades (Llama3-8B → Llama3-70B) following the setup in Section 4.1 with average quality requirement of 80 on Traces 1 and 2. For fair comparison, we tune the deployment for each model for RouteLLM. Results show that Cascadia achieves on average **21.3%** lower SLO scale in achieving 95% SLO attainment (4.6, 2.8 vs. 5.8, 3.6) and **18.8%** higher throughput (2.2, 3.5 vs. 1.9, 2.9) compared to RouteLLM.\n\nCascadia's performance advantage stems from its **system-algorithm co-design** (Section 3): While RouteLLM focuses solely on routing optimization and fails to consider how system-side optimization (e.g., resource allocation, parallelism) impacts routing decisions and latency-quality trade-offs, Cascadia jointly optimizes both aspects for better end-to-end performance.\n\nThank you for your valuable suggestion; we have added this additional experiment to the paper (see Section 4.2 in page 8 and Appendix I in the revised manuscript).\n\n> Q1. What is the typical/expected number of iterations required to achieve stable solutions? If the number of iteration tends to be huge, it can lead to non-negligible overheads and compromise the efficiency gains.\n\n**Scheduling overhead.** The typical number of iterations required to achieve stable solutions is **~40 iterations**. We present a detailed analysis of the total scheduling overhead in Section 4.4 (Figure 10). In our setup (32 GPUs), scheduling completes within **20s**. For larger clusters (e.g., 80 GPUs), it finishes within **one minute**. These results demonstrate the algorithm’s efficiency and scalability across test cases and cluster sizes."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763641311567,"tcdate":1763641311567,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission12568/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission12568/Authors"],"forum":"wkrWbrHTJQ","number":6,"license":"CC BY 4.0","cdate":1763641311567,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission12568/-/Official_Comment"],"mdate":1763641311567,"domain":"ICLR.cc/2026/Conference","replyto":"pr2PCralpM","id":"p8ywXNHzeA","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Distributed","Parallel","and Cluster Computing"]},"primary_area":{"value":"infrastructure, software libraries, hardware, systems, etc."},"abstract":{"value":"Recent advances in large language models (LLMs) have intensified the need to deliver both rapid responses and high-quality outputs. More powerful models yield better results but incur higher inference latency, whereas smaller models are faster yet less capable. Recent work proposes balancing this latency–quality trade-off using model cascades, which route simpler queries to smaller models and more complex ones to larger models. However, enabling efficient cascade serving remains challenging. Current frameworks lack effective mechanisms for handling (i) the huge and varying resource demands of different LLMs, (ii) the inherent heterogeneity of LLM workloads, and (iii) the co-optimization of system deployment and routing strategy.\n\nMotivated by these observations, we introduce Cascadia, a novel cascade serving framework designed explicitly to schedule request routing and deploy model cascades for fast, quality-preserving LLM serving. Cascadia employs a bi-level optimization method: at the deployment level, it uses a mixed-integer linear program to select resource allocations and parallelism strategies based on LLM information and workload characteristics; at the routing level, it applies a Chebyshev-guided method to iteratively co-optimize the routing strategy and the system deployment produced by the deployment level. Our extensive evaluation on diverse workload traces and different model cascades (DeepSeek and the Llama series) demonstrates that Cascadia significantly outperforms both single-model deployments and the state-of-the-art cascade serving baseline, achieving up to 4$\\times$ (2.3$\\times$ on average) tighter latency SLOs and up to 5$\\times$ (2.4$\\times$ on average) higher throughput while maintaining target answer quality."},"_bibtex":{"value":"@inproceedings{\njiang2026cascadia,\ntitle={Cascadia: An Efficient Cascade Serving System for Large Language Models},\nauthor={Youhe Jiang and Fangcheng Fu and Wanru Zhao and Stephan Rabanser and Jintao Zhang and Nicholas D. Lane and Binhang Yuan},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=wkrWbrHTJQ}\n}"},"title":{"value":"Cascadia: An Efficient Cascade Serving System for Large Language Models"},"pdf":{"value":"/pdf/a1ec00676c2c3a4e0e9b432a8676721778d8cd3d.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"jiang|cascadia_an_efficient_cascade_serving_system_for_large_language_models"},"authorids":{"value":["~Youhe_Jiang3","~Fangcheng_Fu1","~Wanru_Zhao1","~Stephan_Rabanser1","~Jintao_Zhang2","~Nicholas_D._Lane2","~Binhang_Yuan1"]},"authors":{"value":["Youhe Jiang","Fangcheng Fu","Wanru Zhao","Stephan Rabanser","Jintao Zhang","Nicholas D. Lane","Binhang Yuan"]}},"version":2},{"content":{"summary":{"value":"The paper introduces a novel approach by combining causal inference with end-to-end regret minimization for LLM routing. It addresses a gap in the literature where existing methods (RouteLLM, CARROT) rely on full-feedback data that is impractical to obtain in real deployments"},"code_of_conduct_acknowledgement":{"value":"Yes"},"confidence":{"value":2},"originality":{"value":4},"questions":{"value":"How sensitive is the method to violations of key assumptions (Lipschitz continuity, unconfoundedness, affine utility)?\nHow does the method compare to more sophisticated baselines and recent routing advances?\nHow realistic is the accuracy-proportional sampling for simulating observational data?"},"rating":{"value":5},"paper_formatting_concerns":{"value":"Paper is well formatted."},"strengths_and_weaknesses":{"value":"Strengths\nThe theoretical framework is well-constructed and mathematically principled. The two surrogate objectives (classification-based upper bound and softmax-weighted regret) are motivated with formal propositions. Proposition 2's proof that the softmax surrogate recovers optimal treatment assignment provides crucial theoretical grounding. \n\nThe paper also addresses the reliance on expensive full-feedback datasets for current LLM routing. The treatment of observational data with historical routing policies is practically important and represents a significant advancement over existing approaches that assume unrealistic data availability.\n\nEvaluation on RouterBench and SPROUT benchmarks with multiple embedding models (BERT, LLaMA) is comprehensive and good. It includes reasonable baselines and the heterogeneous preference evaluation shows practical applicability.\n\nWeaknesses\nThere is no comparison with recent state-of-the-art methods like the latest RouteLLM variants or more sophisticated ensemble approaches. There is also limited analysis of individual component contributions (causal inference vs end-to-end learning vs regret minimization)\n\nThere are some assumptions made that may not be true for all models. The L-Lipschitz assumption may not hold for discrete model capabilities across diverse tasks. The restriction to linear cost functions limits applicability to more complex preference structures"},"quality":{"value":3},"significance":{"value":2},"responsible_reviewing_acknowledgement":{"value":"Yes"},"clarity":{"value":3},"limitations":{"value":"Authors mention restrictions to affine utility functions and suggesting extensions to hard constraints and online settings. However, there are a few more missing limitations. There is no discussion of robustness to causal inference assumption violations. There is limited analysis of scenarios where the method would perform poorly. This approach \n\nThe approach relies on strong assumptions including Lipschitz continuity of utility functions (which may not hold for discrete model capabilities), standard causal inference assumptions (unconfoundedness, overlap, SUTVA) that are difficult to verify in practice, and affine utility structures that may not capture complex user preferences. The convergence guarantees for surrogate objectives lack finite-sample bounds. The experimental evaluation relies on simulated observational data that may not reflect the complexity of real-world routing biases, limiting the generalizability of the empirical results."},"ethical_concerns":{"value":["NO or VERY MINOR ethics concerns only"]}},"parentInvitations":"NeurIPS.cc/2025/Conference/-/Official_Review","nonreaders":[],"tmdate":1761706397376,"tcdate":1751595163769,"writers":["NeurIPS.cc/2025/Conference","NeurIPS.cc/2025/Conference/Submission3861/Reviewer_J5tc"],"signatures":["NeurIPS.cc/2025/Conference/Submission3861/Reviewer_J5tc"],"forum":"iZC5xoQQkX","number":4,"license":"CC BY 4.0","cdate":1751595163769,"readers":["everyone"],"invitations":["NeurIPS.cc/2025/Conference/Submission3861/-/Official_Review","NeurIPS.cc/2025/Conference/-/Edit"],"mdate":1761706397376,"domain":"NeurIPS.cc/2025/Conference","replyto":"iZC5xoQQkX","id":"o7TNb2YJMo","forumContent":{"venue":{"value":"NeurIPS 2025 poster"},"keywords":{"value":["LLMs","Routing","Causal ML"]},"primary_area":{"value":"deep_learning"},"abstract":{"value":"LLM routing aims to select the most appropriate model for each query, balancing competing performance metrics such as accuracy and cost across a pool of language models. Prior approaches typically adopt a decoupled strategy, where the metrics are first predicted and the model is then selected based on these estimates. This setup is prone to compounding errors and often relies on full-feedback data, where each query is evaluated by all candidate models, which is costly to obtain and maintain in practice. In contrast, we learn from observational data, which records only the outcome of the model actually deployed. We propose a causal end-to-end framework that learns routing policies by minimizing decision-making regret from observational data. To enable efficient optimization, we introduce two theoretically grounded surrogate objectives: a classification-based upper bound, and a softmax-weighted regret approximation shown to recover the optimal policy at convergence. We further extend our framework to handle heterogeneous cost preferences via an interval-conditioned architecture. Experiments on public benchmarks show that our method outperforms existing baselines, achieving state-of-the-art performance across different embedding models."},"_bibtex":{"value":"@inproceedings{\ntsiourvas2025causal,\ntitle={Causal {LLM} Routing: End-to-End Regret Minimization from Observational Data},\nauthor={Asterios Tsiourvas and Wei Sun and Georgia Perakis},\nbooktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},\nyear={2025},\nurl={https://openreview.net/forum?id=iZC5xoQQkX}\n}"},"title":{"value":"Causal LLM Routing: End-to-End Regret Minimization from Observational Data"},"pdf":{"value":"/pdf/511a9b68d25f5e15c06a87acb5a057f90f002bf8.pdf"},"venueid":{"value":"NeurIPS.cc/2025/Conference"},"paperhash":{"value":"tsiourvas|causal_llm_routing_endtoend_regret_minimization_from_observational_data"},"authorids":{"value":["~Asterios_Tsiourvas1","~Wei_Sun7","~Georgia_Perakis1"]},"authors":{"value":["Asterios Tsiourvas","Wei Sun","Georgia Perakis"]}},"version":2},{"content":{"comment":{"value":"***Q1. As the heterogeneous graph is constructed directly from the interaction history, it should not contain more information than the history itself. Could the authors elaborate on why such a graph-based representation improves preference elicitation compared to directly learning from the raw interaction table?***\n\n**Author Response:**\n\nWe thank the reviewer for raising the question regarding the advantages of a graph-based representation. While the topological structure of the graph is indeed derived from the interaction history, **our graph-based approach provides superior preference elicitation because it explicitly models the relationships among key entities in the interactions.**\n\nGMTRouter extracts the key entities from the interaction history and carefully designs the graph structure to capture the rich relational patterns within the interactions. Through message passing, each node explicitly aggregates information from its related nodes, dynamically refining its representation. **The graph structure imposes a strong relational inductive bias, allowing the model to capture user preferences more effectively than treating interaction rows independently.**\n\nTo further ground this discussion, **we selected TIGER [5], a recent method that learns directly from raw interaction data, and RouteLLM [6] as new baselines**. The experimental results are shown below:\n\n**Table of TIGER & GMTRouter performance over datasets**\n\n| Model | Chatbot-Arena | MT-Bench | GSM8K | MMLU | LaMP |\n|--------|:--------:|:--------:|:--------:|:--------:|:--------:|\n|        | ACC / AUC | ACC / AUC | ACC / AUC | ACC / AUC | ACC / AUC |\n| RouteLLM   | 0.492 / 0.485 | 0.480 / 0.475 | 0.499 / 0.498 | 0.532 / 0.513 | - |\n| TIGER  | 0.739 / 0.735 | 0.656 / 0.691 | 0.639 / 0.683 | 0.710 / 0.764 | 0.339 / 0.698 |\n| GMTRouter  | 0.774 / 0.875 | 0.784 / 0.859 | 0.773 / 0.859 | 0.771 / 0.870 | 0.349 / 0.662 |\n\n*Note that RouteLLM is designed for routing between two LLMs, whereas the LaMP dataset requires routing among five LLMs; therefore, RouteLLM was not evaluated on LaMP.*\n\nMoreover, as reported in ***Table 2***, we have included several non-graph-based baselines (e.g FrugalGPT, Personalized LLM) as comparisons.\nThe results confirm that representing the interaction history as a heterogeneous graph enables GMTRouter to achieve more effective personalization compared to learning directly from raw data. These empirical results complement the conceptual argument: the graph structure imposes a relational inductive bias that  encourages the model to exploit dependencies across users, queries, responses, and LLMs, rather than treating interaction rows as independent.\n\n***Q2. The graph construction includes multiple users within the same graph (e.g., Figure 3(b)). However, user preferences are typically independent of one another. Would constructing individual user-specific graphs (and learning over them) be more intuitive? How does the shared multi-user graph influence personalization performance?***\n\n**Author Response:**\n\nWe thank the reviewer for this insightful question. As the reviewer correctly notes, user preferences are diverse, and signals from different users can interfere with one another. \nAlthough we conceptually construct a global interaction graph (Fig. 3(b)), **message passing is always performed on per-user subgraphs**:\n- During training ***[Line 248-251]***, each sampled subgraph only contains interactions from a single user.\n- During inference ***[Line 273-277]***, we also build a subgraph using only that user's history and candidate LLMs.\n\nThus, in practice, GMTRouter operates on **individual user-specific graphs**, and there is no cross-user message passing; the global graph is simply the union of these user-specific components for convenience of storage and indexing. Personalization arises from user-specific node representations together with parameter sharing across users.\nWe have clarified these details in the paper and will update ***Figure 3*** in future revisions to better illustrate the design. We, once again, thank the reviewer for the insightful comments to help us better polish our work. \n\n**[5]** Rajput, Shashank, et al. \"Recommender systems with generative retrieval.\" Advances in Neural Information Processing Systems 36 (2023): 10299-10315.\n\n**[6]** Ong, Isaac, et al. \"Routellm: Learning to route llms with preference data.\" arXiv preprint arXiv:2406.18665 (2024)."},"title":{"value":"Author response to Reviewer 8ann (3/3)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764046878381,"tcdate":1764046571670,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission14657/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission14657/Authors"],"forum":"oRe9tHpSpn","number":9,"license":"CC BY 4.0","cdate":1764046571670,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission14657/-/Official_Comment"],"mdate":1764046878381,"domain":"ICLR.cc/2026/Conference","replyto":"ryB9mqPWFS","id":"9vTMaudVNw","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Heterogeneous GNN","LLM routing","Preference Learning"]},"primary_area":{"value":"learning on graphs and other geometries & topologies"},"abstract":{"value":"Large Language Model (LLM) routing has demonstrated strong capability in balancing response quality with computational cost. As users exhibit diverse preferences, personalization has attracted increasing attention in LLM routing, since even identical queries may require different models to generate responses tailored to individual needs. However, existing approaches are not fully personalized and often fail to faithfully capture the complex interactions between specific users and LLMs. Moreover, user preference data is typically scarce, noisy, and inconsistent in format, which limits the effectiveness of methods that rely solely on user-specific data. To address these challenges, we propose GMTRouter, which represents multi-turn user–LLM interactions as a heterogeneous graph with four node types: user, LLM, query, and response, thereby maximally preserving the rich relational structure of the interaction. Through a tailored message-passing mechanism, GMTRouter learns to capture user preferences from few-shot data within a lightweight inductive graph learning framework, enabling effective personalization. Extensive experiments demonstrate that GMTRouter consistently outperforms the strongest baselines, achieving 0.9%–21.6% higher accuracy and 0.006–0.309 higher AUC across multiple datasets. More importantly, we further demonstrate that GMTRouter can adapt to new users and evolving preferences using only few-shot data, without extensive fine-tuning."},"_bibtex":{"value":"@misc{\nxie2026gmtrouter,\ntitle={{GMTR}outer: Personalized {LLM} Router over Multi-turn User Interactions},\nauthor={Encheng Xie and Yihang Sun and Tao Feng and Jiaxuan You},\nyear={2026},\nurl={https://openreview.net/forum?id=oRe9tHpSpn}\n}"},"title":{"value":"GMTRouter: Personalized LLM Router over Multi-turn User Interactions"},"pdf":{"value":"/pdf/b4f227b8ee59ab4f2a14d8fc028c0e06e9455c44.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"xie|gmtrouter_personalized_llm_router_over_multiturn_user_interactions"},"authorids":{"value":["~Encheng_Xie1","~Yihang_Sun2","~Tao_Feng5","~Jiaxuan_You2"]},"authors":{"value":["Encheng Xie","Yihang Sun","Tao Feng","Jiaxuan You"]}},"version":2},{"content":{"summary":{"value":"The paper introduces a novel visual SLAM framework that integrates multi-agent large language models (LLMs) to improve robustness under image degradation (e.g., blur, low light, rain) and enhance 3D reconstruction quality using 3D Gaussian Splatting (3DGS)."},"soundness":{"value":3},"confidence":{"value":5},"questions":{"value":"See the weaknesses."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"The work has broad potential impact on both SLAM and embodied AI communities. Rather than viewing SLAM as a fixed pipeline, this work positions it as an adaptive, reasoning system that perceives environmental conditions and self-modifies—a step toward truly autonomous robotic perception. By using LLMs not just for language but for orchestrating geometric processing, the paper offers a blueprint for hybrid neuro-symbolic systems in robotics. Image degradation (blur, low light, rain) is a real-world bottleneck in drone, automotive, and AR/VR applications. A system that autonomously detects and compensates for these issues without manual tuning is of clear industrial value. The modular, agent-based design invites extensions—e.g., adding failure recovery, incorporating inertial data, or integrating with world models. It also sets a precedent for using LLMs as controllers rather than just annotators in vision pipelines."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The paper attributes performance gains to the multi-agent orchestration framework, but it never isolates the impact of the agent structure versus the underlying components (e.g., InstructIR for restoration, RouteLLM for depth selection). For instance, Is the Orchestrator truly necessary, or could a monolithic LLM with function calling achieve the same result? Does the ReAct loop provide measurable benefits over static pipelines?\n\n2. The Assessor uses Co-Instruct for IQA, but the paper does not clarify how degradation types (blur, rain, etc.) are quantitatively identified. Is this a classification head? Zero-shot VLM prompting? If the Assessor relies on textual prompts like “Is this image blurry?”, then its reliability hinges on VLM robustness—yet no failure analysis or confidence calibration is provided. Table 2 reports counts of blur/low-light frames, but it’s unclear if these labels come from ground truth or the Assessor itself. If the latter, the evaluation may be circular: the system is evaluated on its own diagnostic output.\n\n3. The RouteLLM is trained on Replica (room2) and TUM-RGBD (freiburg2) and tested on other sequences from the same datasets. However, both datasets are indoor, synthetic or controlled, with limited visual diversity (e.g., no outdoor scenes, weather, dynamic objects). The paper claims robustness to “real-world scenarios” (Abstract), but provides no evaluation on truly unstructured environments (e.g., KITTI, TartanAir, or custom drone footage with motion blur + rain)."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762919238719,"tcdate":1761653702476,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7039/Reviewer_vj6u"],"signatures":["ICLR.cc/2026/Conference/Submission7039/Reviewer_vj6u"],"forum":"duIZUIRKJB","number":2,"license":"CC BY 4.0","cdate":1761653702476,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7039/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762919238719,"domain":"ICLR.cc/2026/Conference","replyto":"duIZUIRKJB","id":"Gm7EIk1q8B","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Deep SLAM; Multi-agent; Deep Estimation ;3DGS"]},"primary_area":{"value":"applications to robotics, autonomy, planning"},"abstract":{"value":"Visual Simultaneous Localization and Mapping (SLAM) reconstructs the metric structure of the physical world from sensor imagery, enabling precise robotic pose estimation. However, environmentally induced image degradation and varied image processing strategie significantly compromise localization accuracy. Intelligent SLAM systems address this challenge by autonomously perceiving dynamic perturbations and formulating adaptive processing strategies, further identifying and deploying optimal methodologies to achieve target localization objectives with enhanced metric precision. This paper introduces MAV-SLAM, a novel Multi-LLM-Agent-Orchestrated visual SLAM framework that proactively identifies and compensates for suboptimal image quality while autonomously selecting optimal depth estimation models. Specifically, we integrate a visual-language model that performs autonomous image restoration guided by image quality assessment, significantly enhancing SLAM localization performance. Furthermore, we implement a routing large language model for adaptive depth estimation, which consequently elevates the quality of 3D reconstruction via 3D Gaussian Splatting (3DGS). Rigorous evaluation across multiple benchmarks demonstrates that MAV-SLAM exhibits superior performance in both localization accuracy and 3DGS-based reconstruction fidelity, validating its effectiveness in real-world scenarios."},"_bibtex":{"value":"@misc{\nyao2026mavslam,\ntitle={{MAV}-{SLAM}: Multi-{LLM}-Agent Crew for Visual {SLAM} with 3D Gaussian Splatting},\nauthor={Jian Yao and Lisha Tang and Yang Li and Danping Zou},\nyear={2026},\nurl={https://openreview.net/forum?id=duIZUIRKJB}\n}"},"title":{"value":"MAV-SLAM: Multi-LLM-Agent Crew for Visual SLAM with 3D Gaussian Splatting"},"pdf":{"value":"/pdf/8fccc6924a5fddfae91e0c152e944d9a70c4b58f.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"yao|mavslam_multillmagent_crew_for_visual_slam_with_3d_gaussian_splatting"},"authorids":{"value":["~Jian_Yao9","~Lisha_Tang1","~Yang_Li26","~Danping_Zou1"]},"authors":{"value":["Jian Yao","Lisha Tang","Yang Li","Danping Zou"]}},"version":2},{"content":{"comment":{"value":"Thanks for your valuable feedback. We have uploaded the latest revised version.  \n\n**Q1.**\nWe agree that RouteLLM introduces a theoretically grounded utility function to make routing decisions between two candidate LLMs. However, in real-world applications, it is often necessary to handle routing among multiple candidate LLMs simultaneously. (e.g., AI coding tools that integrate several LLMs). Moreover, [1] and [2] point out that restricting routing methods to a limited pool of models is a limitation. RouteLLM is restricted to binary routing and cannot naturally extend to multi-LLM routing. To address this issue, we follow the GraphRouter and further explore a graph-based approach. PersonalizedRouter mainly focuses on multi-user and multi-LLM settings, leveraging user interaction information to capture latent user preferences for personalized routing. In contrast, RouteLLM treats all users identically in its routing process, restricting personalization in settings with multiple users. Nevertheless, we highly appreciate the reviewer’s insightful comments, and in the revised version we have added a discussion in the Limitation section about the utility function.\n\n**[1]** Feng, T., Shen, Y., & You, J. GraphRouter: A Graph-based Router for LLM Selections. In The Thirteenth International Conference on Learning Representations.\n\n**[2]** Tran, C., Paracha, S., Hafeez, A., & Chen, S. (2025). Arch-Router: Aligning LLM Routing with Human Preferences. arXiv preprint arXiv:2506.16655.\n\n\n**Q2.**\n**In the multi-cost-efficiency simulation strategy,** each weight pair represents one user. We set 1,000 different weight pairs to capture diverse user preferences over performance and cost. The details are presented in Appendix A.1. **In the LLM-as-a-Judge strategy,** the output generated under a given judge configuration for a user profile corresponds to one user. The details are provided in Section 5.2. We selected three different LLMs and applied two distinct instruction prompts to each model, resulting in six judge configurations. For each configuration, we simulated users based on 200 user profiles, yielding a total of 1,200 simulated user preferences across the six configurations. **In other words, each user profile is evaluated under six distinct judge configurations, effectively yielding six different user perspectives.** Ultimately, the combination of 200 user profiles with six judge configurations results in a total of 1,200 simulated user samples.\nWe used GPT-4o to generate 200 user profiles. Since presenting all 200 user profiles would be excessive, we only show 15 examples in Table 12 of the Appendix. In the latest revised version, we have supplemented Appendix A.4 with the method for generating the user profiles."},"title":{"value":"Response to Preference Modeling and User Simulation"}},"parentInvitations":"TMLR/-/Official_Comment","tmdate":1759471844164,"tcdate":1759431331038,"writers":["TMLR","TMLR/Paper5500/Authors"],"signatures":["TMLR/Paper5500/Authors"],"forum":"W80eE3ArAl","number":11,"license":"CC BY 4.0","cdate":1759431331038,"readers":["everyone"],"invitations":["TMLR/Paper5500/-/Official_Comment"],"mdate":1759471844164,"domain":"TMLR","replyto":"WfA7D74GQL","id":"K9Un6RIovr","forumContent":{"submission_length":{"value":"Regular submission (no more than 12 pages of main content)"},"venue":{"value":"Accepted by TMLR"},"code":{"value":"https://github.com/ulab-uiuc/PersonalizedRouter"},"supplementary_material":{"value":"/attachment/e4bbeecf2accf23bec60f721de3508b2cf488096.zip"},"abstract":{"value":"The growing number of Large Language Models (LLMs) with diverse capabilities and response styles provides users with a wider range of choices, which presents challenges in selecting appropriate LLMs, as user preferences vary in terms of performance, cost, and response style. Current LLM selection methods typically optimize for a single fixed objective, such as performance, cost, or a trade-off between them, and fail to learn individual user preferences from interaction data. To address these limitations, we propose PersonalizedRouter, a graph-based framework that models diverse user profiles and performs personalized LLM selection by leveraging interaction data that includes task context, queries, candidate LLMs, and user decisions. To capture contextual information between user queries and optimal LLMs, PersonalizedRouter converts the interaction data into a heterogeneous graph, where the relationships between different types of nodes are represented by edges. To evaluate adaptability across users, we design two strategies: the multi-cost-efficiency simulation strategy and the LLM-as-a-Judge strategy. In addition, we construct PersonaRoute-Bench, a large-scale benchmark with 1,000 simulated users and 10 LLMs. Experimental results show that PersonalizedRouter significantly outperforms existing LLM selection methods and surpasses the strongest methods by a large margin of 15.38% and 9.83% under two simulation strategies. On the PersonaRoute-Bench with 1,000 users, it further surpasses the best methods by 16.19% and 59.69% while maintaining higher efficiency. Moreover, PersonalizedRouter demonstrates strong few-shot generalization, achieving 64.81% and 85.80% of the fully trained model’s performance when adapting to new users and new LLMs."},"_bibtex":{"value":"@article{\ndai2025personalizedrouter,\ntitle={PersonalizedRouter: Personalized {LLM} Routing via Graph-based User Preference Modeling},\nauthor={Zhongjie Dai and Tao Feng and Jiaxuan You},\njournal={Transactions on Machine Learning Research},\nissn={2835-8856},\nyear={2025},\nurl={https://openreview.net/forum?id=W80eE3ArAl},\nnote={}\n}"},"title":{"value":"PersonalizedRouter: Personalized LLM Routing via Graph-based User Preference Modeling"},"pdf":{"value":"/pdf/1c7d7c4a7a37c3cd71672b102988c3788f67c8e7.pdf"},"venueid":{"value":"TMLR"},"paperhash":{"value":"dai|personalizedrouter_personalized_llm_routing_via_graphbased_user_preference_modeling"},"authorids":{"value":["~Zhongjie_Dai1","~Tao_Feng5","~Jiaxuan_You2"]},"assigned_action_editor":{"value":"~Yen-Chang_Hsu1"},"authors":{"value":["Zhongjie Dai","Tao Feng","Jiaxuan You"]}},"version":2},{"content":{"comment":{"value":"Thanks a lot for the review! We are glad to hear that the reviewer acknowledges our work addresses a timely challenge to orchestrate diverse models and tools, the RL formulation is sound and comprehensive that highlights not only outcome, but also efficiency and preference. We also thank the reviewer for appreciating the thorough analysis and concrete insights in Section 6. Below we address the concerns raised by the reviewer. We used purple text to clarify the parts that we modified in the revision.\n\n## Q1. Diverse baselines and more comparisons\nIn Table 1, we compared Orchestrator-8B to four types of baselines: (1). LLMs not using tools; (2). LLMs using basic tools, e.g., web search, code interpreter, etc.; (3). LLMs using both basic tools and models; (4). Existing results. We demonstrate that Orchestrator-8B boosts both system intelligence and efficiency compared to these baselines. In addition, we added more baseline results about orchestrator (RestGPT[1], ToolLlama[2], ToRL[3]) and router works (RouteLLM[4]). The detailed comparison is shown as below.\n\n|                                           | HLE  | FRAMES | τ$^2$-Bench | Cost | Latency |\n| ----------------------------------------- | ---- | ------ | ---------- | ---- | ------- |\n| RestGPT with basic tools              | 35.6 | 74.3   | 78.2       | 47.4 | 29.7    |\n| RestGPT with basic tools and models   | 21.6 | 57.7   | 62.8       | 27.2 | 18.9    |\n| ToolLLaMA-2-7b-v2 with basic tools            | 1.3  | 12.6   | 11.4       | 1.0  | 1.6     |\n| ToolLLaMA-2-7b-v2 with basic tools and models | 18.8 | 48.6   | 52.4       | 16.2 | 11.3    |\n| ToRL                                  | 0.9  | 14.8   | 29.8       | 0.2  | 0.5     |\n| RouteLLM                              | 7.2  | 28.3   | 24.9       | 13.4 | 8.7     |\n| Orchestrator-8B (ours)                    | 37.1 | 76.3   | 80.2       | 9.2  | 8.2     |\n\nWe use GPT-5 as the backbone LLM for RestGPT and leverage the default configuration for other baselines. For RestGPT and ToolLlama where the tool set can be flexibly defined, we run experiments with two settings: (1). use only basic tools; (2). used both basic tools and specialized/generalist LLMs, which are aligned with the configuration of Orchestrator-8B. As shown in the above table, Orchestrator-8B is better than all other baselines in both performance and efficiency. RestGPT relies on prompting LLMs in complex tool calling, which may not make the best use of specialized and generalist models. ToolLlama is limited to the reasoning and long-context understanding capability, which constraints the ability to address multi-step agentic tasks that would require large context windows. ToRL is trained to use only code interpreter and RouteLLM is trained to operate a fixed set of subordinate LLMs, which makes them less generalizable to broad tool-calling and model routing scenarios. In contrast, Orchestrator-8B is trained to operate arbitrary tool sets and targets general settings with demonstrably better intelligence and efficiency. We update Section 4, Section 5 and Table 1 in the paper to include more baselines for a more comprehensive comparison.\n\n[1]. https://arxiv.org/abs/2306.06624 \\\n[2]. https://arxiv.org/abs/2307.16789 \\\n[3]. https://arxiv.org/abs/2503.23383 \\\n[4]. https://arxiv.org/abs/2406.18665"},"title":{"value":"Thank you for the review"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763754370061,"tcdate":1763752217240,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission13689/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission13689/Authors"],"forum":"6vS1lNjPTa","number":1,"license":"CC BY 4.0","cdate":1763752217240,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission13689/-/Official_Comment"],"mdate":1763754370061,"domain":"ICLR.cc/2026/Conference","replyto":"SG8xor1sq5","id":"KzooU6ZlqJ","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["tool use; agentic AI; small language model;"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Large language models are powerful generalists, yet solving deep and complex problems such as those of the Humanity’s Last Exam (HLE) remains both conceptually challenging and computationally expensive. We show that small orchestrators managing other models and a variety of tools are able to both push the upper bound of intelligence and improve efficiency in solving difficult agentic tasks.\nWe introduce ToolOrchestra, a method for training small orchestrators that coordinate the use of intelligent tools. ToolOrchestra makes explicit use of reinforcement learning with outcome-, efficiency-, and user-preference-aware rewards. Using ToolOrchestra, we produce Orchestrator, an 8B model that achieves higher accuracy at lower cost than previous tool-use agents while aligning with user preferences on which tools are to be used for a given query. On HLE, Orchestrator achieves a score of 37.1\\%, outperforming GPT-5 (35.1\\%) while being 2.5x more efficient. On $\\tau^2$-Bench and FRAMES, Orchestrator surpasses GPT-5 by a wide margin while using only about 30\\% of the cost. Extensive analysis shows that Orchestrator achieves the best trade-off between performance and cost under multiple metrics, and generalizes robustly to previously unseen tools.  These results demonstrate that composing diverse tools with a lightweight orchestration model is both more efficient and more effective than existing methods, paving the way for practical and scalable tool-augmented reasoning systems. These results demonstrate that orchestrating diverse tools with lightweight agents is not only more efficient, but also more effective, paving the way for practical and scalable tool-augmented reasoning systems."},"_bibtex":{"value":"@misc{\nsu2026toolorchestra,\ntitle={ToolOrchestra: Elevating Intelligence via Efficient Model and Tool Orchestration},\nauthor={Hongjin SU and Shizhe Diao and Ximing Lu and Mingjie Liu and Jiacheng Xu and Xin Dong and Yonggan Fu and Peter Belcak and Hanrong Ye and Hongxu Yin and Yi Dong and Evelina Bakhturina and Tao Yu and Yejin Choi and Jan Kautz and Pavlo Molchanov},\nyear={2026},\nurl={https://openreview.net/forum?id=6vS1lNjPTa}\n}"},"title":{"value":"ToolOrchestra: Elevating Intelligence via Efficient Model and Tool Orchestration"},"pdf":{"value":"/pdf/a99d11745db82cbe0860c9c778e8d463f86f4578.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"su|toolorchestra_elevating_intelligence_via_efficient_model_and_tool_orchestration"},"authorids":{"value":["~Hongjin_SU1","~Shizhe_Diao2","~Ximing_Lu1","~Mingjie_Liu2","~Jiacheng_Xu2","~Xin_Dong1","~Yonggan_Fu1","~Peter_Belcak1","~Hanrong_Ye1","~Hongxu_Yin2","~Yi_Dong4","~Evelina_Bakhturina1","~Tao_Yu5","~Yejin_Choi1","~Jan_Kautz1","~Pavlo_Molchanov1"]},"authors":{"value":["Hongjin SU","Shizhe Diao","Ximing Lu","Mingjie Liu","Jiacheng Xu","Xin Dong","Yonggan Fu","Peter Belcak","Hanrong Ye","Hongxu Yin","Yi Dong","Evelina Bakhturina","Tao Yu","Yejin Choi","Jan Kautz","Pavlo Molchanov"]}},"version":2},{"content":{"comment":{"value":"Thank you very much for your time and constructive feedback. Below, we will try our best to address the questions raised, hoping to clarify your concerns.\n\n> ### Question #1 on Schema Linking\n\n1. Perfect schema linking is indeed challenging, its primary goal is to identify relevant tables and columns for the current task. As long as we maintain a high recall, some noise is tolerable. This is because missing a column will negatively impact SQL generation, whereas a redundant column can often be ignored during the SQL generation stage. This makes redundant noise more acceptable. In this regard, recall is prioritized. We examined the schema linking performance of GPT-4o-mini in our experiments and found that it achieves a high recall of 93% while reducing the number of columns by over 90% on average. This suggests that the schema linking in our experiments is sufficiently robust.\n2. Moreover, as schema linking is a standard procedure in text-to-SQL, we are motivated to utilize the results, which is why we position the routers after schema linking rather than before it. The database schema in the Bird dataset can be quite large, making it very difficult to route tasks solely based on the query and the full database schema. In contrast, since schema linking is a standard procedure, its intermediate results are valuable. Although these results may contain some noise from redundant elements, we still filter out over 90% of columns on average. This provides much more informative references for the routers compared to the full database schema. Therefore, we believe that placing routers right after schema linking is an appropriate design choice to utilize the linked schema.\n\n> ### Question #2 on Routing\n\nThank you for highlighting the relevant work of RouteLLM; we have cited it in the related work section. RouteLLM is an illuminating work on LLM routing and inspired some of our router implementations. \n\nRouteLLM proposes supervised fine-tuning (SFT) open-source LLMs to perform binary selection of the more suitable LLM using a special classification token. Our classification-based routers trained through SFT are similar to this approach. However, our method differs in that it supports multi-class classification by adding a linear classifier head. Furthermore, we introduced Direct Preference Optimization (DPO) to further help routers learn preferences, and these DPO-trained routers demonstrate better routing performance compared with SFT-trained ones. They are also the most effective routers in our studies. We believe that investigating and integrating more advanced routers could be an interesting topic for future research.\n\n\nThanks again for your insightful feedback. We hope our response helps clarify and address your concerns. If you have follow-up questions, please feel free to continue the discussion with us."}},"parentInvitations":"colmweb.org/COLM/2025/Conference/-/Official_Comment","tmdate":1756226360415,"tcdate":1748865803421,"writers":["colmweb.org/COLM/2025/Conference","colmweb.org/COLM/2025/Conference/Submission868/Authors"],"signatures":["colmweb.org/COLM/2025/Conference/Submission868/Authors"],"forum":"8OqGNXKwo8","number":1,"license":"CC BY 4.0","cdate":1748865803421,"readers":["everyone"],"invitations":["colmweb.org/COLM/2025/Conference/Submission868/-/Official_Comment","colmweb.org/COLM/2025/Conference/-/Edit"],"mdate":1756226360415,"domain":"colmweb.org/COLM/2025/Conference","replyto":"P4gmUNgbfU","id":"7iin6rasby","forumContent":{"TLDR":{"value":"We propose EllieSQL, a complexity-aware routing framework for Text-to-SQL, enhancing cost-efficiency while maintaining performance by directing queries to suitable SQL generation methods."},"venue":{"value":"COLM 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the COLM Code of Ethics on https://colmweb.org/CoE.html"},"keywords":{"value":["Text-to-SQL","Routing","Cost-Efficiency","Large Language Model"]},"supplementary_material":{"value":"/attachment/5d463428cf69a22e085ad7c79d3aa77603ca0c36.zip"},"abstract":{"value":"Text-to-SQL automatically translates natural language queries to SQL, allowing non-technical users to retrieve data from databases without specialized SQL knowledge. Despite the success of advanced LLM-based Text-to-SQL approaches on leaderboards, their unsustainable computational costs—often overlooked—stand as the \"elephant in the room\" in current leaderboard-driven research, limiting their economic practicability for real-world deployment and widespread adoption. To tackle this, we exploratively propose EllieSQL, a complexity-aware routing framework that assigns queries to suitable SQL generation pipelines based on estimated complexity. We investigate multiple routers to direct simple queries to efficient approaches while reserving computationally intensive methods for complex cases. Drawing from economics, we introduce the Token Elasticity of Performance (TEP) metric, capturing cost-efficiency by quantifying the responsiveness of performance gains relative to token investment in SQL generation. Experiments show that compared to always using the most advanced methods in our study, EllieSQL with the Qwen2.5-0.5B-DPO router reduces token use by over 40% without compromising performance on Bird development set, achieving more than a 2× boost in TEP over non-routing approaches. This not only advances the pursuit of cost-efficient Text-to-SQL but also invites the community to weigh resource efficiency alongside performance, contributing to progress in sustainable Text-to-SQL. Our source code and model are available at https://elliesql.github.io/."},"_bibtex":{"value":"@inproceedings{\nzhu2025elliesql,\ntitle={Ellie{SQL}: Cost-Efficient Text-to-{SQL} with Complexity-Aware Routing},\nauthor={Yizhang Zhu and Runzhi JIANG and Boyan Li and Nan Tang and Yuyu Luo},\nbooktitle={Second Conference on Language Modeling},\nyear={2025},\nurl={https://openreview.net/forum?id=8OqGNXKwo8}\n}"},"title":{"value":"EllieSQL: Cost-Efficient Text-to-SQL with Complexity-Aware Routing"},"pdf":{"value":"/pdf/22cb7760daa673d5ebdefb89c491dfcf425d7196.pdf"},"venueid":{"value":"colmweb.org/COLM/2025/Conference"},"paperhash":{"value":"zhu|elliesql_costefficient_texttosql_with_complexityaware_routing"},"authorids":{"value":["~Yizhang_Zhu1","~Runzhi_JIANG1","~Boyan_Li2","~Nan_Tang3","~Yuyu_Luo1"]},"author_guide":{"value":"I certify that this submission complies with the submission instructions as described on https://colmweb.org/AuthorGuide.html"},"authors":{"value":["Yizhang Zhu","Runzhi JIANG","Boyan Li","Nan Tang","Yuyu Luo"]}},"version":2},{"content":{"comment":{"value":"Dear Reviewer opSX,\n\nThank you for your constructive feedback. We also appreciate your initial assessment, particularly your comments highlighting \"**Well-Motivated Problem**\", \"**Comprehensive System Design**\", \"**Solid Experimental Results**\", and \"**Detailed Reproducibility Information**\".\n\nIn the rebuttal, we have provided detailed, point-by-point responses to all your comments and made revisions to the manuscript, mainly including:\n* **Stronger and fairer baseline.** We add RouteLLM as a directly comparable routing baseline and report results on both ID and OOD datasets. CogER achieves higher EM than RouteLLM on both ID and OOD benchmarks. \n* **Broader evaluation beyond training benchmarks.** We evaluate CogER on additional benchmarks, including MBPP (code generation), QuALITY (long-context QA), and Natural Questions (retrieval-augmented QA), to demonstrate generalization beyond the training distribution. \n* **Clearer definition and use of $L_{\\min}(S)$.** We explain in detail how $L_{\\min}(S)$ is obtained via deterministic runs of the four fixed levels (no human annotation or stochastic thresholds) and clarify that it is only used as a shaping signal in the hierarchical reward, not as a test-time oracle. \n* **Comprehensive routing diagnostics and ablations.** We add per-level precision/recall/F1, accuracy for queries routed to each level, a decomposition of errors into routing vs. execution, data-scale and router-size ablations, and a study of robustness under model upgrades (e.g., replacing back-end experts without retraining the router). \n* **Failure case analysis.** We provide representative failure cases and characterize the types of queries most prone to misrouting (e.g., borderline-complex and knowledge-hard questions), to make the limitations of CogER more explicit. \n\nWe are confident that these revisions address your concerns and greatly improve the clarity of our work.\n\nThank you again for your time and consideration.\n\nBest regards,\n\nThe Authors"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764336936486,"tcdate":1764336936486,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission705/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission705/Authors"],"forum":"z9HUMon2Br","number":24,"license":"CC BY 4.0","cdate":1764336936486,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission705/-/Official_Comment"],"mdate":1764336936486,"domain":"ICLR.cc/2026/Conference","replyto":"M93xVmPWSg","id":"Umasmdqf3y","forumContent":{"TLDR":{"value":"We propose the  Cognitive-Inspired Elastic Reasoning framework for efficient scaling of language model reasoning, which dynamically selects the most appropriate processing mode for each query."},"venue":{"value":"ICLR 2026 Conference Desk Rejected Submission"},"keywords":{"value":["Large language models; Reinforcement Learning; Markov decision process"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Large language models (LLMs) have demonstrated impressive performance across various language tasks. However, existing LLM reasoning strategies mainly rely on the LLM itself with fast or slow mode (like o1 thinking) and thus struggle to balance reasoning efficiency and accuracy across queries of varying difficulties. In this paper, we propose Cognitive-Inspired Elastic Reasoning (CogER), a framework inspired by human hierarchical reasoning that dynamically selects the most suitable reasoning strategy for each query. Specifically, CogER first assesses the complexity of incoming queries and assigns them to one of several predefined levels, each corresponding to a tailored processing strategy, thereby addressing the challenge of unobservable query difficulty. To achieve automatic strategy selection, we model the process as a Markov Decision Process and train a CogER-Agent using reinforcement learning. The agent is guided by a reward function that balances solution quality and computational cost, ensuring resource-efficient reasoning. Moreover, for queries requiring external tools, we introduce Cognitive Tool-Assisted Reasoning, which enables the LLM to autonomously invoke external tools within its chain-of-thought. Extensive experiments demonstrate that CogER outperforms state‑of‑the‑art Test‑Time scaling methods, achieving at least a 13% relative improvement in average exact match on In‑Domain tasks and an 8% relative gain on Out‑of‑Domain tasks."},"_bibtex":{"value":"@misc{\nanonymous2026beyond,\ntitle={Beyond Fast and Slow: Cognitive-Inspired Elastic Reasoning for Large Language Models},\nauthor={Anonymous},\nyear={2026},\nurl={https://openreview.net/forum?id=z9HUMon2Br}\n}"},"title":{"value":"Beyond Fast and Slow: Cognitive-Inspired Elastic Reasoning for Large Language Models"},"pdf":{"value":"/pdf/1252c0b740227fa4d6c59edd0735bd20fd07cbe8.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Desk_Rejected_Submission"},"paperhash":{"value":"hu|beyond_fast_and_slow_cognitiveinspired_elastic_reasoning_for_large_language_models"},"authorids":{"value":["~Jinwu_Hu1","~Dongjin_Yang1","~LangYu_Bian1","~Zhiquan_Wen1","~Yufeng_Wang5","~Yaofo_Chen1","~Bin_Xiao4","~Yuanqing_Li2","~Mingkui_Tan2"]},"authors":{"value":["Jinwu Hu","Dongjin Yang","LangYu Bian","Zhiquan Wen","Yufeng Wang","Yaofo Chen","Bin Xiao","Yuanqing Li","Mingkui Tan"]}},"version":2},{"content":{"summary":{"value":"This study introduces a novel framework for optimizing the selection of large language models (LLMs) based on user preferences, effectively balancing accuracy and cost. The research addresses the challenges associated with selecting appropriate LLMs, given their diverse capabilities and associated costs. By employing a multi-armed bandit approach with preference conditioning, the authors provide a flexible and efficient solution for optimizing LLM selection tailored to individual user needs. The results indicate significant improvements in both performance and cost management, positioning this framework as a promising avenue for future research in LLM utilization."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"1. The predictor baseline shows strong results when dealing with a family of LLMs in the experimental results. How should we interpret the differences between baselines in the \"Family of LLMs\" setting?\n2. The additional computational requirements for dynamically generating and processing user preferences may lead to increased latency and resource consumption. Could the authors provide an efficiency analysis regarding this?\n3. Data sensitivity is a critical issue. The effectiveness of dynamic routing heavily relies on accurately capturing user preferences. If users are unable to clearly articulate their needs or if their preferences change frequently, the system's performance may suffer due to misalignment between model selection and actual user requirements. How should datasets be prepared to address this?\n4. RouteLLM provides MT-Bench results to demonstrate its effectiveness. According to the results in this study, RouteLLM appears to perform poorly on selected benchmarks. Could the authors provide a comparison on MT-Bench for both the baselines and the proposed methods?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. The proposed methods effectively address the issue of routing among multiple models (more than two), allowing for scalability beyond the RouteLLM baselines.\n2. The authors present a pipeline consisting of model quizzing and preference-conditioned routing. For each component, they conduct optimization analysis and develop corresponding solution methods.\n3. The comprehensive results across multiple tasks and domains demonstrate that the routing mechanism provides superior performance."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The experimental results presented solely in figures are difficult to interpret. The authors could enhance clarity by including results in tables along with relevant metrics.\n2. The integration of a multi-armed bandit framework with dynamic routing based on user preferences introduces complexity to the overall system. This complexity may present challenges in terms of implementation, maintenance, and scalability, particularly for organizations with limited technical resources.\n3. There is a lack of detailed exploration of the complex components of the routing policy, such as the specific parameter selection for model quizzing and comparison methods."}},"nonreaders":[],"tmdate":1731427983955,"tcdate":1730755781094,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission10949/Reviewer_N8Q4"],"signatures":["ICLR.cc/2025/Conference/Submission10949/Reviewer_N8Q4"],"forum":"rEqETC88RY","number":4,"license":"CC BY 4.0","cdate":1730755781094,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission10949/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1731427983955,"domain":"ICLR.cc/2025/Conference","replyto":"rEqETC88RY","id":"EhCS0fnsh4","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["LLM Routing","Reinforcement Learning","Multi-objective Optimization"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The rapid advancement in large language models (LLMs) has brought forth a diverse range of models with varying capabilities that excel in different tasks and domains. However, selecting the optimal LLM for user queries often involves a challenging trade-off between accuracy and cost, a problem exacerbated by the diverse demands of individual queries. In this work, we present a novel framework that formulates the LLM selection process as a multi-armed bandit problem, enabling dynamic and intelligent routing of queries to the most appropriate model. Our approach incorporates a preference-conditioned dynamic routing mechanism, allowing users to specify their preferences at inference time, thereby offering a customizable balance between performance and cost. Additionally, our selection policy is designed to generalize to unseen LLMs, ensuring adaptability to new models as they emerge. Experimental results demonstrate that our method achieves significant improvements in both accuracy and cost-effectiveness across various LLM platforms, showcasing the potential of our framework to adaptively optimize LLM selection in real-world scenarios."},"_bibtex":{"value":"@misc{\nli2025llm,\ntitle={{LLM} Bandit: Cost-Efficient {LLM} Generation via Preference-Conditioned Dynamic Routing},\nauthor={Yang Li},\nyear={2025},\nurl={https://openreview.net/forum?id=rEqETC88RY}\n}"},"title":{"value":"LLM Bandit: Cost-Efficient LLM Generation via Preference-Conditioned Dynamic Routing"},"pdf":{"value":"/pdf/3965a78198d6f2123d2e62592cf27641212d9c5a.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"li|llm_bandit_costefficient_llm_generation_via_preferenceconditioned_dynamic_routing"},"authorids":{"value":["~Yang_Li19"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Yang Li"]}},"version":2},{"content":{"comment":{"value":"Dear Reviewer,\n\nThank you so much for your time and effort in reviewing our paper. Your initial comments are very helpful in refining our paper. We evaluate the schema linking performance in our experiments, which is robust. We also clarify the position of routers, as well as the connection and the difference between RouteLLM and our SFT-based routers. \n\n**Additionally, we conducted further experiments during the discussion period, which are summarized in the general response. We kindly invite you to refer to those results if you are interested in.**\n\nWe sincerely hope that these revisions have addressed your feedback. If you have any further questions, please do not hesitate to discuss them with us.\n\nBest regards,\n\nAuthors"}},"parentInvitations":"colmweb.org/COLM/2025/Conference/-/Official_Comment","tmdate":1756226359757,"tcdate":1749218510533,"writers":["colmweb.org/COLM/2025/Conference","colmweb.org/COLM/2025/Conference/Submission868/Authors"],"signatures":["colmweb.org/COLM/2025/Conference/Submission868/Authors"],"forum":"8OqGNXKwo8","number":17,"license":"CC BY 4.0","cdate":1749218510533,"readers":["everyone"],"invitations":["colmweb.org/COLM/2025/Conference/Submission868/-/Official_Comment","colmweb.org/COLM/2025/Conference/-/Edit"],"mdate":1756226359757,"domain":"colmweb.org/COLM/2025/Conference","replyto":"P4gmUNgbfU","id":"qvLmfMvMKm","forumContent":{"TLDR":{"value":"We propose EllieSQL, a complexity-aware routing framework for Text-to-SQL, enhancing cost-efficiency while maintaining performance by directing queries to suitable SQL generation methods."},"venue":{"value":"COLM 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the COLM Code of Ethics on https://colmweb.org/CoE.html"},"keywords":{"value":["Text-to-SQL","Routing","Cost-Efficiency","Large Language Model"]},"supplementary_material":{"value":"/attachment/5d463428cf69a22e085ad7c79d3aa77603ca0c36.zip"},"abstract":{"value":"Text-to-SQL automatically translates natural language queries to SQL, allowing non-technical users to retrieve data from databases without specialized SQL knowledge. Despite the success of advanced LLM-based Text-to-SQL approaches on leaderboards, their unsustainable computational costs—often overlooked—stand as the \"elephant in the room\" in current leaderboard-driven research, limiting their economic practicability for real-world deployment and widespread adoption. To tackle this, we exploratively propose EllieSQL, a complexity-aware routing framework that assigns queries to suitable SQL generation pipelines based on estimated complexity. We investigate multiple routers to direct simple queries to efficient approaches while reserving computationally intensive methods for complex cases. Drawing from economics, we introduce the Token Elasticity of Performance (TEP) metric, capturing cost-efficiency by quantifying the responsiveness of performance gains relative to token investment in SQL generation. Experiments show that compared to always using the most advanced methods in our study, EllieSQL with the Qwen2.5-0.5B-DPO router reduces token use by over 40% without compromising performance on Bird development set, achieving more than a 2× boost in TEP over non-routing approaches. This not only advances the pursuit of cost-efficient Text-to-SQL but also invites the community to weigh resource efficiency alongside performance, contributing to progress in sustainable Text-to-SQL. Our source code and model are available at https://elliesql.github.io/."},"_bibtex":{"value":"@inproceedings{\nzhu2025elliesql,\ntitle={Ellie{SQL}: Cost-Efficient Text-to-{SQL} with Complexity-Aware Routing},\nauthor={Yizhang Zhu and Runzhi JIANG and Boyan Li and Nan Tang and Yuyu Luo},\nbooktitle={Second Conference on Language Modeling},\nyear={2025},\nurl={https://openreview.net/forum?id=8OqGNXKwo8}\n}"},"title":{"value":"EllieSQL: Cost-Efficient Text-to-SQL with Complexity-Aware Routing"},"pdf":{"value":"/pdf/22cb7760daa673d5ebdefb89c491dfcf425d7196.pdf"},"venueid":{"value":"colmweb.org/COLM/2025/Conference"},"paperhash":{"value":"zhu|elliesql_costefficient_texttosql_with_complexityaware_routing"},"authorids":{"value":["~Yizhang_Zhu1","~Runzhi_JIANG1","~Boyan_Li2","~Nan_Tang3","~Yuyu_Luo1"]},"author_guide":{"value":"I certify that this submission complies with the submission instructions as described on https://colmweb.org/AuthorGuide.html"},"authors":{"value":["Yizhang Zhu","Runzhi JIANG","Boyan Li","Nan Tang","Yuyu Luo"]}},"version":2},{"content":{"summary":{"value":"This paper explores policy optimization as a way to route between small and large LLMs for inference cost savings. The authors frame token-level routing as an MDP w/ a policy that can be optimized. They evaluate this method on CSQA, ARC-Challenge, MMLU-PP, GSM-8k and MATH. The baselines are classifier/algorithm based token level and query level methods like Speculative Decoding, RouteLLM, Co-LLM. They also do some analysis on the possible reasons behind this method being better."},"ethics_flag":{"value":1},"reasons_to_reject":{"value":"N/A  - this paper should be published, though I think spending time on my suggestions for improving draft quality would greatly improve clarity."},"confidence":{"value":5},"reasons_to_accept":{"value":"- Routing a sequence of tokens between 2 possible models is an MDP, exploring a Policy Optimization for this instead of independently classifying these tokens makes sense to me and is well motivated. \n- The authors way of using ground truth to get preference labels for cheap makes sense (like teacher forcing).\n- Intuitions given for differences in quality+tradeoffs between different methods is good.\n\nNote on thoroughness: \n- I checked the math and it seems to be correct.\n- I skimmed the Related Work and it seems reasonable, though it's always possible to miss a few papers."},"rating":{"value":8},"questions_to_authors":{"value":"Questions: \n- Could you explain why choosing just SLM for CITER-S ablates on long-term influence of the trajectory chosen by the policy? Is this just hoping that LLM has better representation even if the token prediction is wrong?\n\nSuggestions to improve:\n- More thorough setup of background for readers without a RL background would be helpful. (Eg. Why *this* specific RL policy method? Most of this setup is clear from the DPO paper, but more setup is helpful along w/ citing DPO earlier in 2.1, though you do this in 2.2)\n- More description of how you get data for RouteLLM and Co-LLM would be helpful\n- More description of (data transformation) x-axis definition in Figure 2-5. Details in Appendix are good, but a 1-2 line high level definition in paper would be good.\n- Move Section 3.7 to Appendix if you need space for more clear descriptions. More extensive analysis on this could have been nice.\n\nTypos etc:\n- Line 199: \"In addition, its , Furthermore,\"\n- Compatibility Analysis: maybe just say \"Results on Different Model Families\" or something\n- Case Study Analysis on the Router -> Qualitative Analysis on the Router"}},"parentInvitations":"colmweb.org/COLM/2025/Conference/-/Official_Review","nonreaders":[],"tmdate":1756169405774,"tcdate":1747095920677,"writers":["colmweb.org/COLM/2025/Conference","colmweb.org/COLM/2025/Conference/Submission1200/Reviewer_ZgLP"],"signatures":["colmweb.org/COLM/2025/Conference/Submission1200/Reviewer_ZgLP"],"forum":"nqX9UYW9Af","number":3,"license":"CC BY 4.0","cdate":1747095920677,"readers":["everyone"],"invitations":["colmweb.org/COLM/2025/Conference/Submission1200/-/Official_Review","colmweb.org/COLM/2025/Conference/-/Edit"],"mdate":1756169405774,"domain":"colmweb.org/COLM/2025/Conference","replyto":"nqX9UYW9Af","id":"oZNcnrUyYf","forumContent":{"TLDR":{"value":"A novel Collaborative Inference with Token-lEvel Routing (CITER) framework that introduces a token-level routing mechanism, enabling efficient collaboration between small and large language models (SLMs & LLMs)."},"venue":{"value":"COLM 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the COLM Code of Ethics on https://colmweb.org/CoE.html"},"keywords":{"value":["collaborative inference","efficient inference","token-level routing","large language model"]},"abstract":{"value":"Large language models have achieved remarkable success in various tasks but suffer from high computational costs during inference, limiting their deployment in resource-constrained applications. To address this issue, we propose a novel Collaborative Inference with Token-lEvel Routing (CITER) framework that enables efficient collaboration between small and large language models (SLMs & LLMs) through a token-level routing strategy. Specifically, CITER routes non-critical tokens to an SLM for efficiency and routes critical tokens to an LLM for generalization quality. We formulate router training as a policy optimization, where the router receives rewards based on both the quality of predictions and the inference costs of generation. This allows the router to learn to predict token-level routing scores and make routing decisions based on both the current token and the future impact of its decisions. To further accelerate the reward evaluation process, we introduce a shortcut which significantly reduces the costs of the reward estimation and improving the practicality of our approach. Extensive experiments on five benchmark datasets demonstrate that CITER reduces the inference costs while preserving high-quality generation, offering a promising solution for real-time and resource-constrained applications."},"_bibtex":{"value":"@inproceedings{\nzheng2025citer,\ntitle={{CITER}: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routing},\nauthor={Wenhao Zheng and Yixiao Chen and Weitong Zhang and Souvik Kundu and Yun Li and Zhengzhong Liu and Eric P. Xing and Hongyi Wang and Huaxiu Yao},\nbooktitle={Second Conference on Language Modeling},\nyear={2025},\nurl={https://openreview.net/forum?id=nqX9UYW9Af}\n}"},"title":{"value":"CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routing"},"pdf":{"value":"/pdf/44adc116c02c1f30777bdd97bc65accd40cb593a.pdf"},"venueid":{"value":"colmweb.org/COLM/2025/Conference"},"paperhash":{"value":"zheng|citer_collaborative_inference_for_efficient_large_language_model_decoding_with_tokenlevel_routing"},"authorids":{"value":["~Wenhao_Zheng4","~Yixiao_Chen2","~Weitong_Zhang2","~Souvik_Kundu2","~Yun_Li7","~Zhengzhong_Liu1","~Eric_Xing1","~Hongyi_Wang1","~Huaxiu_Yao1"]},"author_guide":{"value":"I certify that this submission complies with the submission instructions as described on https://colmweb.org/AuthorGuide.html"},"authors":{"value":["Wenhao Zheng","Yixiao Chen","Weitong Zhang","Souvik Kundu","Yun Li","Zhengzhong Liu","Eric P. Xing","Hongyi Wang","Huaxiu Yao"]}},"version":2},{"content":{"summary":{"value":"This paper introduces CASCADIA, a novel cascade serving framework designed explicitly to schedule request routing and deploy model cascades for fast, quality-preserving LLM serving. CASCADIA employs a bi-level optimization method: at the deployment level, it uses a mixedinteger linear program to select resource allocations and parallelism strategies based on LLM information and workload characteristics; at the routing level, it applies a Chebyshev-guided method to iteratively co-optimize the routing strategy and the system deployment produced by the deployment level. \n\nIt uses mixed-integer linear programming (MILP) to determine the optimal deployment plan given a routing strategy. It balances response latency and output quality. Within each cascade stage, CASCADIA supports various parallelism strategies (e.g., tensor and pipeline parallelism), which allows it to automatically select the optimal strategy based on model size, incoming workload, and routing decisions."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"1. Maybe instead of using GPT-4, fine tune a BERT for cascading?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"Stengths:\n\n1. Efficient serving multiple LLM to balance accuracy and latency is an important topic.\n\n2. The proposed cascading method intuitively can help the multi-model serving system.\n\n3. Extensive experiments show the performance."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The main concern is on the real-time efficiency and cost. LLM serving is an online process. If using GPT-4 to judge the small model response, runs GPT-4 takes a few seconds and the cost is expensive. \n\n2. Time to first token is also very long. For simple prompt, it also needs to wait until GPT-4 finishes the judge.\n\n3. The baselines are insufficient. BERT-based router [1, 2, 3] that directly routes prompt to multiple LLMs can be compared.\n\n\n[1] https://github.com/vllm-project/semantic-router\n\n[2] Tensoropera router: A multi-model router for efficient llm inference\n\n[3] RouteLLM: Learning to Route LLMs with Preference Data"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762923421775,"tcdate":1762149764917,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission12568/Reviewer_11UR"],"signatures":["ICLR.cc/2026/Conference/Submission12568/Reviewer_11UR"],"forum":"wkrWbrHTJQ","number":3,"license":"CC BY 4.0","cdate":1762149764917,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission12568/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762923421775,"domain":"ICLR.cc/2026/Conference","replyto":"wkrWbrHTJQ","id":"32VpwIWmgt","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Distributed","Parallel","and Cluster Computing"]},"primary_area":{"value":"infrastructure, software libraries, hardware, systems, etc."},"abstract":{"value":"Recent advances in large language models (LLMs) have intensified the need to deliver both rapid responses and high-quality outputs. More powerful models yield better results but incur higher inference latency, whereas smaller models are faster yet less capable. Recent work proposes balancing this latency–quality trade-off using model cascades, which route simpler queries to smaller models and more complex ones to larger models. However, enabling efficient cascade serving remains challenging. Current frameworks lack effective mechanisms for handling (i) the huge and varying resource demands of different LLMs, (ii) the inherent heterogeneity of LLM workloads, and (iii) the co-optimization of system deployment and routing strategy.\n\nMotivated by these observations, we introduce Cascadia, a novel cascade serving framework designed explicitly to schedule request routing and deploy model cascades for fast, quality-preserving LLM serving. Cascadia employs a bi-level optimization method: at the deployment level, it uses a mixed-integer linear program to select resource allocations and parallelism strategies based on LLM information and workload characteristics; at the routing level, it applies a Chebyshev-guided method to iteratively co-optimize the routing strategy and the system deployment produced by the deployment level. Our extensive evaluation on diverse workload traces and different model cascades (DeepSeek and the Llama series) demonstrates that Cascadia significantly outperforms both single-model deployments and the state-of-the-art cascade serving baseline, achieving up to 4$\\times$ (2.3$\\times$ on average) tighter latency SLOs and up to 5$\\times$ (2.4$\\times$ on average) higher throughput while maintaining target answer quality."},"_bibtex":{"value":"@inproceedings{\njiang2026cascadia,\ntitle={Cascadia: An Efficient Cascade Serving System for Large Language Models},\nauthor={Youhe Jiang and Fangcheng Fu and Wanru Zhao and Stephan Rabanser and Jintao Zhang and Nicholas D. Lane and Binhang Yuan},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=wkrWbrHTJQ}\n}"},"title":{"value":"Cascadia: An Efficient Cascade Serving System for Large Language Models"},"pdf":{"value":"/pdf/a1ec00676c2c3a4e0e9b432a8676721778d8cd3d.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"jiang|cascadia_an_efficient_cascade_serving_system_for_large_language_models"},"authorids":{"value":["~Youhe_Jiang3","~Fangcheng_Fu1","~Wanru_Zhao1","~Stephan_Rabanser1","~Jintao_Zhang2","~Nicholas_D._Lane2","~Binhang_Yuan1"]},"authors":{"value":["Youhe Jiang","Fangcheng Fu","Wanru Zhao","Stephan Rabanser","Jintao Zhang","Nicholas D. Lane","Binhang Yuan"]}},"version":2},{"content":{"comment":{"value":"> *R4. The steering vector is static. The extracted steering vector is a single fixed vector applied uniformly at all reasoning step boundaries during inference. Recent work such as SADI (ICLR 2025) and the Activation State Machine (ASM, NeurIPS 2025) demonstrates that dynamic, input-adaptive steering significantly outperforms static vectors. RankGuide's steering should be compared to more recent competitive dynamic steering baselines (e.g., SADI [1]).*\n\nWe thank the reviewer for highlighting this direction. **Our contribution is the TT-rank–filtered calibration, which is orthogonal to the static-vs-dynamic axis.** The base steering paradigm in §4.2 follows SEAL [Chen et al., 2025]; our novelty is the use of TT-rank signals to select high-quality calibration samples before computing the steering vector (Eq. 7). The same filter can be applied to any downstream steering method that consumes a calibration set — including SADI, whose binary mask `M` is built by averaging activation differences over contrastive pairs. Removing low-rank (collapsed) pairs before computing `M` should yield a cleaner, more discriminative mask, so we expect **TT-rank filtering and dynamic steering to compose additively rather than compete**.\nIn the revision, we will add SADI and ASM to the related-work discussion as complementary dynamic-steering methods.\n\n> *R5. I feel like the paper can benefit from more coverage of the routing literature in related work: R2R [2], Stop Overthinking [3], RouteLLM [4], and HybridLLM [5].*\n\nWe thank the reviewer for these pointers. These methods operate at different routing granularities: **RouteLLM** [ICLR 2025] and **HybridLLM** [ICLR 2024] focus on *query-level* routing between strong and weak LLMs, **R2R** [NeurIPS 2025] focuses on *token-level* routing when reasoning paths diverge, and RankGuide focuses on *step-level* routing using cross-step hidden-state signals. We will add this discussion to §2.2 to sharpen RankGuide's positioning relative to the broader routing literature. **Stop Overthinking** [TMLR 2025] is a survey of efficient-reasoning methods covering CoT pruning and length compression, which we will cite in §3.3 as complementary motivation for our heavy-revalidation analysis."},"title":{"value":"Reply to Reviewer QKiy (part 2)"}},"parentInvitations":"colmweb.org/COLM/2026/Conference/-/Official_Comment","tmdate":1787537957260,"tcdate":1780312770271,"writers":["colmweb.org/COLM/2026/Conference","colmweb.org/COLM/2026/Conference/Submission3089/Authors"],"signatures":["colmweb.org/COLM/2026/Conference/Submission3089/Authors"],"forum":"uf90RrLRHA","number":8,"license":"CC BY 4.0","cdate":1780312770271,"readers":["everyone"],"invitations":["colmweb.org/COLM/2026/Conference/Submission3089/-/Official_Comment","colmweb.org/COLM/2026/Conference/-/Edit"],"mdate":1787537957260,"domain":"colmweb.org/COLM/2026/Conference","replyto":"sd5sPqibJB","id":"ZuDA0Pm5Qz","forumContent":{"TLDR":{"value":"We propose \\textbf{RankGuide}, a framework that improves the efficiency and effectiveness of SRM–LRM collaboration through tensor-rank-guided routing and steering."},"venue":{"value":"COLM 2026"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the COLM Code of Ethics on https://colmweb.org/CoE.html"},"LLM_usage":{"value":"In accordance with the Policy on the use of Large Language Models in the Call for Papers https://colmweb.org/cfp.html, I certify that my submission discloses any substantive use of LLMs in the research and paper-writing process."},"keywords":{"value":["Tensor Decomposition; Model Collaboration; Routing"]},"abstract":{"value":"Large reasoning models (LRMs) enhance problem-solving capabilities by generating explicit multi-step chains of thought (CoT) reasoning; however, they incur substantial inference latency and computational overhead.\nTo mitigate this issue, recent works have explored model collaboration paradigms, where small reasoning models (SRMs) generate intermediate reasoning steps to achieve a better accuracy–latency trade-off.\nDespite recent progress, effectively and efficiently detecting and mitigating SRM failures in collaborative systems remains a key challenge.\nTo address this issue, we analyze SRM inference in both the generated text and hidden-state spaces, and identify three types of failure modes: \\textit{overconfidence}, \\textit{uncertainty}, and \\textit{heavy revalidation}.\nBuilding on these insights, we propose \\textbf{RankGuide}, a framework that improves the efficiency and effectiveness of SRM–LRM collaboration through tensor-rank-guided routing and steering. \nSpecifically, RankGuide leverages a routing signal that incorporates \\textit{tensor-rank signals} derived from consecutive hidden states to detect when SRMs are likely to fail and selectively invoke LRMs. \nIn addition, we introduce a tensor-rank-filtered steering vector extraction method to modulate the reasoning trajectory of SRMs, thereby improving their generation quality.\nBy improving both routing and steering through tensor-rank signals, RankGuide enables SRM–LRM collaborative systems to achieve more efficient reasoning with fewer steps and improved accuracy.\nExperiments \\rev{across three reasoning domains -- mathematics, code generation, and scientific QA --} demonstrate the efficacy of RankGuide in reducing latency by up to $1.75\\times$ compared to LRM, while maintaining competitive accuracy relative to prior methods."},"_bibtex":{"value":"@inproceedings{\ntian2026rankguide,\ntitle={RankGuide: Tensor-Rank-Guided Routing and Steering for Efficient Reasoning},\nauthor={Jiayi Tian and Yupeng Su and Ryan Solgi and Souvik Kundu and Zheng Zhang},\nbooktitle={Third Conference on Language Modeling},\nyear={2026},\nurl={https://openreview.net/forum?id=uf90RrLRHA}\n}"},"title":{"value":"RankGuide: Tensor-Rank-Guided Routing and Steering for Efficient Reasoning"},"pdf":{"value":"/pdf/83b6af112def89a7b6d1ea58a6e6b7bc8b157cf7.pdf"},"venueid":{"value":"colmweb.org/COLM/2026/Conference"},"paperhash":{"value":"tian|rankguide_tensorrankguided_routing_and_steering_for_efficient_reasoning"},"authorids":{"value":["~Jiayi_Tian2","~Yupeng_Su2","~Ryan_Solgi1","~Souvik_Kundu2","~Zheng_Zhang2"]},"author_guide":{"value":"I certify that this submission complies with the submission instructions as described on https://colmweb.org/AuthorGuide.html"},"authors":{"value":["Jiayi Tian","Yupeng Su","Ryan Solgi","Souvik Kundu","Zheng Zhang"]}},"version":2},{"content":{"comment":{"value":"> [3] (HybridLLM) uses DeBERTa for score prediction, [4] (RouteLLM) uses text-embedding-3-small for its matrix factorization router and the similarity-weighted ranking model and also explores BERT and Llama 3 8B based classifiers. During test time, all of these models predict performance of LLMs without executing the query on them.\n\nAforementioned works **failed to generalize** their models beyond two LLMs (small and large), while our Routoo model is applicable across all LLMs in Huggingface's OpenLLM benchmark (over 1,000 models). These prior approaches relied on handpicked models, limiting generalizability in real-world scenarios with many LLMs. In contrast, Routoo uniquely and effectively handles diverse LLMs, making it the first routing model to **outperform GPT-4** (Figure 2) while reducing inference costs by 50%. **None of the referenced works achieve this**.  \nMoreover, the **RouteLLM paper is not peer-reviewed** and is currently under submission to the same conference ([OpenReview link](https://openreview.net/forum?id=8sSqNntaMr)). Penalizing us for not citing it is **inconsistent with ICLR review guidelines**, which state submissions should not be penalized for missing non-peer-reviewed references ([link](https://iclr.cc/Conferences/2023/ReviewerGuide)). Additionally, Table 1 of RouteLLM shows their LLM-based router fails to surpass other baselines.   \nOur main contribution (lines 99–130) is the Routoo model, which integrates a Performance Predictor, Cost-Aware Selector, and Universe Constructor. While prior works addressed some of these components, they neither generalized to many LLMs (56 in our case) nor surpassed expensive models like GPT-4. Routoo not only reduces costs by 50% (Routoo (mix)) but also surpasses GPT-4’s performance by leveraging 56 LLMs (Figure 2, right most green point).\n\n> Why would an embedding similarity between a question text to question+answer text be predictive of the correctness of the answer? The embedding model is not trained to do so nor does it have the reasoning capability. A better approach would be: for the query during test time, find the most similar query from training time, and use the LLM that can correctly answer  with the lowest cost. To smoothen the noise, you can expand to topN retrieved queries and topK lowest budget LLMs that answer each query and pick the most frequently occurring LLM from that list.\n\nFor the encoding of $q_{train}$, as in the MMLU benchmark, **answers are single characters** (see [dataset](https://huggingface.co/datasets/cais/mmlu/viewer/all)), the difference between encoding the question with the answer or just the question is minimal. We chose cosine similarity because the sizes of the underlying models range from 7B to 34B, with many smaller models (e.g., 7B) having similar sizes. Assigning identical scores (“1” for all top-10) would prevent our cost-aware selector from functioning effectively, as all models would appear equally suitable, requiring random selection. Retaining cosine similarity scores provides more granular information for our cost-aware selector.  \nLastly, our KNN baseline performs impressively, achieving competitive results with models like LLaMA2 13B, Mistral 7B, and LLaMA2 7B, and even showing competitiveness with LLaMA2 70B. This confirms it as a robust baseline for comparison (Table 1).\n\nWe kindly request you to review our responses and consider them in your evaluation score."},"title":{"value":"Author Response to Reviewer o5Db"}},"tmdate":1732744943718,"tcdate":1732744943718,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission11514/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission11514/Authors"],"forum":"RQ9fQLEajC","number":11,"license":"CC BY 4.0","cdate":1732744943718,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission11514/-/Official_Comment"],"mdate":1732744943718,"domain":"ICLR.cc/2025/Conference","replyto":"0Ed07Wf6qH","id":"SXsHXiLt6Q","forumContent":{"TLDR":{"value":"An LLM-based Router trained to route to the most effective LLM optimizing performance, cost, and efficiency, creating a new state-of-the-art model by integrating knowledge of underlying models."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["LLM","efficiency","knowledge integration","generation"]},"supplementary_material":{"value":"/attachment/2e84b1f1c381960311d11bdd23f4d081c4e4d14a.zip"},"primary_area":{"value":"generative models"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"LLMs with superior response quality—particularly larger or closed-source models—often come with higher inference costs, making their deployment inefficient and costly. Meanwhile, developing foundational LLMs from scratch is becoming increasingly resource-intensive and impractical for many applications. To address the challenge of balancing quality and cost, we introduce Routoo, an architecture designed to optimize the selection of LLMs for specific prompts based on performance, cost, and efficiency. Routoo provides controllability over the trade-off between inference cost and quality, enabling significant reductions in inference costs for a given quality requirement.\nRoutoo comprises two key components: a performance predictor and cost-aware selector. The performance predictor is a lightweight LLM that estimates the expected performance of various underlying LLMs on a given prompt without executing them. The cost-aware selector module then selects the most suitable model based on these predictions and constraints such as cost and latency, significantly reducing inference costs for the same quality. \nWe evaluated Routoo using the MMLU benchmark across 57 domains employing open-source models. Our results show that Routoo matches the performance of the Mixtral 8x7b model while reducing inference costs by one-third. Additionally, by allowing increased costs, Routoo surpasses Mixtral's accuracy by over 5\\% at equivalent costs, achieving an accuracy of 75.9\\%. When integrating GPT4 into our model pool, Routoo nearly matches GPT4's performance at half the cost and exceeds it with a 25\\% cost reduction. \nThese outcomes highlight Routoo's potential to significantly reduce inference costs without compromising quality, and even to establish new state-of-the-art results by leveraging the collective capabilities of multiple LLMs."},"_bibtex":{"value":"@misc{\nmohammadshahi2025routoo,\ntitle={Routoo: Learning to Route to Large Language Models Effectively},\nauthor={Alireza Mohammadshahi and Arshad Rafiq Shaikh and Majid Yazdani},\nyear={2025},\nurl={https://openreview.net/forum?id=RQ9fQLEajC}\n}"},"title":{"value":"Routoo: Learning to Route to Large Language Models Effectively"},"pdf":{"value":"/pdf/454cdce2a12d959ded0bc50a6c5b05c494a2d4dd.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"mohammadshahi|routoo_learning_to_route_to_large_language_models_effectively"},"authorids":{"value":["~Alireza_Mohammadshahi1","~Arshad_Rafiq_Shaikh1","~Majid_Yazdani1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Alireza Mohammadshahi","Arshad Rafiq Shaikh","Majid Yazdani"]}},"version":2},{"content":{"summary":{"value":"This paper introduces Agora, a novel market-based framework for coordinating heterogeneous Vision-Language Models (VLMs) in multi-agent systems. The key idea is to treat epistemic uncertainty as a tradable asset across perceptual, semantic, and inferential dimensions. Agents trade uncertainty based on a profitability-driven protocol, guided by a market-aware Thompson Sampling broker, to minimize system-wide cost while maintaining task performance. The authors provide both theoretical and empirical evidence that Agora outperforms strong baselines (e.g., Mixture-of-Agents, KABB) in terms of accuracy (+8.5% on MMMU) and cost-efficiency (3× reduction)."},"soundness":{"value":4},"confidence":{"value":2},"questions":{"value":"- How would Agora perform in the presence of adversarial or strategic agents that intentionally misreport uncertainty or cost to exploit the market? For instance, how does the framework handle strategic manipulation or collusion that could distort trading dynamics or market equilibrium?\n- Could you clarify the computational complexity of the broker and trading protocol, particularly their overhead within a more realistic analysis?\n- Would Agora still work if agents have overlapping expertise or non-stationary cost profiles?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":3},"strengths":{"value":"- The core concept of a decentralized market for uncertainty is a powerful and novel metaphor that drives the entire paper.\n- The work tackles a foundational challenge (economic scalability) with a comprehensive solution that includes theoretical formulation, algorithm design, and empirical validation.\n- The framework demonstrates consistent and sometimes substantial gains in accuracy and cost-efficiency across multiple benchmarks."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"-  The framework introduces significant complexity with its multi-dimensional uncertainty quantification, trading ledger, and sophisticated broker. While the paper claims scalability, the experiments are limited to ≤15 agents. It’s unclear how the market mechanism would behave with hundreds of agents or high-frequency trading.\n- The paper overlooks key transaction costs (coordination latency, broker computation, and communication overhead), which are excluded from its cost metrics. A system that is 3x cheaper in API calls but 10x slower due to coordination may not be viable for real-time applications.\n-  While numerous baselines are included, the adaptation of language-model-focused routers (like FrugalGPT, RouteLLM) to the multi-modal, multi-agent setting is only briefly mentioned in the appendix."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762915553066,"tcdate":1762281948121,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission582/Reviewer_oqqC"],"signatures":["ICLR.cc/2026/Conference/Submission582/Reviewer_oqqC"],"forum":"zeqCjGQB4U","number":4,"license":"CC BY 4.0","cdate":1762281948121,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission582/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762915553066,"domain":"ICLR.cc/2026/Conference","replyto":"zeqCjGQB4U","id":"aj5Eea9564","forumContent":{"TLDR":{"value":"We presents a framework that reframes coordination as a decentralized market for uncertainty."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["agent; Vision Language Model; Uncernity"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Vision-Language Models (VLMs) enable powerful multi-agent systems, but scaling them is economically unsustainable: coordinating heterogeneous agents under information asymmetry often spirals costs. Existing paradigms, such as Mixture-of-Agents and knowledge-based routers, rely on heuristic proxies that ignore costs and collapse uncertainty structure, leading to provably suboptimal coordination.\nWe introduce Agora, a framework that reframes coordination as a decentralized market for uncertainty. Agora formalizes epistemic uncertainty into a structured, tradable asset (perceptual, semantic, inferential), and enforces profitability-driven trading among agents based on rational economic rules. A market-aware broker, extending Thompson Sampling, initiates collaboration and guides the system toward cost-efficient equilibria. Experiments on five multimodal benchmarks (MMMU, MMBench, MathVision, InfoVQA, CC-OCR) show that Agora outperforms strong VLMs and heuristic multi-agent strategies, e.g., achieving +8.5% accuracy over the best baseline on MMMU while reducing cost by over 3×. These results establish market-based coordination as a principled and scalable paradigm for building economically viable multi-agent visual intelligence systems."},"_bibtex":{"value":"@inproceedings{\nzhang2026why,\ntitle={Why Keep Your Doubts to Yourself? Trading Visual Uncertainties among Vision-Language Models},\nauthor={Jusheng Zhang and Yijia Fan and Kaitong Cai and Jing Yang and Jiawei Yao and Jian Wang and Guanlong Qu and Ziliang Chen and Keze Wang},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=zeqCjGQB4U}\n}"},"title":{"value":"Why Keep Your Doubts to Yourself? Trading Visual Uncertainties among Vision-Language Models"},"pdf":{"value":"/pdf/03f4bead771e441ade14013c814de209cd0630ea.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"zhang|why_keep_your_doubts_to_yourself_trading_visual_uncertainties_among_visionlanguage_models"},"authorids":{"value":["~Jusheng_Zhang3","~Yijia_Fan1","~Kaitong_Cai1","~Jing_Yang39","~Jiawei_Yao3","~Jian_Wang10","~Guanlong_Qu2","~Ziliang_Chen1","~Keze_Wang1"]},"authors":{"value":["Jusheng Zhang","Yijia Fan","Kaitong Cai","Jing Yang","Jiawei Yao","Jian Wang","Guanlong Qu","Ziliang Chen","Keze Wang"]}},"version":2},{"content":{"comment":{"value":"### **W3. Evaluation is restricted exclusively to mathematical reasoning tasks. The paper completely ignores other important domains like general question answering, commonsense reasoning, coding, summarization, or multi-turn dialogue. This severely limits claims about generalizability, as mathematical problems may have unique characteristics that don't transfer to other collaborative inference scenarios where routing decisions differ fundamentally.**\n\nWe agree with the reviewer that designing reward functions for open-ended or subjective tasks is inherently challenging. However, this challenge is **not specific to our method**, it is a well-recognized limitation across the entire reinforcement-learning-with-verified-rewards (RLVR) family of approaches. Existing RLVR-based post-training studies (e.g., GRPO and GRPO variants) are almost exclusively evaluated on tasks with objective correctness signals, such as mathematics reasoning. Reliable reward design for open-ended tasks remains an open research problem. \n\nIn this context, our work follows the same established practice. We intentionally focus on mathematical and arithmetic reasoning tasks because they provide **deterministic, verifiable reward signals** essential for evaluating the effectiveness of a new collaborative training paradigm. Our contribution is to make the **first attempt** at an end-to-end optimization framework for on-device–cloud collaborative inference, with a particular focus on enabling **collaborative reasoning**.\n\nWe would like to clarify that **we do not claim that our methodology currently applies to all categories of tasks**. Extending collaborative device–cloud RL training to more complex or subjective tasks would require **reward-model-based evaluation** or **LLM-as-a-Judge mechanisms**, which is an important direction for future work.\n\n> The limitation noted by the reviewer is **a limitation of the RLVR regime**, not of our framework specifically. Our method inherits the same scope as other RLVR methods and can be applied to the tasks where deterministic reward signals exist. To clarify this, we revised the manuscript and added a discussion on the potential limitation of our work.\n\n### **W4. Only one routing method, notable omissions include FrugalGPT, RouteLLM variants, context-aware cascading policies, and ensemble-based routing methods**\n\nWe would like to clarify that some works mentioned by the reviewer are **not applicable** to the device–cloud LLM collaboration setting studied in this paper, and some of them are actually **similar** as the baseline we considered in our original manuscript.\n\n**FrugalGPT [R1]** also adopt a router to select LLMs. However, it select multiple LLMs for one query and sequentially ask these LLMs to generate response until a target quality is reached. This is fundamentally different from our device–cloud collaboration setup.\n\n**Context-aware cascading policy [R2]** extend FrugalGPT by adding contextual features but still operate in the same **multi-LLM cascade** regime, where several LLMs may be queried sequentially for the same prompt, making it not an appropriate baseline.\n\n**Ensemble-based routing method [R3]** also assumes access to several diverse LLMs and predicts a routing probability distribution over them. This amounts to a **multiclass classification problem across**, which is not aligned with our binary device–cloud setting. Conceptually, this is similar to the baseline we already included (Task-Tuning&Router).\n\n**RouteLLM [R4]** take BERT-base and LLaMA3-8B as the classifier to route the query between a small model and powerful model, which is essentially similar to the baseline we already included (Task-Tuning&Router).\n\n[R1] \"Frugalgpt: How to use large language models while reducing cost and improving performance.\" *2023*.\n\n[R2] \"Efficient contextual llm cascades through budget-constrained policy learning.\" *2024*.\n\n[R3] \"Routing to the expert: Efficient reward-guided ensemble of large language models.\" *2024*.\n\n[R4] \"Routellm: Learning to route llms with preference data.\" *2024*."},"title":{"value":"Responses (2/4)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763965340795,"tcdate":1763941829110,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission6850/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission6850/Authors"],"forum":"SwHnok3JtD","number":5,"license":"CC BY 4.0","cdate":1763941829110,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission6850/-/Official_Comment"],"mdate":1763965340795,"domain":"ICLR.cc/2026/Conference","replyto":"7ReKF9CHhr","id":"pZQHJAt9cq","forumContent":{"TLDR":{"value":"We propose a unified post-training framework that integrates routing optimization, enabling the on-device LLM to improve its problem-solving ability while learning routing strategies."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Large Language Models","Device-Cloud Collaboration","Post-Training","Routing Optimization","Reinforcement Learning"]},"supplementary_material":{"value":"/attachment/0abec1d957d5dfe99e2938019c58c05fb162a7f5.zip"},"primary_area":{"value":"other topics in machine learning (i.e., none of the above)"},"abstract":{"value":"Device-cloud collaboration has emerged as a promising paradigm for deploying large language models (LLMs), combining the efficiency of lightweight on-device inference with the superior performance of powerful cloud LLMs. An essential problem in this scenario lies in deciding whether a given query is best handled locally or delegated to the cloud. Existing approaches typically rely on external routers, implemented as binary classifiers, which often struggle to determine task difficulty from the prompt's surface pattern. To address these limitations, we propose a framework where the on-device LLM makes routing decisions at the end of its solving process, with this capability instilled through post-training. In particular, we formulate a reward maximization problem with carefully designed rewards that encourage effective problem solving and judicious offloading to the cloud. To solve this problem, we develop a group-adaptive policy gradient algorithm, featuring a group-level policy gradient, designed to yield an unbiased gradient estimator of the reward, and adaptive prompt filtering, developed to provide complementary learning signals. Extensive experiments across models and benchmarks show that the proposed methodology consistently outperforms existing baselines and significantly narrows the gap to full cloud LLM performance."},"_bibtex":{"value":"@misc{\nfang2026collaborative,\ntitle={Collaborative Device-Cloud {LLM} Inference through Reinforcement Learning},\nauthor={Wenzhi Fang and Dong-Jun Han and Liangqi Yuan and Christopher Brinton},\nyear={2026},\nurl={https://openreview.net/forum?id=SwHnok3JtD}\n}"},"title":{"value":"Collaborative Device-Cloud LLM Inference through Reinforcement Learning"},"pdf":{"value":"/pdf/b0a3698c6181d5c8e4245b48f5ea8700da6b0ff7.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"fang|collaborative_devicecloud_llm_inference_through_reinforcement_learning"},"authorids":{"value":["~Wenzhi_Fang1","~Dong-Jun_Han1","~Liangqi_Yuan1","~Christopher_Brinton1"]},"authors":{"value":["Wenzhi Fang","Dong-Jun Han","Liangqi Yuan","Christopher Brinton"]}},"version":2},{"content":{"comment":{"value":"## Q2. Baseline comparisons\nIn Table 1, we compared Orchestrator-8B to four types of baselines: (1). LLMs not using tools; (2). LLMs using basic tools, e.g., web search, code interpreter, etc.; (3). LLMs using both basic tools and models; (4). Existing results. We demonstrate that Orchestrator-8B boosts both system intelligence and efficiency compared to these baselines. In addition, we added more baseline results about orchestrator (RestGPT[1], ToolLlama[2], ToRL[3]) and router works (RouteLLM[4]). The detailed comparison is shown as below.\n\n|                                           | HLE  | FRAMES | τ$^2$-Bench | Cost | Latency |\n| ----------------------------------------- | ---- | ------ | ---------- | ---- | ------- |\n| RestGPT with basic tools              | 35.6 | 74.3   | 78.2       | 47.4 | 29.7    |\n| RestGPT with basic tools and models   | 21.6 | 57.7   | 62.8       | 27.2 | 18.9    |\n| ToolLLaMA-2-7b-v2 with basic tools            | 1.3  | 12.6   | 11.4       | 1.0  | 1.6     |\n| ToolLLaMA-2-7b-v2 with basic tools and models | 18.8 | 48.6   | 52.4       | 16.2 | 11.3    |\n| ToRL                                  | 0.9  | 14.8   | 29.8       | 0.2  | 0.5     |\n| RouteLLM                              | 7.2  | 28.3   | 24.9       | 13.4 | 8.7     |\n| Orchestrator-8B (ours)                    | 37.1 | 76.3   | 80.2       | 9.2  | 8.2     |\n\nWe use GPT-5 as the backbone LLM for RestGPT and leverage the default configuration for other baselines. For RestGPT and ToolLlama where the tool set can be flexibly defined, we run experiments with two settings: (1). use only basic tools; (2). used both basic tools and specialized/generalist LLMs, which are aligned with the configuration of Orchestrator-8B. As shown in the above table, Orchestrator-8B is better than all other baselines in both performance and efficiency. RestGPT relies on prompting LLMs in complex tool calling, which may not make the best use of specialized and generalist models. ToolLlama is limited to the reasoning and long-context understanding capability, which constraints the ability to address multi-step agentic tasks that would require large context windows. ToRL is trained to use only code interpreter and RouteLLM is trained to operate a fixed set of subordinate LLMs, which makes them less generalizable to broad tool-calling and model routing scenarios. In contrast, Orchestrator-8B is trained to operate arbitrary tool sets and targets general settings with demonstrably better intelligence and efficiency. We update Section 4, Section 5 and Table 1 in the paper to include more baselines for a more comprehensive comparison.\n\n[1]. https://arxiv.org/abs/2306.06624 \\\n[2]. https://arxiv.org/abs/2307.16789 \\\n[3]. https://arxiv.org/abs/2503.23383 \\\n[4]. https://arxiv.org/abs/2406.18665 \n\n## Q3. Controllability evaluation\nIn Section 6.4, we calculate the preference rewards to comprehensively evaluate the capabilities of Orchestrator-8B to align with the user preferences (web search or local search, third-party APIs or local host models, etc.). We further conducted more fine-grained ablation studies to understand the agent's behavior when different user preferences are given at test-time inference.  Specifically, we provide Orchestrator-8B with different preferences to trade off between performance and cost in the instructions, and evaluate both metrics across the HLE, FRAMES, and τ$^2$-bench.\n\n| Preference  instruction                                         | HLE  | FRAMES | τ$^2$-bench | cost |\n| ----------------------------------------------------------------------------- | ---- | ------ | ---------- | ---- |\n| Exclusively optimize the performance without considering the cost            | 39.2 | 79.8   | 82.4       | 25.7 |\n| Top-tier performance, but occasionally reducing the cost when possible                     | 37.7 | 77.5   | 80.8       | 16.4 |\n| Good performance, but need to balance the cost and reduce the usage of expensive models and tools | 37.1 | 76.3   | 80.2       | 9.2  |\n| Reasonable performance under limited budgets                                  | 30.6 | 71.5   | 74.9       | 4.6  |\n\nThe results show that Orchestrator-8B demonstrates strong alignment capabilities to follow user instructions when considering the trade-off between the performance and the cost. The change in the user instructions effectively alters the behavior of Orchestrator-8B to finish tasks during test-time inference. We add this controllability evaluation to the paper in Appendix P."},"title":{"value":"Thank you for the review"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763754429694,"tcdate":1763752803924,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission13689/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission13689/Authors"],"forum":"6vS1lNjPTa","number":5,"license":"CC BY 4.0","cdate":1763752803924,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission13689/-/Official_Comment"],"mdate":1763754429694,"domain":"ICLR.cc/2026/Conference","replyto":"qFuRrThldg","id":"fMsfIAQoJS","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["tool use; agentic AI; small language model;"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Large language models are powerful generalists, yet solving deep and complex problems such as those of the Humanity’s Last Exam (HLE) remains both conceptually challenging and computationally expensive. We show that small orchestrators managing other models and a variety of tools are able to both push the upper bound of intelligence and improve efficiency in solving difficult agentic tasks.\nWe introduce ToolOrchestra, a method for training small orchestrators that coordinate the use of intelligent tools. ToolOrchestra makes explicit use of reinforcement learning with outcome-, efficiency-, and user-preference-aware rewards. Using ToolOrchestra, we produce Orchestrator, an 8B model that achieves higher accuracy at lower cost than previous tool-use agents while aligning with user preferences on which tools are to be used for a given query. On HLE, Orchestrator achieves a score of 37.1\\%, outperforming GPT-5 (35.1\\%) while being 2.5x more efficient. On $\\tau^2$-Bench and FRAMES, Orchestrator surpasses GPT-5 by a wide margin while using only about 30\\% of the cost. Extensive analysis shows that Orchestrator achieves the best trade-off between performance and cost under multiple metrics, and generalizes robustly to previously unseen tools.  These results demonstrate that composing diverse tools with a lightweight orchestration model is both more efficient and more effective than existing methods, paving the way for practical and scalable tool-augmented reasoning systems. These results demonstrate that orchestrating diverse tools with lightweight agents is not only more efficient, but also more effective, paving the way for practical and scalable tool-augmented reasoning systems."},"_bibtex":{"value":"@misc{\nsu2026toolorchestra,\ntitle={ToolOrchestra: Elevating Intelligence via Efficient Model and Tool Orchestration},\nauthor={Hongjin SU and Shizhe Diao and Ximing Lu and Mingjie Liu and Jiacheng Xu and Xin Dong and Yonggan Fu and Peter Belcak and Hanrong Ye and Hongxu Yin and Yi Dong and Evelina Bakhturina and Tao Yu and Yejin Choi and Jan Kautz and Pavlo Molchanov},\nyear={2026},\nurl={https://openreview.net/forum?id=6vS1lNjPTa}\n}"},"title":{"value":"ToolOrchestra: Elevating Intelligence via Efficient Model and Tool Orchestration"},"pdf":{"value":"/pdf/a99d11745db82cbe0860c9c778e8d463f86f4578.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"su|toolorchestra_elevating_intelligence_via_efficient_model_and_tool_orchestration"},"authorids":{"value":["~Hongjin_SU1","~Shizhe_Diao2","~Ximing_Lu1","~Mingjie_Liu2","~Jiacheng_Xu2","~Xin_Dong1","~Yonggan_Fu1","~Peter_Belcak1","~Hanrong_Ye1","~Hongxu_Yin2","~Yi_Dong4","~Evelina_Bakhturina1","~Tao_Yu5","~Yejin_Choi1","~Jan_Kautz1","~Pavlo_Molchanov1"]},"authors":{"value":["Hongjin SU","Shizhe Diao","Ximing Lu","Mingjie Liu","Jiacheng Xu","Xin Dong","Yonggan Fu","Peter Belcak","Hanrong Ye","Hongxu Yin","Yi Dong","Evelina Bakhturina","Tao Yu","Yejin Choi","Jan Kautz","Pavlo Molchanov"]}},"version":2},{"content":{"summary":{"value":"The authors formalize the concept of \"chemistry\" between LLMs as the change in benefit when models are combined, proposing the CHEME algorithm to compute chemistry scores using Model Interaction Graphs (MIGs) and the RECOMMEND algorithm for optimal ensemble selection. Theoretical analysis demonstrates that chemistry emerges primarily under heterogeneous model performance profiles, with effects modulated by task type, complexity, and ensemble size. Empirical evaluation on the benchmarks provide evidence that chemistry-based selection can improve ensemble effectiveness in specific contexts."},"soundness":{"value":1},"confidence":{"value":4},"questions":{"value":"0. Authors are requested to respond to my comments in the weakness section above.\n1. In line 119, the quality and accuracy scores are both defined over the same index $i$, but the quality is defined on the outputs, and accuracy is defined over the LLMs. Can you clarify this?\n2. Why does equation 2 need to account for the benefit of a w.r.t. X? Shouldn't the benefit (or cost) of a be only dependent on b?\n3. How much historical performance data is required to reliably construct MIGs and estimate chemistry? For a new task domain without existing performance histories, what cold-start strategy would you recommend?\n4. How does the proposed RECOMMEND algorithm distinguish between beneficial (synergistic) and detrimental (antagonistic) high-chemistry pairs? Would a signed (non-absolute) metric that explicitly separates synergy from antagonism be more appropriate for a recommendation task?\n5. The paper reports a negative correlation between chemistry and ensemble complementarity for medium- and high-complexity tasks, attributing this to \"saturation effects.\" Could this finding also be interpreted as a failure of the chemistry metric itself, where it preferentially assigns high scores to redundant model pairs (whose individual benefits drop sharply when combined), thus actively selecting for lower complementarity?"},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. The paper makes a notable contribution by formalizing LLM interactions through a principled cost function framework with proven properties. Theorem 1 and Corollary 1 provide counterintuitive insights that chemistry emerges only under heterogeneous performance profiles, establishing clear theoretical boundaries for when the approach is applicable. \n2. The authors ground their novel concept in a solid mathematical framework, providing clear and formal definitions for cost, benefit, and chemistry.\n3. Rather than overclaiming effectiveness, the authors honestly report mixed results and provide thoughtful analysis of when chemistry helps versus hurts (e.g., saturation effects in high-performing ensembles, task complexity moderation)."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. My biggest concern lies in the computational complexity of the proposed CHEME algorithm. It requires finding the maximum interaction effect over all possible subsets X in S\\{a,b}. This complexity renders the method computationally intractable for any realistic scenario where the pool of candidate LLMs, $S$, is non-trivial (for example, |S| > 20)\n2. Despite claiming \"≈1,800 total experimental task executions,\" the actual dataset sizes are modest—only 10 records per trial across 10 trials means just 100 evaluation instances per configuration. For Liar benchmark (4,000 statements available), this represents 2.5% coverage. The high variance in results (e.g., Clinical Notes N=3 chemistry at 0.700 vs. baseline at 1.000) and the note about \"temporary service issues\" affecting claude-3-7 performance raise questions about result stability and whether observed differences reflect genuine chemistry effects or experimental noise.\n3. The core cost formulation (Equation 1) uses arbitrary weights (1/i) and a 75%/25% weighting between generation and review accuracy that lack principled justification. The paper doesn't explore sensitivity to these hyperparameter choices, leaving uncertainty about whether results depend critically on these specific formulations.\n4. The baseline selection strategies (Random, Performance, Local, Remote) are relatively simple. The paper doesn't compare against recent LLM routing methods (RouteLLM [1] , FrugalGPT [2] mentioned in related work), mixture-of-experts approaches, or learned selection (such as Smoothie [3] and RELM [4] which also uses a combination of model selection and evaluation) policies.\n\nRefs:\n1. Ong, Isaac, et al. \"Routellm: Learning to route llms with preference data.\" arXiv preprint arXiv:2406.18665 (2024).\n2. Chen, Lingjiao, Matei Zaharia, and James Zou. \"Frugalgpt: How to use large language models while reducing cost and improving performance.\" arXiv preprint arXiv:2305.05176 (2023).\n3. Guha, Neel, et al. \"Smoothie: Label free language model routing.\" Advances in Neural Information Processing Systems 37 (2024): 127645-127672.\n4. Kumar, Tarun, et al. \"Co-Optimizing Recommendation and Evaluation for LLM Selection\", ICLR 2025 Workshop on Foundation Models in the Wild."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762933004493,"tcdate":1761904281742,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission20110/Reviewer_zKce"],"signatures":["ICLR.cc/2026/Conference/Submission20110/Reviewer_zKce"],"forum":"T6IIvog2Bk","number":4,"license":"CC BY 4.0","cdate":1761904281742,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission20110/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762933004493,"domain":"ICLR.cc/2026/Conference","replyto":"T6IIvog2Bk","id":"D0CMlZiMfq","forumContent":{"TLDR":{"value":"We introduce the notion of LLM Chemistry resulting in a framework that measures where model combinations exhibit synergistic or antagonistic behaviors that shape collective performance beyond individual capabilities."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["LLM Chemistry","Multi-LLM Recommendation","Multi-LLM Collaboration"]},"primary_area":{"value":"generative models"},"abstract":{"value":"Multi-LLM collaboration promises accurate, robust, and context-aware solutions, yet existing approaches rely on implicit selection and output assessment without analyzing whether collaborating models truly complement or conflict. We introduce LLM Chemistry – a framework that measures when LLM combinations exhibit synergistic or antagonistic behaviors that shape collective performance beyond individual capabilities. We formalize the notion of chemistry among LLMs, propose algorithms that quantify it by analyzing interaction dependencies, and recommend optimal model ensembles accordingly. Our theoretical analysis shows that chemistry among collaborating LLMs is most evident under heterogeneous model profiles, with its outcome impact shaped by task type, group size, and complexity. Evaluation on classification, summarization, and program repair tasks provides initial evidence for these task-dependent effects, thereby reinforcing our theoretical results. This establishes LLM Chemistry as both a diagnostic factor in multi-LLM systems and a foundation for ensemble recommendation."},"_bibtex":{"value":"@misc{\nsanchez2026llm,\ntitle={{LLM} Chemistry Estimation for Multi-{LLM} Recommendation},\nauthor={Huascar Sanchez and Briland Hitaj},\nyear={2026},\nurl={https://openreview.net/forum?id=T6IIvog2Bk}\n}"},"title":{"value":"LLM Chemistry Estimation for Multi-LLM Recommendation"},"pdf":{"value":"/pdf/c0810b2144726d0d016332f3c8934851d4ebe084.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"sanchez|llm_chemistry_estimation_for_multillm_recommendation"},"authorids":{"value":["~Huascar_Sanchez1","~Briland_Hitaj1"]},"authors":{"value":["Huascar Sanchez","Briland Hitaj"]}},"version":2},{"content":{"summary":{"value":"This paper proposes Cascadia, a novel cascade serving framework for efficient & effective LLM serving. Cascadia employs a bi-level optimization strategy to compute the optimal serving strategy which includes the deployment strategy (i.e., the resource allocation and parallelism strategies) as well as the routing strategy (i.e., which user requests should be consumed by which models). Extensive experiments on real-world datasets demonstrate the effectiveness of the proposed approach."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"What is the typical/expected number of iterations required to achieve stable solutions? If the number of iteration tends to be huge, it can lead to non-negligible overheads and compromise the efficiency gains."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"1. This paper studies efficient and effective LLM serving, which is critical problem for a wide range of real-world applications.\n2. This paper introduces Cascadia with rigorous and solid technical developments.\n3. Cascadia achieves up to 4x lower latency deadlines and 5x higher system throughputs, which are impressive."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. In Algorithm 1, Cascadia relies on iteratively optimizing both deployment and routing strategies. It remains unclear if the bi-level optimization is theoretically optimal or not. Such guarantees could be critical in practical scenarios.\n2. LLM routing is another well-studied technique aiming for efficient & effective LLM serving, which is under-discussed in this paper. Authors may want to discuss and compare to this line of work to better position the contribution of this paper. Several example references are as follows,\n\n[1] Ong, Isaac, et al. \"RouteLLM: Learning to Route LLMs from Preference Data.\" The Thirteenth International Conference on Learning Representations.  \n[2] Ding, Dujian, et al. \"BEST-Route: Adaptive LLM Routing with Test-Time Optimal Compute.\" Forty-second International Conference on Machine Learning."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762923422066,"tcdate":1761998801963,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission12568/Reviewer_jBdC"],"signatures":["ICLR.cc/2026/Conference/Submission12568/Reviewer_jBdC"],"forum":"wkrWbrHTJQ","number":2,"license":"CC BY 4.0","cdate":1761998801963,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission12568/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762923422066,"domain":"ICLR.cc/2026/Conference","replyto":"wkrWbrHTJQ","id":"pr2PCralpM","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Distributed","Parallel","and Cluster Computing"]},"primary_area":{"value":"infrastructure, software libraries, hardware, systems, etc."},"abstract":{"value":"Recent advances in large language models (LLMs) have intensified the need to deliver both rapid responses and high-quality outputs. More powerful models yield better results but incur higher inference latency, whereas smaller models are faster yet less capable. Recent work proposes balancing this latency–quality trade-off using model cascades, which route simpler queries to smaller models and more complex ones to larger models. However, enabling efficient cascade serving remains challenging. Current frameworks lack effective mechanisms for handling (i) the huge and varying resource demands of different LLMs, (ii) the inherent heterogeneity of LLM workloads, and (iii) the co-optimization of system deployment and routing strategy.\n\nMotivated by these observations, we introduce Cascadia, a novel cascade serving framework designed explicitly to schedule request routing and deploy model cascades for fast, quality-preserving LLM serving. Cascadia employs a bi-level optimization method: at the deployment level, it uses a mixed-integer linear program to select resource allocations and parallelism strategies based on LLM information and workload characteristics; at the routing level, it applies a Chebyshev-guided method to iteratively co-optimize the routing strategy and the system deployment produced by the deployment level. Our extensive evaluation on diverse workload traces and different model cascades (DeepSeek and the Llama series) demonstrates that Cascadia significantly outperforms both single-model deployments and the state-of-the-art cascade serving baseline, achieving up to 4$\\times$ (2.3$\\times$ on average) tighter latency SLOs and up to 5$\\times$ (2.4$\\times$ on average) higher throughput while maintaining target answer quality."},"_bibtex":{"value":"@inproceedings{\njiang2026cascadia,\ntitle={Cascadia: An Efficient Cascade Serving System for Large Language Models},\nauthor={Youhe Jiang and Fangcheng Fu and Wanru Zhao and Stephan Rabanser and Jintao Zhang and Nicholas D. Lane and Binhang Yuan},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=wkrWbrHTJQ}\n}"},"title":{"value":"Cascadia: An Efficient Cascade Serving System for Large Language Models"},"pdf":{"value":"/pdf/a1ec00676c2c3a4e0e9b432a8676721778d8cd3d.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"jiang|cascadia_an_efficient_cascade_serving_system_for_large_language_models"},"authorids":{"value":["~Youhe_Jiang3","~Fangcheng_Fu1","~Wanru_Zhao1","~Stephan_Rabanser1","~Jintao_Zhang2","~Nicholas_D._Lane2","~Binhang_Yuan1"]},"authors":{"value":["Youhe Jiang","Fangcheng Fu","Wanru Zhao","Stephan Rabanser","Jintao Zhang","Nicholas D. Lane","Binhang Yuan"]}},"version":2},{"content":{"comment":{"value":"> W2: The paper lacks a deeper theoretical justification for why modular predictors generalize better than joint models\n\n> W3: why they perform well in OOD tasks\n\nWe thank the reviewer for the request for deeper theoretical justification. The empirical superiority of modular predictors over joint models (e.g., RouteLLM, RTR) can be understood through recent theory (Boopathy et al., 2024 [1]). Boopathy et al. define a task as modularly structured if its target function can be decomposed into independent sub-functions. In our setting, the goal of predicting accuracy and cost for all strategies on a task can naturally be decomposed into independent sub-goals, predicting accuracy and cost for each strategy separately, so our problem satisfies this notion of modularity.\n\nA key theoretical insight discovered by Boopathy et al. is that as task dimensionality increases, training modular models becomes much easier than training joint models, leading to better generalization both in-distribution and out-of-distribution. For our framework, a joint model must consider all strategies simultaneously, resulting in a high task dimension, whereas each modular predictor only handles a single strategy, greatly reducing task dimensionality. This provides a theoretical explanation for why our modular design generalizes more effectively and performs well on OOD tasks. We will add a more thorough discussion in our revised paper.\n\n[1] Boopathy, Akhilan, et al. \"Breaking neural network scaling laws with modularity.\" arXiv preprint arXiv:2409.05780 (2024).\n\n> W2: The optimization formulations are straightforward but not novel in theory.\n\nWhile the optimization formulations themselves are straightforward, the novelty of our work lies in addressing key limitations of model architectures proposed by existing routing frameworks: (1) limited scalability caused by monolithic models, and (2) restricted parametrizations that reduce effectiveness. To address these limitations, our key contribution is a novel modular architecture that utilizes multiple representations of input tasks and strategies, which, as demonstrated in our experiments, consistently outperforms previous routing systems across a wide range of tasks.\n\nWe adopt the predict-then-optimize approach rather than end-to-end learning proposed in prior work (e.g., RouteLLM) because it (a) allows users to explicitly incorporate constraints, and (b) achieves substantially better performance than end-to-end approaches as seen in experiments."},"title":{"value":"Author response 2 (A2)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764806959566,"tcdate":1764800681286,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission14463/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission14463/Authors"],"forum":"gCUY6QIv8r","number":2,"license":"CC BY 4.0","cdate":1764800681286,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission14463/-/Official_Comment"],"mdate":1764806959566,"domain":"ICLR.cc/2026/Conference","replyto":"9Qy7RA0WxN","id":"0Lnt06Hlv4","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["model routing","continual routing","constrained routing","unconstrained routing"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"AI tasks differ in complexity and are best addressed with different computation strategies (e.g., combinations of models and decoding methods). Hence, an effective routing system that maps tasks to the appropriate strategies is crucial.\nMost prior methods build the routing framework by training a *single* model across *all* strategies, which demands full retraining whenever new strategies appear and leads to high overhead. Attempts at such continual routing, however, often face difficulties with generalization.\nPrior models also typically use a *single* input representation, limiting their ability to capture the full complexity of the routing problem and leading to sub-optimal routing decisions.\nTo address these gaps, we propose CONCUR, a **con**tinual routing framework that supports both **c**onstrained and **u**nconstrained **r**outing (i.e., routing with or without a budget).\nOur *modular* design trains a separate predictor model for each strategy, enabling seamless incorporation of new strategies with low additional training cost.\nOur predictors also leverage *multiple* representations of both tasks and computation strategies to better capture overall problem complexity.\nExperiments on both in-distribution and out-of-distribution, knowledge- and reasoning-intensive tasks show that our method outperforms the best single strategy and strong existing routing techniques with higher end-to-end accuracy and lower inference cost in both continual and non-continual settings, while also reducing training cost in the continual setting."},"_bibtex":{"value":"@inproceedings{\nchen2026concur,\ntitle={{CONCUR}: A Framework for Continual Constrained and Unconstrained Routing},\nauthor={Peter Baile Chen and Weiyue Li and Dan Roth and Mike Cafarella and Samuel Madden and Jacob Andreas},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=gCUY6QIv8r}\n}"},"title":{"value":"CONCUR: A Framework for Continual Constrained and Unconstrained Routing"},"pdf":{"value":"/pdf/b59e1bbe2f0d9d26266b4e43191885655a9464f7.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"chen|concur_a_framework_for_continual_constrained_and_unconstrained_routing"},"authorids":{"value":["~Peter_Baile_Chen1","~Weiyue_Li1","~Dan_Roth3","~Mike_Cafarella1","~Samuel_Madden1","~Jacob_Andreas1"]},"authors":{"value":["Peter Baile Chen","Weiyue Li","Dan Roth","Mike Cafarella","Samuel Madden","Jacob Andreas"]}},"version":2},{"content":{"comment":{"value":"> Moreover, the RouteLLM paper is not peer-reviewed and is currently under submission to the same conference (OpenReview link). Penalizing us for not citing it is inconsistent with ICLR review guidelines, which state submissions should not be penalized for missing non-peer-reviewed references (link).\n\nYou do cite RouteLLM in your original manuscript and in its current version so I am not penalizing your work for not citing them. I am questioning your claim of being the first to do evaluation without execution:  \nFrom your rebuttal response:\n> We propose the first LLM-based performance predictor capable of estimating the performance of individual LLMs without executing them.\n\nFrom your paper:\n> Different from previous work, our Routoo identifies the most suitable expert without executing the inference of underlying LLMs. \n\nI agree on the other novelties (diverse LLMs, number of LLMs, lowest inference cost so far) and am not questioning them so there isn't a need to bring them up during the discussion of this claim.\n\n\n> For the encoding of $q_{train}$, as in the MMLU benchmark, answers are single characters (see dataset), the difference between encoding the question with the answer or just the question is minimal. We chose cosine similarity because the sizes of the underlying models range from 7B to 34B, with many smaller models (e.g., 7B) having similar sizes. Assigning identical scores (“1” for all top-10) would prevent our cost-aware selector from functioning effectively, as all models would appear equally suitable, requiring random selection. Retaining cosine similarity scores provides more granular information for our cost-aware selector.\n\nThat is a fair argument that the answer does not matter so the cosine score is really measuring the difference between $q_{test}$ and $q_{train}$. So why include the answer? Also, could you elaborate how that would help differentiate between all the models in consideration that answer the $q_{train}$ question correctly? They would all have the same $q_{train}+a_{train}$, where $a_{train}$ is the correct single-character answer to $q_{train}$ and thus same similarity score. So wouldn't you have to rely on random selection to select among them?"}},"tmdate":1732919145339,"tcdate":1732919145339,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission11514/Reviewer_o5Db"],"signatures":["ICLR.cc/2025/Conference/Submission11514/Reviewer_o5Db"],"forum":"RQ9fQLEajC","number":18,"license":"CC BY 4.0","cdate":1732919145339,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission11514/-/Official_Comment"],"mdate":1732919145339,"domain":"ICLR.cc/2025/Conference","replyto":"YMv1ZAGG99","id":"PchyVk4vvj","forumContent":{"TLDR":{"value":"An LLM-based Router trained to route to the most effective LLM optimizing performance, cost, and efficiency, creating a new state-of-the-art model by integrating knowledge of underlying models."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["LLM","efficiency","knowledge integration","generation"]},"supplementary_material":{"value":"/attachment/2e84b1f1c381960311d11bdd23f4d081c4e4d14a.zip"},"primary_area":{"value":"generative models"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"LLMs with superior response quality—particularly larger or closed-source models—often come with higher inference costs, making their deployment inefficient and costly. Meanwhile, developing foundational LLMs from scratch is becoming increasingly resource-intensive and impractical for many applications. To address the challenge of balancing quality and cost, we introduce Routoo, an architecture designed to optimize the selection of LLMs for specific prompts based on performance, cost, and efficiency. Routoo provides controllability over the trade-off between inference cost and quality, enabling significant reductions in inference costs for a given quality requirement.\nRoutoo comprises two key components: a performance predictor and cost-aware selector. The performance predictor is a lightweight LLM that estimates the expected performance of various underlying LLMs on a given prompt without executing them. The cost-aware selector module then selects the most suitable model based on these predictions and constraints such as cost and latency, significantly reducing inference costs for the same quality. \nWe evaluated Routoo using the MMLU benchmark across 57 domains employing open-source models. Our results show that Routoo matches the performance of the Mixtral 8x7b model while reducing inference costs by one-third. Additionally, by allowing increased costs, Routoo surpasses Mixtral's accuracy by over 5\\% at equivalent costs, achieving an accuracy of 75.9\\%. When integrating GPT4 into our model pool, Routoo nearly matches GPT4's performance at half the cost and exceeds it with a 25\\% cost reduction. \nThese outcomes highlight Routoo's potential to significantly reduce inference costs without compromising quality, and even to establish new state-of-the-art results by leveraging the collective capabilities of multiple LLMs."},"_bibtex":{"value":"@misc{\nmohammadshahi2025routoo,\ntitle={Routoo: Learning to Route to Large Language Models Effectively},\nauthor={Alireza Mohammadshahi and Arshad Rafiq Shaikh and Majid Yazdani},\nyear={2025},\nurl={https://openreview.net/forum?id=RQ9fQLEajC}\n}"},"title":{"value":"Routoo: Learning to Route to Large Language Models Effectively"},"pdf":{"value":"/pdf/454cdce2a12d959ded0bc50a6c5b05c494a2d4dd.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"mohammadshahi|routoo_learning_to_route_to_large_language_models_effectively"},"authorids":{"value":["~Alireza_Mohammadshahi1","~Arshad_Rafiq_Shaikh1","~Majid_Yazdani1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Alireza Mohammadshahi","Arshad Rafiq Shaikh","Majid Yazdani"]}},"version":2},{"content":{"summary":{"value":"GMTRouter tackles personalized LLM routing by modeling multi-turn user–LLM interactions as a heterogeneous graph whose nodes are users, LLMs, queries, and responses, with user feedback embedded as a preference feature. A lightweight inductive Heterogeneous Graph Transformer aggregates information over sampled conversation “turn” nodes, and a cross-attention prediction head ranks candidate LLMs for a new (user, query) pair using only a few past interactions. GMTRouter consistently outperforms baselines and generalizes well to unseen users with few-shot data. The model is compact and trainable on a single GPU."},"soundness":{"value":4},"confidence":{"value":4},"questions":{"value":"See weaknesses."},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. The paper tests on one real-world and three synthetic benchmarks, builds multi-user labels that combine quality, cost, length, and rare-word signals, reports clear metrics, and assess generalization.\n2. Practical, resource-efficient design suitable for deployment. GMTRouter is small, with modest storage and max GPU usage (~4.3 GB), and the experiments run on a single RTX A6000."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Missing baselines. It seems that the paper is missing some baselines achieving the personalization through graph. For example, Knowledge Graph Tuning: Real-time Large Language Model Personalization based on Human Feedback. \n2. This paper is claiming it is a LLM routing based model, however, it does not compare to baselines of routed LLMs. For example, RouteLLM: Learning to Route LLMs with Preference Data.\n3. What is the key challenge that this paper is trying to solve, all the modules seems pretty standard to the reviewer to form a GNN. \n4. The writing is a little bit confusing. All previous parts are emphasizing the routing and selection, but the majority of the methods is on constructing the GNN, until the very last of the section, the reviewer can see how you really make the selection."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762925026882,"tcdate":1761938163195,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission14657/Reviewer_s59p"],"signatures":["ICLR.cc/2026/Conference/Submission14657/Reviewer_s59p"],"forum":"oRe9tHpSpn","number":3,"license":"CC BY 4.0","cdate":1761938163195,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission14657/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762925026882,"domain":"ICLR.cc/2026/Conference","replyto":"oRe9tHpSpn","id":"P3WLMUKrsc","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Heterogeneous GNN","LLM routing","Preference Learning"]},"primary_area":{"value":"learning on graphs and other geometries & topologies"},"abstract":{"value":"Large Language Model (LLM) routing has demonstrated strong capability in balancing response quality with computational cost. As users exhibit diverse preferences, personalization has attracted increasing attention in LLM routing, since even identical queries may require different models to generate responses tailored to individual needs. However, existing approaches are not fully personalized and often fail to faithfully capture the complex interactions between specific users and LLMs. Moreover, user preference data is typically scarce, noisy, and inconsistent in format, which limits the effectiveness of methods that rely solely on user-specific data. To address these challenges, we propose GMTRouter, which represents multi-turn user–LLM interactions as a heterogeneous graph with four node types: user, LLM, query, and response, thereby maximally preserving the rich relational structure of the interaction. Through a tailored message-passing mechanism, GMTRouter learns to capture user preferences from few-shot data within a lightweight inductive graph learning framework, enabling effective personalization. Extensive experiments demonstrate that GMTRouter consistently outperforms the strongest baselines, achieving 0.9%–21.6% higher accuracy and 0.006–0.309 higher AUC across multiple datasets. More importantly, we further demonstrate that GMTRouter can adapt to new users and evolving preferences using only few-shot data, without extensive fine-tuning."},"_bibtex":{"value":"@misc{\nxie2026gmtrouter,\ntitle={{GMTR}outer: Personalized {LLM} Router over Multi-turn User Interactions},\nauthor={Encheng Xie and Yihang Sun and Tao Feng and Jiaxuan You},\nyear={2026},\nurl={https://openreview.net/forum?id=oRe9tHpSpn}\n}"},"title":{"value":"GMTRouter: Personalized LLM Router over Multi-turn User Interactions"},"pdf":{"value":"/pdf/b4f227b8ee59ab4f2a14d8fc028c0e06e9455c44.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"xie|gmtrouter_personalized_llm_router_over_multiturn_user_interactions"},"authorids":{"value":["~Encheng_Xie1","~Yihang_Sun2","~Tao_Feng5","~Jiaxuan_You2"]},"authors":{"value":["Encheng Xie","Yihang Sun","Tao Feng","Jiaxuan You"]}},"version":2},{"content":{"summary":{"value":"The paper introduces CARROT (Cost AwaRe Rate Optimal rouTer), a theoretically grounded and empirically validated framework for cost-aware routing among large language models (LLMs). While prior routing methods either ignore query-dependent cost variations or rely on costly multi-model inference cascades, CARROT achieves minimax-optimal efficiency by jointly predicting both accuracy and inference cost for each candidate model. The authors first present a minimax analysis of the routing problem, deriving a lower bound on excess risk and proving that a plug-in router—estimating conditional expectations of model metrics given a query—can asymptotically attain this bound. Building on this insight, CARROT implements a two-stage plug-in approach: it learns predictors for model-wise accuracy and cost from training data, then selects the model minimizing a convex combination of the two estimated risks. To support realistic training and evaluation, the authors introduce SPROUT (Smart Price-aware ROUTing), a large benchmark covering 14 state-of-the-art LLMs and over 45k prompts from reasoning, retrieval-augmented generation, and general instruction datasets."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"Please see Weaknesses."},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":1},"strengths":{"value":"1. The performance of CARROT is competitive."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. For better or worse, the paper is heavily packed with formal theorems and minimax analyses. While this mathematical framing is thorough, it sometimes overshadows the practical contribution. It would be helpful if the authors clearly indicated where the proofs for each theorem are located.\n2. What's the difference between CARROT and prior routing work, i.e. the main novelty here? I believe similar multi-model risk estimators have been used in earlier “model selection” and “LLM routing” papers, e.g. RouteLLM. \n3. Although SPROUT is introduced as a new benchmark, it largely repackages existing benchmarks—specifically GPQA, MuSR, MMLU-Pro, MATH, OpenHermes, and RAGBench—rather than creating new data or evaluation protocols. Its construction mainly involves aggregating prompts and responses from prior datasets under a unified cost annotation framework, which limits its originality."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762923550909,"tcdate":1762548554010,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission12731/Reviewer_n1HQ"],"signatures":["ICLR.cc/2026/Conference/Submission12731/Reviewer_n1HQ"],"forum":"5O4YK4eUeT","number":5,"license":"CC BY 4.0","cdate":1762548554010,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission12731/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762923550909,"domain":"ICLR.cc/2026/Conference","replyto":"5O4YK4eUeT","id":"z9Vd26DtfQ","forumContent":{"TLDR":{"value":"We present a theoretical and empirical investigation of performance and cost trade-offs in LLM routing."},"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["LLM Routing","minimax rate"]},"supplementary_material":{"value":"/attachment/34ba0a59e064883721f4d0cc6cbc0e74c811435b.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"With the rapid growth in the number of Large Language Models (LLMs), there has been a recent interest in LLM routing, or directing queries to the cheapest LLM that can deliver a suitable response. We conduct a minimax analysis of the routing problem, providing a lowerbound and finding that a simple router that predicts both cost and accuracy for each question can be minimax optimal. Inspired by this, we introduce CARROT, a Cost AwaRe Rate Optimal rouTer that selects a model based on estimates of the models’ cost and performance. Alongside CARROT, we also introduce the Smart Price-aware ROUTing (SPROUT) dataset to facilitate routing on a wide spectrum of queries with the latest state-of-the-art LLMs. Using SPROUT and prior benchmarks such as Routerbench and open-LLM-leaderboard- v2 we empirically validate CARROT’s performance against several alternative routers."},"_bibtex":{"value":"@misc{\nsomerstep2025carrot,\ntitle={{CARROT}: A Cost Aware Rate Optimal Router},\nauthor={Seamus Somerstep and Felipe Maia Polo and Allysson Flavio Melo de Oliveira and Prattyush Mangal and M{\\'\\i}rian Silva and Onkar Bhardwaj and Mikhail Yurochkin and Subha Maity},\nyear={2025},\nurl={https://openreview.net/forum?id=5O4YK4eUeT}\n}"},"title":{"value":"CARROT: A Cost Aware Rate Optimal Router"},"pdf":{"value":"/pdf/2d016d75c2d2f1edf6a669a3fb870ce88ed07993.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"somerstep|carrot_a_cost_aware_rate_optimal_router"},"authorids":{"value":["~Seamus_Somerstep1","~Felipe_Maia_Polo1","~Allysson_Flavio_Melo_de_Oliveira1","~Prattyush_Mangal1","~Mírian_Silva1","~Onkar_Bhardwaj1","~Mikhail_Yurochkin1","~Subha_Maity1"]},"authors":{"value":["Seamus Somerstep","Felipe Maia Polo","Allysson Flavio Melo de Oliveira","Prattyush Mangal","Mírian Silva","Onkar Bhardwaj","Mikhail Yurochkin","Subha Maity"]}},"version":2},{"content":{"venue":{"value":"CoRR 2025"},"pdf":{"value":"https://arxiv.org/pdf/2510.06078v1"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"zhe|constraintaware_route_recommendation_from_natural_language_via_hierarchical_llm_agents"},"html":{"value":"https://doi.org/10.48550/arXiv.2510.06078"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2510-06078,\n  publtype={informal},\n  author={Tao Zhe and Rui Liu and Fateme Memar and Xiao Luo and Wei Fan and Xinyue Ye and Zhongren Peng and Dongjie Wang},\n  title={Constraint-Aware Route Recommendation from Natural Language via Hierarchical LLM Agents},\n  year={2025},\n  month={October},\n  cdate={1759276800000},\n  journal={CoRR},\n  volume={abs/2510.06078},\n  url={https://doi.org/10.48550/arXiv.2510.06078}\n}\n"},"abstract":{"value":"Route recommendation aims to provide users with optimal travel plans that satisfy diverse and complex requirements. Classical routing algorithms (e.g., shortest-path and constraint-aware search) are efficient but assume structured inputs and fixed objectives, limiting adaptability to natural-language queries. Recent LLM-based approaches enhance flexibility but struggle with spatial reasoning and the joint modeling of route-level and POI-level preferences. To address these limitations, we propose RouteLLM, a hierarchical multi-agent framework that grounds natural-language intents into constraint-aware routes. It first parses user queries into structured intents including POIs, paths, and constraints. A manager agent then coordinates specialized sub-agents: a constraint agent that resolves and formally check constraints, a POI agent that retrieves and ranks candidate POIs, and a path refinement agent that refines routes via a routing engine with preference-conditioned costs. A final verifier agent ensures constraint satisfaction and produces the final route with an interpretable rationale. This design bridges linguistic flexibility and spatial structure, enabling reasoning over route feasibility and user preferences. Experiments show that our method reliably grounds textual preferences into constraint-aware routes, improving route quality and preference satisfaction over classical methods."},"title":{"value":"Constraint-Aware Route Recommendation from Natural Language via Hierarchical LLM Agents"},"authors":{"value":[{"fullname":"Tao Zhe","username":""},{"fullname":"Rui Liu","username":""},{"fullname":"Fateme Memar","username":""},{"fullname":"Xiao Luo","username":"~Xiao_Luo2"},{"fullname":"Wei Fan","username":""},{"fullname":"Xinyue Ye","username":""},{"fullname":"Zhongren Peng","username":""},{"fullname":"Dongjie Wang","username":""}]}},"tmdate":1791937329757,"pdate":1767139200000,"externalIds":["dblp:journals/corr/abs-2510-06078"],"tcdate":1791937327524,"writers":["~","OpenReview.net/Public_Article/DBLP.org","OpenReview.net/Support"],"signatures":["~Xiao_Luo3"],"forum":"wyBq94MloZ","license":"CC BY-SA 4.0","number":180490,"cdate":1759276800000,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/DBLP.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1791937329757,"domain":"OpenReview.net/Public_Article","id":"wyBq94MloZ","version":2},{"content":{"TLDR":{"value":"Step-level model routing across agent trajectories reduces frontier-model inference costs by 72% while maintaining task quality, by routing planning steps to large models and formatting steps to 7B models."},"venue":{"value":"LM4Plan@ICML-26 Poster"},"keywords":{"value":["model routing","agentic planning","cost-optimal LLM selection","multi-step workflows","trajectory-level optimization","LLM planning efficiency"]},"abstract":{"value":"Enterprise agentic systems that route every trajectory step to a frontier model waste 60-80% of their inference budget on subtasks that smaller models handle equally well. Existing routing solutions optimize single-turn query assignment but ignore a property unique to agentic workflows: subtask complexity varies widely within a single trajectory. A planning step may require frontier-class reasoning while a subsequent formatting step needs only a 7B model. We formalize step-level model routing as a sequential assignment problem over agent trajectories and propose AgentRouter, a lightweight classifier (12M parameters, <5ms overhead per step on an A100 GPU) that maps each trajectory step to one of four model tiers using five features extractable at routing time. Trained on 50,000 annotated agent trajectory steps across planning, coding, research, and data analysis, AgentRouter achieves 72% cost reduction with less than 3% quality degradation; per-step routing accuracy reaches 91% on minimal-complexity steps, 85% on efficient-tier steps, and 76-82% on the harder tiers. On the same benchmarks, RouteLLM and FrugalGPT (applied per-step) achieve only 31% and 44% cost reduction respectively, because their single-turn training signal misses trajectory-level quality dependencies."},"_bibtex":{"value":"@inproceedings{\nanonymous2026agentrouter,\ntitle={AgentRouter: Heterogeneous Model Routing for Cost-Optimal Multi-Step Agentic Workflows},\nauthor={Anonymous},\nbooktitle={ICML 2026 Workshop LM4Plan},\nyear={2026},\nurl={https://openreview.net/forum?id=05yAwXIImz}\n}"},"title":{"value":"AgentRouter: Heterogeneous Model Routing for Cost-Optimal Multi-Step Agentic Workflows"},"pdf":{"value":"/pdf/9a76c41c60b55a5f5d02fad74b22eb389f1d34d8.pdf"},"venueid":{"value":"ICML.cc/2026/Workshop/LM4Plan"}},"tmdate":1791933510110,"pdate":1779502869618,"tcdate":1775875700844,"writers":["ICML.cc/2026/Workshop/LM4Plan","ICML.cc/2026/Workshop/LM4Plan/Submission3/Authors"],"signatures":["ICML.cc/2026/Workshop/LM4Plan/Submission3/Authors"],"forum":"05yAwXIImz","license":"CC BY 4.0","number":3,"cdate":1775875700844,"readers":["everyone"],"invitations":["ICML.cc/2026/Workshop/LM4Plan/-/Submission","ICML.cc/2026/Workshop/LM4Plan/-/Post_Submission","ICML.cc/2026/Workshop/LM4Plan/-/Edit","ICML.cc/2026/Workshop/LM4Plan/Submission3/-/Revision"],"mdate":1791933510110,"odate":1791933510088,"domain":"ICML.cc/2026/Workshop/LM4Plan","id":"05yAwXIImz","version":2},{"content":{"TLDR":{"value":"RIDGE distills critic and forensic feedback into a role-conditioned hypernetwork that predicts model capability boundaries for dynamic, cost-efficient routing in multi-agent systems."},"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["Large Language Models","Multi-Agent Systems","Model Routing","Dynamic Model Selection","Cost-Efficient Inference","Role-Adaptive Routing"]},"supplementary_material":{"value":"/attachment/d0c8ca21f76edbfbb8b859856004894a88b112ca.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"LLM-based multi-agent systems (MASs) can support complex iterative workflows. However, they commonly face a challenging trade-off between performance and cost. Existing systems typically assign a fixed model to each agent role, ignoring variations in subtask difficulty. Furthermore, current LLM routers are mainly designed for single-turn conversations and rely on paired model outputs that are difficult to obtain in multi-agent workflows. To address these issues, we propose RIDGE, a lightweight router that dynamically identifies model capability boundaries during task execution. It combines online Critic evaluations with forensic attribution of failed tasks to distill a strong model’s assessment of execution quality, failure risk, and failure-critical steps into a role-conditioned hypernetwork. The hypernetwork generates role-specific prediction heads that separately estimate the potential execution states of fast and strong models from the current task semantics. An online MemoryBank further supports routing decisions. On SWE-bench Lite and WebArena-Verified, RIDGE matches the success rate of a static strong-model baseline while reducing strong-model token usage by up to 41.6%. It also routes 76.19% of failure-critical steps to the strong model, compared with 21.99% for OmniRouter and 15.06% for RouteLLM."},"_bibtex":{"value":"@inproceedings{\nanonymous2026ridge,\ntitle={{RIDGE}: Role-Adaptive Routing for {LLM}-based Multi-Agent Systems},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=wKPxx6lU4C},\nnote={under review}\n}"},"title":{"value":"RIDGE: Role-Adaptive Routing for LLM-based Multi-Agent Systems"},"pdf":{"value":"/pdf/d9ea9baca641dabe5c824071787e8ab1e344a256.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791232945752,"tcdate":1789725631349,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission40898/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission40898/Authors"],"forum":"wKPxx6lU4C","license":"CC BY 4.0","number":40898,"cdate":1789725631349,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission40898/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791232945752,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"wKPxx6lU4C","version":2},{"content":{"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["LLM reasoning","Inhibitory control","Test-time compute allocation"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Selective reasoning in language models must balance opportunities to correct an available answer against additional computation and harmful replacements. Inspired by inhibitory control in the human brain, we introduce Inhibitory Deliberation for Problem Reasoning (IDPR), which separates answer formation from answer release. A fast policy generates a candidate, and a response-conditioned gate either releases it or inhibits its release to invoke slow reasoning. Decomposed supervision uses paired fast--slow outcomes to learn slow-over-fast quality gain, generation cost, and corrective potential from the question, fast answer, and generation evidence. Calibration selects the most accurate of three scoring rules at fixed 8\\% coverage. On a fixed-policy benchmark of 5,000 mathematical problems, IDPR raises accuracy from 47.90\\% to 48.92\\% at an 8.20\\% slow-call rate, with 519.42 average generated tokens. The gain over Always-Fast is significant under a Holm-adjusted paired McNemar test ($p=0.0012$), with 111 corrected errors and 60 harmful replacements. With response-and-evidence inputs and encoder family held fixed, IDPR-Matched's joint supervision-and-scoring scheme achieves 1.40 percentage points higher accuracy than RouteLLM-Q+R+E while reducing slow calls from 9.46\\% to 7.14\\% at their respective validation-calibrated thresholds."},"_bibtex":{"value":"@inproceedings{\nanonymous2026when,\ntitle={When to Think Deeply: Response-Conditioned Inhibitory Routing for {LLM} Reasoning},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=37MRu7OmGp},\nnote={under review}\n}"},"title":{"value":"When to Think Deeply: Response-Conditioned Inhibitory Routing for LLM Reasoning"},"pdf":{"value":"/pdf/6398c7449621258fee114de00355e00c00573760.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791232028247,"tcdate":1789698086832,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission35899/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission35899/Authors"],"forum":"37MRu7OmGp","license":"CC BY 4.0","number":35899,"cdate":1789698086832,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/Submission35899/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791232028247,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"37MRu7OmGp","version":2},{"content":{"venue":{"value":"SCALE@ICML2026 Poster"},"LLM_usage":{"value":"No or minor, the authors took negligible or no help from any kind of LLM in the whole process."},"keywords":{"value":["agentic systems","multi-agent orchestration","LLM agents","bandits","optimal transport","uncertainty"]},"Funding":{"value":"Yes, the presenting author of this submission falls under ICML’s funding aims, and funding would significantly impact their ability to attend the workshop in person."},"abstract":{"value":"Adaptive orchestration of heterogeneous agents requires making sequential delegation decisions under uncertain and evolving agent behaviour, e.g., coordinating specialised AI models with varying reliability, cost, and response quality. While prior work on agent orchestration focuses on performance or cost, uncertainty in agent reliability and output distributions is typically not modelled explicitly at the orchestration level. In this work, we study the problem of adaptive orchestration of heterogeneous agents under uncertainty, where a meta-controller must decide when to delegate to an agent, accounting for reliability, cost, and uncertainty. We propose BOT-Orch, a lightweight framework that recasts orchestration as a bandit problem over agents, regularized by OT distances between agent output distributions and task-specific reference distributions. We show that the regularised orchestration enjoys $\\mathcal{O}(\\sqrt{T})$ regret under standard assumptions, and provably induces preference ordering among agents with identical mean rewards but differing distributional alignment. Empirically, we demonstrate that BOT-Orch outperforms standard and contextual bandits (UCB1, Thompson, LinUCB/LinTS) as well as deployed cost-aware LLM-routing baselines (RouteLLM, FrugalGPT, Auto) across synthetic and semi-synthetic (MNIST) settings with heterogeneous, non-i.i.d. agent behaviour, and we quantify the computational cost of the optimal-transport step."},"_bibtex":{"value":"@inproceedings{\noliver2026learning,\ntitle={Learning to Orchestrate Heterogeneous Agents under Uncertainty},\nauthor={Mary Chriselda Antony Oliver and Lan Jiang and Elaf Almahmoud and Francesco Quinzan and Umang Bhatt},\nbooktitle={ICML 2026 Workshop on Multimodal AI Agents},\nyear={2026},\nurl={https://openreview.net/forum?id=l4TnpuiC2j}\n}"},"title":{"value":"Learning to Orchestrate Heterogeneous Agents under Uncertainty"},"Anonymization":{"value":"This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs."},"pdf":{"value":"/pdf/9f2bc29d7be081027fdb43dc355ba8455be4bc3f.pdf"},"Format":{"value":"Yes, the presenting author will attend in person if this work is accepted to the workshop."},"Presenter":{"value":"~Lan_Jiang7"},"venueid":{"value":"ICML.cc/2026/Workshop/SCALE"},"paperhash":{"value":"oliver|learning_to_orchestrate_heterogeneous_agents_under_uncertainty"},"authorids":{"value":["~Mary_Chriselda_Antony_Oliver1","~Lan_Jiang7","~Elaf_Almahmoud1","~Francesco_Quinzan1","~Umang_Bhatt1"]},"Track":{"value":"Main track (up to 7 pages excluding references and appendix)"},"authors":{"value":["Mary Chriselda Antony Oliver","Lan Jiang","Elaf Almahmoud","Francesco Quinzan","Umang Bhatt"]}},"tmdate":1783264827043,"pdate":1778875849566,"tcdate":1777395512767,"writers":["ICML.cc/2026/Workshop/SCALE","ICML.cc/2026/Workshop/SCALE/Submission59/Authors"],"signatures":["ICML.cc/2026/Workshop/SCALE/Submission59/Authors"],"forum":"l4TnpuiC2j","license":"CC BY 4.0","number":59,"cdate":1777395512767,"readers":["everyone"],"invitations":["ICML.cc/2026/Workshop/SCALE/-/Submission","ICML.cc/2026/Workshop/SCALE/-/Post_Submission","ICML.cc/2026/Workshop/SCALE/-/Edit","ICML.cc/2026/Workshop/SCALE/Submission59/-/Camera-Ready_Revision"],"mdate":1783264827043,"odate":1783264826742,"domain":"ICML.cc/2026/Workshop/SCALE","id":"l4TnpuiC2j","version":2},{"content":{"data_release":{"value":"We authorize the release of our submission and author names to the public in the event of acceptance."},"TLDR":{"value":"AgentRouter demonstrates that step-level routing of agent trajectory steps to heterogeneous model tiers achieves 72% cost reduction at scale, outperforming adapted single-turn routers by 28–41 percentage points."},"venue":{"value":"ICML 2026 Workshop AgenticUQ Poster"},"email_sharing":{"value":"We authorize the sharing of all author emails with Program Chairs."},"keywords":{"value":["scalable AI inference","model routing","cost optimization","heterogeneous model tiers","agentic workflows","efficient foundation models","step-level routing"]},"abstract":{"value":"Enterprise agentic systems that route every trajectory step to a frontier model waste 60–80% of their inference budget on subtasks that smaller models handle equally well. Existing routing solutions optimize single-turn query assignment but ignore a property unique to agentic workflows: subtask complexity varies widely within a single trajectory. A planning step may require frontier-class reasoning while a subsequent formatting step needs only a 7B model. We formalize step-level model routing as a sequential assignment problem over agent trajectories and propose AgentRouter, a lightweight classifier (12M parameters, <5ms overhead per step on an A100 GPU) that maps each trajectory step to one of four model tiers using five features extractable at routing time. Trained on 50,000 annotated agent trajectory steps across planning, coding, research, and data analysis, AgentRouter achieves 72% cost reduction with less than 3% quality degradation; per-step routing accuracy reaches 91% on minimal-complexity steps, 85% on efficient-tier steps, and 76–82% on the harder tiers. On the same benchmarks, RouteLLM and FrugalGPT (applied per-step) achieve only 31% and 44% cost reduction respectively, because their single-turn training signal misses trajectory-level quality dependencies."},"_bibtex":{"value":"@inproceedings{\npaul2026agentrouter,\ntitle={AgentRouter: Heterogeneous Model Routing for Cost-Optimal Multi-Step Agentic Workflows},\nauthor={Rudrendu Kumar Paul and Sourav Nandy},\nbooktitle={ICML 2026 Statistical Frameworks for Uncertainty in Agentic Systems},\nyear={2026},\nurl={https://openreview.net/forum?id=XWc0Yw8cig}\n}"},"title":{"value":"AgentRouter: Heterogeneous Model Routing for Cost-Optimal Multi-Step Agentic Workflows"},"pdf":{"value":"/pdf/6ee2c8daa3b7a23707d1c064b1d383576465333b.pdf"},"venueid":{"value":"ICML.cc/2026/Workshop/AgenticUQ"},"paperhash":{"value":"paul|agentrouter_heterogeneous_model_routing_for_costoptimal_multistep_agentic_workflows"},"authorids":{"value":["~Rudrendu_Kumar_Paul1","~Sourav_Nandy2"]},"authors":{"value":["Rudrendu Kumar Paul","Sourav Nandy"]}},"tmdate":1782482402708,"pdate":1780246182349,"tcdate":1776403676339,"writers":["ICML.cc/2026/Workshop/AgenticUQ","ICML.cc/2026/Workshop/AgenticUQ/Submission8/Authors"],"signatures":["ICML.cc/2026/Workshop/AgenticUQ/Submission8/Authors"],"forum":"XWc0Yw8cig","license":"CC BY 4.0","number":8,"cdate":1776403676339,"readers":["everyone"],"invitations":["ICML.cc/2026/Workshop/AgenticUQ/-/Submission","ICML.cc/2026/Workshop/AgenticUQ/-/Submission_Change_Before_Reviewing","ICML.cc/2026/Workshop/AgenticUQ/-/Edit","ICML.cc/2026/Workshop/AgenticUQ/Submission8/-/Camera_Ready_Revision","ICML.cc/2026/Workshop/AgenticUQ/-/Submission_Release"],"mdate":1782482402708,"odate":1782482401292,"domain":"ICML.cc/2026/Workshop/AgenticUQ","id":"XWc0Yw8cig","version":2},{"content":{"summary":{"value":"- Formulates LLM routing as a minimax learning problem, deriving lower and upper bounds on excess risk and proving that a plug-in router predicting both cost and accuracy is rate-optimal.\n- Introduces CARROT, an implementation of this theoretical plug-in router that selects the cheapest model expected to meet performance goals.\n- Releases SPROUT, a new dataset (~45 k prompts × 14 LLMs) including per-query cost, accuracy, and chat-template formatting for studying cost-aware routing.\n- Empirically shows CARROT achieves similar accuracy as GPT-4-level models at much lower cost, outperforming prior routers such as RouteLLM and RouterBench."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"what do the authors mean by \"Unlike previous benchmarks, it does not rely on few-shot prompting\"? Few shot prompting is critical in many cases to get good task performance. In fact, different models prefer different type of few shot prompting (along the lines of system prompts are adapted for the model)"},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"the paper provides a clear mathematical framework for the routing problem with a principled treatment of routing - a increasingly practical relevant problem"},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- Limited novelty. The main idea of routing queries between models based on cost or accuracy is well established. The theoretical analysis adds formality but does not lead to a substantively new method or insight.\n- Model–prompt coupling not addressed. In practice, system prompts are heavily engineered for specific models. Swapping models without re-tuning instructions is rarely valid, and the mild adjustments through per-model chat templates seem insufficient to ensure fair comparison or practical relevance.\n- Non-deterministic model behavior ignored. LLM outputs can vary substantially across runs, yet the paper assumes deterministic outputs and evaluates each (prompt, model) pair only once. This undermines the realism of both the theoretical and empirical results.\n- Unclear link between “accuracy” and task performance. It is not evident how the reported benchmark accuracy maps to real task success. Many queries have no single correct answer, and performance may depend on stylistic or preference factors. If the authors addressed this, it was not clearly explained.\n- Zero-shot prompting assumption. The authors frame zero-shot prompting as a strength, but in practice few-shot or instruction-tuned prompting is essential for good performance. This limits the applicability of the results to real deployment settings.\n- Incomplete literature coverage and outdated baselines. The paper is not the first to consider cost-aware routing. Prior work and other dynamic routing frameworks address similar questions with more advanced or realistic approaches."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762923553437,"tcdate":1761699105797,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission12731/Reviewer_4zBY"],"signatures":["ICLR.cc/2026/Conference/Submission12731/Reviewer_4zBY"],"forum":"5O4YK4eUeT","number":1,"license":"CC BY 4.0","cdate":1761699105797,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission12731/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762923553437,"domain":"ICLR.cc/2026/Conference","replyto":"5O4YK4eUeT","id":"4ldx57hw65","forumContent":{"TLDR":{"value":"We present a theoretical and empirical investigation of performance and cost trade-offs in LLM routing."},"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["LLM Routing","minimax rate"]},"supplementary_material":{"value":"/attachment/34ba0a59e064883721f4d0cc6cbc0e74c811435b.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"With the rapid growth in the number of Large Language Models (LLMs), there has been a recent interest in LLM routing, or directing queries to the cheapest LLM that can deliver a suitable response. We conduct a minimax analysis of the routing problem, providing a lowerbound and finding that a simple router that predicts both cost and accuracy for each question can be minimax optimal. Inspired by this, we introduce CARROT, a Cost AwaRe Rate Optimal rouTer that selects a model based on estimates of the models’ cost and performance. Alongside CARROT, we also introduce the Smart Price-aware ROUTing (SPROUT) dataset to facilitate routing on a wide spectrum of queries with the latest state-of-the-art LLMs. Using SPROUT and prior benchmarks such as Routerbench and open-LLM-leaderboard- v2 we empirically validate CARROT’s performance against several alternative routers."},"_bibtex":{"value":"@misc{\nsomerstep2025carrot,\ntitle={{CARROT}: A Cost Aware Rate Optimal Router},\nauthor={Seamus Somerstep and Felipe Maia Polo and Allysson Flavio Melo de Oliveira and Prattyush Mangal and M{\\'\\i}rian Silva and Onkar Bhardwaj and Mikhail Yurochkin and Subha Maity},\nyear={2025},\nurl={https://openreview.net/forum?id=5O4YK4eUeT}\n}"},"title":{"value":"CARROT: A Cost Aware Rate Optimal Router"},"pdf":{"value":"/pdf/2d016d75c2d2f1edf6a669a3fb870ce88ed07993.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"somerstep|carrot_a_cost_aware_rate_optimal_router"},"authorids":{"value":["~Seamus_Somerstep1","~Felipe_Maia_Polo1","~Allysson_Flavio_Melo_de_Oliveira1","~Prattyush_Mangal1","~Mírian_Silva1","~Onkar_Bhardwaj1","~Mikhail_Yurochkin1","~Subha_Maity1"]},"authors":{"value":["Seamus Somerstep","Felipe Maia Polo","Allysson Flavio Melo de Oliveira","Prattyush Mangal","Mírian Silva","Onkar Bhardwaj","Mikhail Yurochkin","Subha Maity"]}},"version":2},{"content":{"decision":{"value":"Accept (poster)"},"comment":{"value":"This paper introduces EcoRouter, a recommendation-system-based framework for routing user instructions to the most cost-effective LLM in a candidate pool. By constructing multidimensional model and data representations (intrinsic properties, benchmark capability scores, and cross features) and training a Wide&Deep/DeepFM-style recommender on >1B synthetic interactions, EcoRouter learns to predict which (model, inference mode) pairing best satisfies a user-defined accuracy target under cost constraints. A Pareto frontier–based evaluation metric is also proposed. Empirically, EcoRouter reduces costs by >38% on six benchmarks across 50+ LLMs while maintaining accuracy.\n\nStrengths. Reviewers agreed the framing of routing as a recommendation problem is both novel and practically important (Reviewer k8wC, N3dk, MPGf). The introduction of a Pareto-frontier cost–accuracy metric was noted as a methodological contribution (Reviewer N3dk). The large-scale evaluation across diverse LLMs and reasoning modes demonstrated strong headline improvements (Reviewer 3Co2). Practical deployment advantages, such as avoiding inference on all candidate models and adaptability to new LLMs, were emphasized as real-world strengths (Reviewer MPGf).\n\nWeaknesses. Several reviewers highlighted limitations in evaluation: missing competitive baselines such as RouteLLM, GraphRouter, Hybrid-LLM, and FrugalGPT (Reviewer 3Co2, N3dk, k8wC); absence of RouterBench coding/multi-turn benchmarks (Reviewer N3dk); and limited ablation analyses (Reviewer N3dk, k8wC). Cost modeling was flagged as simplified and heuristic, not tied to real API pricing or latency (Reviewer 3Co2, MPGf). Concerns about router overhead and whether cost savings persist in real deployments were raised (Reviewer 3Co2, MPGf). Reviewer 3Co2 also questioned possible train–test leakage from the MMLU-based coreset. Finally, the approach requires a massive precomputed interaction dataset (>1B samples), raising barriers for replication (Reviewer MPGf).\n\nDiscussion and Rebuttal. The authors’ rebuttal was substantive and constructive. They added nine missing baselines (FrugalGPT, GraphRouter, Hybrid-LLM, RouteLLM, etc.) and seven new benchmarks (including coding and dialogue), as well as ablations on feature groups and sensitivity to training data size. They clarified that router overhead is ~0.3s vs. 5–7s for a 7B LLM, and committed to adding real API pricing experiments. They also confirmed no leakage between the bridge coreset and test sets. Reviewers largely accepted these clarifications: Reviewer k8wC explicitly raised their score; Reviewer N3dk stated all concerns were fully addressed; Reviewer 3Co2 maintained a borderline-accept rating but acknowledged improvements; Reviewer MPGf appreciated the responses but held their borderline-reject score, citing only partial resolution of concerns.\n\nDecision. While one reviewer remains unconvinced, the majority of reviewers shifted positively after rebuttal, recognizing the paper’s strong empirical contribution, novel framing, and practical significance for LLM deployment. The weaknesses (e.g., dataset construction cost, proxy cost model) are real but do not undermine the central technical contributions. Given its relevance to scalable and economical LLM usage—a timely and practically pressing problem—I recommend acceptance. However, due to residual concerns about evaluation completeness and cost realism, I do not recommend spotlight/oral; the work is valuable but not yet fully mature in evaluation rigor.\n\nFollow-up for Camera-Ready. For the final version, please:\n1. Incorporate the newly added baselines (FrugalGPT, GraphRouter, Hybrid-LLM, RouteLLM, etc.) and expanded benchmarks (e.g., coding, dialogue) into the main paper.\n2. Provide empirical results under real API pricing and latency measurements, not only heuristic proxies, to strengthen cost-effectiveness claims.\n3. Explicitly report routing overhead in ms/query and clarify the amortized cost relative to direct large-model inference.\n4. Clearly document how the bridge coreset is separated from evaluation datasets to avoid any impression of train–test leakage.\n5. Discuss the scalability and accessibility of the required 1B+ interaction dataset, and outline strategies for researchers with limited resources (e.g., smaller-scale training feasibility).\n6. Add a short discussion of broader limitations, such as reliance on single-turn prompts and the simplified cost definition, as flagged by reviewers.\n\nRecommendation: Accept."},"title":{"value":"Paper Decision"}},"parentInvitations":"NeurIPS.cc/2025/Conference/-/Decision","nonreaders":[],"tmdate":1761718706890,"tcdate":1758113604062,"writers":["NeurIPS.cc/2025/Conference","NeurIPS.cc/2025/Conference/Program_Chairs"],"signatures":["NeurIPS.cc/2025/Conference/Program_Chairs"],"forum":"uC3DI4YPRv","number":1,"license":"CC BY 4.0","cdate":1758113604062,"readers":["everyone"],"invitations":["NeurIPS.cc/2025/Conference/Submission24785/-/Decision","NeurIPS.cc/2025/Conference/-/Edit"],"mdate":1761718706890,"domain":"NeurIPS.cc/2025/Conference","replyto":"uC3DI4YPRv","id":"UQ5HKW4D1Z","forumContent":{"venue":{"value":"NeurIPS 2025 poster"},"keywords":{"value":["LLM Routing","Deep Learning"]},"primary_area":{"value":"deep_learning"},"abstract":{"value":"Recently, test-time scaling of Large Language Models (LLMs) has emerged as a practical alternative to parameter and data scaling. Reasoning tasks often require large-scale, RLVR-based LLMs, while more economical LLMs can handle simpler tasks. Routing an LLM tailored to *suitability* (*i.e.*, capability and cost) ensures usability and efficiency. We introduce LLMRec, which routes the most suitable LLM to the user query without pre-inference on the candidate LLM zoo. It pioneeringly reframes the LLM routing problem as a comprehensive recommendation system (RecSys) task. Our core insight is that an LLM's suitability for a query is a complex, latent signal equal to user-item preference. LLMRec systematically engineers features for candidate LLMs (intrinsic attributes and capability distributions), queries (general semantics and meta-dimensional info), and context (inference type, cost budgets). It also incorporates behavioral features to learn high-order interactions. LLMRec is designed to generalize to out-of-domain datasets and adapt to new LLMs as the model zoo evolves. We define the metric with the Pareto frontier under user-specified cost budgets. Across six datasets, LLMRec achieves an average cost reduction of over 38% while maintaining accuracy and consistently outperforming baselines in converging toward the Pareto frontier."},"_bibtex":{"value":"@inproceedings{\nzhang2025let,\ntitle={Let the {LLM} Stick to Its Strengths: Learning to Route Economical {LLM}},\nauthor={Yi-Kai Zhang and Shiyin Lu and Qing-Guo Chen and Weihua Luo and De-Chuan Zhan and Han-Jia Ye},\nbooktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},\nyear={2025},\nurl={https://openreview.net/forum?id=uC3DI4YPRv}\n}"},"title":{"value":"Let the LLM Stick to Its Strengths: Learning to Route Economical LLM"},"pdf":{"value":"/pdf/f8ce3f48c335fa784e0e6aef63e636d13c30d93d.pdf"},"venueid":{"value":"NeurIPS.cc/2025/Conference"},"paperhash":{"value":"zhang|let_the_llm_stick_to_its_strengths_learning_to_route_economical_llm"},"authorids":{"value":["~Yi-Kai_Zhang2","~Shiyin_Lu1","~Qing-Guo_Chen1","~Weihua_Luo2","~De-Chuan_Zhan1","~Han-Jia_Ye1"]},"authors":{"value":["Yi-Kai Zhang","Shiyin Lu","Qing-Guo Chen","Weihua Luo","De-Chuan Zhan","Han-Jia Ye"]}},"version":2},{"content":{"research_area_keywords":{"value":"LLM Router, LLM Efficiency, Cost optimization, NLP in resource-constrained settings;"},"languages_studied":{"value":"English"},"venue":{"value":"ACL ARR 2025 May Submission"},"_bibtex":{"value":"@inproceedings{\nanonymous2025dynaroute,\ntitle={Dynaroute: Dynamic Model Routing via Task Profiling and Cost Tiers},\nauthor={Anonymous},\nbooktitle={Submitted to ACL Rolling Review - May 2025},\nyear={2025},\nurl={https://openreview.net/forum?id=W2rbsUE01g},\nnote={under review}\n}"},"title":{"value":"Dynaroute: Dynamic Model Routing via Task Profiling and Cost Tiers"},"contribution_types":{"value":["Model analysis & interpretability","NLP engineering experiment","Approaches to low-resource settings"]},"abstract":{"value":"The rapid advancements in Large Language Models (LLMs) have led to a diverse landscape of models with varying capabilities and associated costs. No single LLM is optimal for all tasks, necessitating intelligent routing systems that can dynamically select the most appropriate model for a given input to balance performance and operational expense. In this study, we propose a novel benchmark-driven LLM routing framework designed to achieve a practical balance between task-specific performance and cost. As opposed to previous studies' frameworks, such as HybridLLM, RouteLLM, and LLMProxy, which often focus on binary classifiers for query assessment, our multi-stage system employs explicit task profiling using a lightweight classifier LLM to determine not only the query's category but also a more granular, multi-level difficulty. A key differentiating aspect is our tiered cost-performance model selection strategy, which systematically buckets models into cost percentiles and then selects the best-performing model within the appropriate tier for the predicted task profile, offering a more structured approach to balancing cost and performance. We evaluate the framework using three routing configurations. The Optimum router consistently achieves performance comparable to or exceeding the best individual models on specific tasks, but at significantly lower total costs."},"paper_type":{"value":"Long"},"pdf":{"value":"/pdf/fcf5e401a2fc39ae6da6e91dbdd12a1d383b6ba9.pdf"},"research_area":{"value":"Efficient/Low-Resource Methods for NLP"},"venueid":{"value":"aclweb.org/ACL/ARR/2025/May/Submission"}},"tmdate":1782962183166,"tcdate":1747711509154,"writers":["aclweb.org/ACL/ARR/2025/May","aclweb.org/ACL/ARR/2025/May/Submission4857/Authors"],"signatures":["aclweb.org/ACL/ARR/2025/May/Submission4857/Authors"],"forum":"W2rbsUE01g","license":"CC BY 4.0","number":4857,"cdate":1747711509154,"readers":["everyone"],"invitations":["aclweb.org/ACL/ARR/2025/May/-/Submission","aclweb.org/ACL/ARR/2025/May/-/Edit","aclweb.org/ACL/ARR/2025/May/-/Post_Submission","aclweb.org/ACL/ARR/2025/May/-/Desk_Reject_Verification_Flag","aclweb.org/ACL/ARR/2025/May/-/Preprint_Release_Submission","aclweb.org/ACL/ARR/2025/May/-/Preprint_Post_Submission"],"mdate":1782962183166,"odate":1753766020949,"domain":"aclweb.org/ACL/ARR/2025/May","id":"W2rbsUE01g","version":2},{"content":{"data_release":{"value":"We authorize the release of our submission and author names to the public in the event of acceptance."},"TLDR":{"value":"AgentRouter, a 12M-parameter step-level classifier, routes each agent trajectory step to one of four model tiers and reduces enterprise inference costs by 72% with less than 3% quality degradation."},"venue":{"value":"AdaptFM Poster"},"email_sharing":{"value":"We authorize the sharing of all author emails with Program Chairs."},"keywords":{"value":["model routing","resource-adaptive inference","agentic workflows","cost optimization","heterogeneous models","step-level routing","foundation models"]},"abstract":{"value":"Enterprise agentic systems that route every trajectory step to a frontier model waste 60-80% of their inference budget on subtasks that smaller models handle equally well. Existing routing solutions optimize single-turn query assignment but ignore a property unique to agentic workflows: subtask complexity varies widely within a single trajectory. A planning step may require frontier-class reasoning while a subsequent formatting step needs only a 7B model. We formalize step-level model routing as a sequential assignment problem over agent trajectories and propose AgentRouter, a lightweight classifier (12M parameters, <5ms overhead per step on an A100 GPU) that maps each trajectory step to one of four model tiers using five features extractable at routing time. Trained on 50,000 annotated agent trajectory steps spanning planning, coding, research, and data analysis tasks, AgentRouter achieves 72% cost reduction relative to frontier-only baselines, retaining 97.3% of frontier-only quality (less than 3% degradation in end-to-end task completion); per-step routing accuracy reaches 91% on minimal-complexity steps and 85% on efficient-tier steps, with 76-82% on the harder mid-range and frontier tiers. On the same benchmarks, RouteLLM and FrugalGPT (applied per-step) achieve only 31% and 44% cost reduction respectively, because their single-turn training signal misses trajectory-level quality dependencies."},"_bibtex":{"value":"@inproceedings{\npaul2026agentrouter,\ntitle={AgentRouter: Heterogeneous Model Routing for Cost-Optimal Multi-Step Agentic Workflows},\nauthor={Rudrendu Kumar Paul and Sourav Nandy},\nbooktitle={AdaptFM: Resource-Adaptive Foundation Model Inference},\nyear={2026},\nurl={https://openreview.net/forum?id=nu3GPfkyJV}\n}"},"title":{"value":"AgentRouter: Heterogeneous Model Routing for Cost-Optimal Multi-Step Agentic Workflows"},"pdf":{"value":"/pdf/6b9ccf623ac62f0a2476da7161d0aa044457b4aa.pdf"},"venueid":{"value":"ICML.cc/2026/Workshop/AdaptFM"},"paperhash":{"value":"paul|agentrouter_heterogeneous_model_routing_for_costoptimal_multistep_agentic_workflows"},"authorids":{"value":["~Rudrendu_Kumar_Paul1","~Sourav_Nandy2"]},"authors":{"value":["Rudrendu Kumar Paul","Sourav Nandy"]}},"tmdate":1780570443231,"pdate":1780353776095,"tcdate":1775875533252,"writers":["ICML.cc/2026/Workshop/AdaptFM","ICML.cc/2026/Workshop/AdaptFM/Submission2/Authors"],"signatures":["ICML.cc/2026/Workshop/AdaptFM/Submission2/Authors"],"forum":"nu3GPfkyJV","license":"CC BY 4.0","number":2,"cdate":1775875533252,"readers":["everyone"],"invitations":["ICML.cc/2026/Workshop/AdaptFM/-/Submission","ICML.cc/2026/Workshop/AdaptFM/-/Submission_Change_Before_Reviewing","ICML.cc/2026/Workshop/AdaptFM/-/Submission_Release","ICML.cc/2026/Workshop/AdaptFM/Submission2/-/Camera_Ready_Revision","ICML.cc/2026/Workshop/AdaptFM/-/Edit"],"mdate":1780570443231,"odate":1780353776095,"domain":"ICML.cc/2026/Workshop/AdaptFM","id":"nu3GPfkyJV","version":2},{"content":{"decision":{"value":"Accept"},"comment":{"value":"This paper describes an efficiency-improving method for text-to-SQL processing in which a router decides whether a question can be answered by a lightweight model or requires a more expensive LLM call. Experiments, including additional results provided in response to reviewer concerns, demonstrate the effectiveness of the approach. Analysis addresses key questions about trade-offs in different variants of the approach.\n\nOverall, this is a novel approach with positive results. The key reviewer questions have been addressed in the discussion period and adding the new results to the paper will strengthen it significantly.\n\nSome particularly important points raised should be addressed in the final version of the paper:\n\n- Add the new results presented in the discussion period and discuss them.\n- Expand the discussion of other routing methods in section 2 to clarify where there is no direct comparison with RouteLLM (for example).\n- Discuss the reliance on schema linking in the limitations section.\n- Discuss other text-to-SQL methods that have some form of routing further, to distinguish this approach from the prior work."},"title":{"value":"Paper Decision"}},"parentInvitations":"colmweb.org/COLM/2025/Conference/-/Decision","nonreaders":[],"tmdate":1756169359522,"tcdate":1751942729285,"writers":["colmweb.org/COLM/2025/Conference","colmweb.org/COLM/2025/Conference/Program_Chairs"],"signatures":["colmweb.org/COLM/2025/Conference/Program_Chairs"],"forum":"8OqGNXKwo8","number":1,"license":"CC BY 4.0","cdate":1751942729285,"readers":["everyone"],"invitations":["colmweb.org/COLM/2025/Conference/Submission868/-/Decision","colmweb.org/COLM/2025/Conference/-/Edit"],"mdate":1756169359522,"domain":"colmweb.org/COLM/2025/Conference","replyto":"8OqGNXKwo8","id":"6kgjSOaIJF","forumContent":{"TLDR":{"value":"We propose EllieSQL, a complexity-aware routing framework for Text-to-SQL, enhancing cost-efficiency while maintaining performance by directing queries to suitable SQL generation methods."},"venue":{"value":"COLM 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the COLM Code of Ethics on https://colmweb.org/CoE.html"},"keywords":{"value":["Text-to-SQL","Routing","Cost-Efficiency","Large Language Model"]},"supplementary_material":{"value":"/attachment/5d463428cf69a22e085ad7c79d3aa77603ca0c36.zip"},"abstract":{"value":"Text-to-SQL automatically translates natural language queries to SQL, allowing non-technical users to retrieve data from databases without specialized SQL knowledge. Despite the success of advanced LLM-based Text-to-SQL approaches on leaderboards, their unsustainable computational costs—often overlooked—stand as the \"elephant in the room\" in current leaderboard-driven research, limiting their economic practicability for real-world deployment and widespread adoption. To tackle this, we exploratively propose EllieSQL, a complexity-aware routing framework that assigns queries to suitable SQL generation pipelines based on estimated complexity. We investigate multiple routers to direct simple queries to efficient approaches while reserving computationally intensive methods for complex cases. Drawing from economics, we introduce the Token Elasticity of Performance (TEP) metric, capturing cost-efficiency by quantifying the responsiveness of performance gains relative to token investment in SQL generation. Experiments show that compared to always using the most advanced methods in our study, EllieSQL with the Qwen2.5-0.5B-DPO router reduces token use by over 40% without compromising performance on Bird development set, achieving more than a 2× boost in TEP over non-routing approaches. This not only advances the pursuit of cost-efficient Text-to-SQL but also invites the community to weigh resource efficiency alongside performance, contributing to progress in sustainable Text-to-SQL. Our source code and model are available at https://elliesql.github.io/."},"_bibtex":{"value":"@inproceedings{\nzhu2025elliesql,\ntitle={Ellie{SQL}: Cost-Efficient Text-to-{SQL} with Complexity-Aware Routing},\nauthor={Yizhang Zhu and Runzhi JIANG and Boyan Li and Nan Tang and Yuyu Luo},\nbooktitle={Second Conference on Language Modeling},\nyear={2025},\nurl={https://openreview.net/forum?id=8OqGNXKwo8}\n}"},"title":{"value":"EllieSQL: Cost-Efficient Text-to-SQL with Complexity-Aware Routing"},"pdf":{"value":"/pdf/22cb7760daa673d5ebdefb89c491dfcf425d7196.pdf"},"venueid":{"value":"colmweb.org/COLM/2025/Conference"},"paperhash":{"value":"zhu|elliesql_costefficient_texttosql_with_complexityaware_routing"},"authorids":{"value":["~Yizhang_Zhu1","~Runzhi_JIANG1","~Boyan_Li2","~Nan_Tang3","~Yuyu_Luo1"]},"author_guide":{"value":"I certify that this submission complies with the submission instructions as described on https://colmweb.org/AuthorGuide.html"},"authors":{"value":["Yizhang Zhu","Runzhi JIANG","Boyan Li","Nan Tang","Yuyu Luo"]}},"version":2},{"content":{"comment":{"value":"Dear reviewers, thanks for your suggestions, we include several additional small-scale experiments in this rebuttal:\n\n- **Correctness experiment on MATH EM:**  \nThis provides an objective correctness comparison between the original model and SOMA, complementing the qualitative and similarity-based metrics.\n\n| Model Family | Original (EM %) | Surrogate       | History-Prefix     | History-FT         | RouteLLM          | SOMA              |\n|--------------|------------------|-----------------|--------------------|--------------------|--------------------|-------------------|\n| **LLaMA**    | 68.0 ± 0.3       | 14.6 ± 0.8      | 28.9 ± 0.9         | 43.7 ± 1.0         | 49.8 ± 0.7         | 55.1 ± 0.6        |\n| **Qwen**     | 62.0 ± 0.4       | 14.0 ± 0.7      | 27.1 ± 0.9         | 39.9 ± 1.0         | 45.3 ± 0.7         | 50.4 ± 0.6        |\n- **Within-family experiment (e.g., LLaMA-3.1-70B → LLaMA-3.1-8B)**  \n\n| Method         | ShareGPT           | ReMeDi             | Craigslist         | Multi-Char         | MATH               | MT-Bench          | Avg                |\n|----------------|--------------------|---------------------|---------------------|---------------------|---------------------|--------------------|--------------------|\n| Surrogate      | 88.7 ± 2.13        | 86.2 ± 2.27         | 82.1 ± 2.41         | 83.4 ± 2.18         | 79.1 ± 2.63         | 85.3 ± 2.12        | 84.1 ± 2.29        |\n| History-Prefix | 91.8 ± 1.77        | 89.4 ± 1.92         | 86.0 ± 1.81         | 87.9 ± 1.73         | 83.5 ± 1.97         | 89.1 ± 1.74        | 87.9 ± 1.82        |\n| History-FT     | 94.2 ± 1.53        | 92.1 ± 1.58         | 89.3 ± 1.49         | 90.8 ± 1.38         | 87.9 ± 1.62         | 92.4 ± 1.41        | 91.1 ± 1.52        |\n| LLMLingua-2    | 90.6 ± 1.88        | 88.0 ± 1.83         | 85.1 ± 1.91         | 86.1 ± 1.76         | 82.2 ± 2.04         | 88.0 ± 1.87        | 86.7 ± 1.88        |\n| RouteLLM       | 95.8 ± 1.22        | 94.0 ± 1.17         | 91.9 ± 1.29         | 92.0 ± 1.09         | 89.9 ± 1.33         | 93.8 ± 1.14        | 92.9 ± 1.20        |\n| SOMA           | **97.1 ± 0.94**    | **95.3 ± 0.89**     | **93.4 ± 1.07**     | **93.7 ± 0.83**     | **91.8 ± 0.97**     | **95.1 ± 0.91**    | **94.4 ± 0.93**    |\n\n\nWe present the results on three datasets due to time constraints. Because LLaMA-3.1-8B is MUCH stronger than LLaMA-2-7B, the surrogate similarity rises substantially, and all baselines shift upward accordingly. However, SOMA still remain the strongest method.\n\n- **More baselines results**  \nWe additionally compare SOMA with speculative decoding and FrugalGPT cascades. Speculative decoding only verifies token-level consistency, leaving higher-level semantic errors uncorrected, while cascades may prematurely trust the small model and fail to escalate to the large model when needed. These mechanisms expose the inherent behavioral mismatch between small and large models, leading to lower similarity across all datasets. In contrast, SOMA explicitly reduces this mismatch by adapting the surrogate to the large model’s local conversational manifold, yielding consistently higher alignment with comparable efficiency.\n\n| Method          | ShareGPT (%)       | ReMeDi (%)         | Craigslist (%)     | Multi-Char (%)     | MATH (%)           | MT-Bench (%)       | Avg (%)             |\n|-----------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|----------------------|\n| Spec-Decoding| 89.7 ± 1.60       | 88.3 ± 1.75         | 82.6 ± 2.10         | 83.1 ± 1.80         | 78.8 ± 2.50         | 87.9 ± 1.46         | 85.1 ± 2.20          |\n| FrugalGPT | 92.0 ± 1.55       | 89.4 ± 1.68         | 84.1 ± 2.00         | 85.5 ± 1.75         | 80.2  ± 2.40         | 89.0 ± 1.33         | 86.7 ± 2.12          |\n| **SOMA**        | **96.4 ± 1.91**     | **93.2 ± 0.98**     | **91.9 ± 2.49**     | **92.3 ± 1.05**     | **90.7 ± 1.12**     | **94.1 ± 0.91**     | **93.1 ± 1.99**      |\n\n- **Empirical variance concentration and theoretical upper bound**  \nWe also conduct new experiments in Appendix E.6 to empirically confirming that early turns provide the most information and that SOMA’s switching rule operates within the regime assumed by Lemma 1 and Theorem 2."},"title":{"value":"Global Responses with Additional Experiment Results"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763925037683,"tcdate":1763925037683,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission22291/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission22291/Authors"],"forum":"B821Pc5huD","number":1,"license":"CC BY 4.0","cdate":1763925037683,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission22291/-/Official_Comment"],"mdate":1763925037683,"domain":"ICLR.cc/2026/Conference","replyto":"B821Pc5huD","id":"ij7oj4ybOi","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Small Language Model","multi-turn conversations","local manifold approximation"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Large Language Models (LLMs) are increasingly deployed in multi-turn dialogue settings where preserving conversational context across turns is essential. A standard serving practice concatenates the full dialogue history at every turn, which reliably maintains coherence but incurs substantial cost in latency, memory, and API expenditure, especially when queries are routed to large proprietary models. Existing approaches often struggle to balance the trade-off between response quality and efficiency. We propose a framework that exploits the early turns of a session to estimate a local response manifold and then adapt a smaller surrogate model to this local region for the remainder of the conversation. Concretely, we learn soft prompts that maximize semantic divergence between the large and surrogate small language models' responses to surface least-aligned local directions, stabilize training with anti-degeneration control, and distill the mined cases into localized LoRA fine-tuning so the surrogate runs without prompts at inference. A simple gate enables a one-time switch with rollback on drift. We further provide a theoretical analysis for key components in SOMA. Extensive experiments show the effectiveness of SOMA. The source code is provided at: https://anonymous.4open.science/r/SOMA-D377."},"_bibtex":{"value":"@misc{\ncheng2026soma,\ntitle={{SOMA}: Efficient Multi-turn {LLM} Serving via Small Language Model},\nauthor={Xueqi Cheng and Qiong Wu and Zhengyi Zhou and Xugui Zhou and Tyler Derr and Yushun Dong},\nyear={2026},\nurl={https://openreview.net/forum?id=B821Pc5huD}\n}"},"title":{"value":"SOMA: Efficient Multi-turn LLM Serving via Small Language Model"},"pdf":{"value":"/pdf/b48d6bf40b465bca46fe81bea8253ea534b19776.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"cheng|soma_efficient_multiturn_llm_serving_via_small_language_model"},"authorids":{"value":["~Xueqi_Cheng2","~Qiong_Wu3","~Zhengyi_Zhou1","~Xugui_Zhou1","~Tyler_Derr1","~Yushun_Dong1"]},"authors":{"value":["Xueqi Cheng","Qiong Wu","Zhengyi Zhou","Xugui Zhou","Tyler Derr","Yushun Dong"]}},"version":2},{"content":{"summary":{"value":"This paper introduces **DiSRouter (Distributed Self-Router)**, a framework that replaces centralized query routing in LLM systems with a distributed paradigm where each agent autonomously decides to answer or route queries based on self-awareness. Key contributions include:\n\n- A **distributed routing architecture** that enhances modularity and scalability compared to centralized routers.\n- A **Self-Awareness Training pipeline** with supervised fine-tuning (SFT) and reinforcement learning (RL) using a localized reward function, enabling independent agent training.\n- **Scenario adaptability** via a preference factor $\\alpha$ to balance performance and cost dynamically. Experiments on datasets like GSM8K and MMLU show DiSRouter outperforms baselines (e.g., RouteLLM, FrugalGPT) in utility. The core idea is that intrinsic self-assessment is more effective than external evaluation."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"1. I am still confused that how the the local routing policy $\\pi_i$ of agent $m_i$ is trained, it seems that $m_i$ only output \"I don't know!\" if choosing reject, how does $m_i$ select the next agent?\n2. Can DiSRouter's self-awareness training be applied to LLMs with different architectures (e.g., encoder-decoder models)? Please provide results with at least one non-Qwen model family to validate broad applicability.\n3. How does DiSRouter perform with large-scale agent pools (e.g., >10 agents)? \n4. What are the practical limitations (e.g., latency, fault tolerance) in deploying DiSRouter? Case studies or simulations with real-world constraints would be valuable."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"- **Originality**: The shift from centralized to distributed routing is novel, creatively integrating self-assessment concepts (e.g., LLM uncertainty) into a routing framework. The scenario-adaptive reward function is an innovative touch.\n- **Quality**: Experimental design is rigorous within its scope, with utility metrics and modularity tests. The pipeline is methodically described.\n- **Clarity**: The problem formulation in Section 2 is precise. Writing is concise.\n- **Significance**: The \"plug-and-play\" modularity addresses a key pain point in LLM systems, though real-world impact depends on scalability beyond the tested settings."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- **Limited Generalizability**: Experiments use only the Qwen2.5-Instruct series, neglecting other LLM families (e.g. Llama). This casts doubt on DiSRouter's applicability to diverse models. The authors should include cross-architecture validation.\n- **Training Efficiency Concerns**: The Self-Awareness Training (SFT + RL) is computationally heavy, but the paper dismisses routing cost as \"negligible\" without quantifying training overhead. A cost-benefit analysis is missing.\n- **Limited Robustness Testing**: There is no evaluation under adversarial conditions (e.g., ambiguous queries) to test the robustness of self-assessment. This is critical for real-world reliability."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762919711527,"tcdate":1761903816475,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7635/Reviewer_xhvr"],"signatures":["ICLR.cc/2026/Conference/Submission7635/Reviewer_xhvr"],"forum":"KDcwXKr0NU","number":4,"license":"CC BY 4.0","cdate":1761903816475,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7635/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762919711527,"domain":"ICLR.cc/2026/Conference","replyto":"KDcwXKr0NU","id":"XKc4HTHIu8","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"TLDR":{"value":"We introduce DiSRouter, a distributed system where LLMs leverage \"self-awareness\" to route queries among themselves, outperforming conventional centralized routers."},"keywords":{"value":["query routing","model selection","distributed system","self-awareness of LLM"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"The proliferation of Large Language Models (LLMs) has created a diverse ecosystem of models with highly varying performance and costs, necessitating effective query routing to balance performance and expense. Current routing systems often rely on a centralized external router trained on a fixed set of LLMs, making them inflexible and prone to poor performance since the small router can not fully understand the knowledge boundaries of different LLMs. We introduce DiSRouter (Distributed Self-Router), a novel paradigm that shifts from centralized control to distributed routing. In DiSRouter, a query traverses a network of LLM agents, each independently deciding whether to answer or route to other agents based on its own self-awareness—its ability to judge its competence. This distributed design offers superior flexibility, scalability, and generalizability. To enable this, we propose a two-stage Self-Awareness Training pipeline that enhances each LLM's self-awareness. Extensive experiments demonstrate that DiSRouter significantly outperforms existing routing methods in utility across various scenarios, effectively distinguishes between easy and hard queries, and shows strong generalization to out-of-domain tasks. Our work validates that leveraging an LLM's intrinsic self-awareness is more effective than external assessment, paving the way for more modular and efficient multi-agent systems."},"_bibtex":{"value":"@inproceedings{\nzheng2026disrouter,\ntitle={Di{SR}outer: Distributed Self-Routing for {LLM} Selections},\nauthor={Hang Zheng and Hongshen Xu and Yongkai.lin and Shuai Fan and Lu Chen and Kai Yu},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=KDcwXKr0NU}\n}"},"title":{"value":"DiSRouter: Distributed Self-Routing for LLM Selections"},"pdf":{"value":"/pdf/0190f98d94890b700d03febdee678e22a2a5e380.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"zheng|disrouter_distributed_selfrouting_for_llm_selections"},"authorids":{"value":["~Hang_Zheng3","~Hongshen_Xu1","~Yongkai.lin1","~Shuai_Fan1","~Lu_Chen3","~Kai_Yu3"]},"authors":{"value":["Hang Zheng","Hongshen Xu","Yongkai.lin","Shuai Fan","Lu Chen","Kai Yu"]}},"version":2},{"content":{"summary":{"value":"This paper introduces a LLM model routing system called Routoo that selects what LLM to route queries to from a pool of LLMs of various capabilities and inference costs. For a particular query, and a particular LLM, a predictor model (based on Mistral7b) predicts the accuracy score, and an average cost heuristic estimates cost. Given a list of queries and a total budget, an optimization heuristic assigns the appropriate LLM for each query to achieve the highest overall accuracy within the total budget. On the MMLU benchmark, the Routoo system achieves higher accuracy at lower costs than using any single state-of-the-art model. Higher accuracy is achieved since the pool of models have specialists for particular type of queries and lower cost is achieved since easier queries don't need more expensive models to solve them."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"1. The Mistral7b based prediction has an inference cost. Is that included in the inference cost?\n2. Is a LLM needed for prediction or would a lightweight retrieval system that retrieves scores on similar queries from the training data work well?\n3. In Section 3.3, step 3 of the algorithm (budget management) mentions that the next best model is chosen if the budget is exceeded. However, the budget is the total budget across all queries so how is the ordering of this step performed across queries? Wouldn't the earlier queries starve the budget of the later queries, leading them to choose worse models?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. This work introduces heuristics for optimizing for multiple queries with a total budget which is novel.\n2. This Routoo system has great results where it Pareto-dominates state-of-the-art single LLMs on inference cost and MMLU performance.\n3. The gathering of non-MMLU dataset for training along with synthetic data is meticulous in avoiding data leakage during evaluation.\n4. The approach is sound: the predictive model for score prediction and budget constrained selector heuristic makes sense. The per-LLM embedding allows for the model to be inferenced only once-per-query instead of once-per-query-per-LLM which saves compute cost."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"While the approach in this work is sound, the contribution of the work over previous works may not be significant enough. Furthermore, the work is more appropriate for machine learning systems without a significant learning aspect.\n1. There are related works regarding predictive routing with multiple-LLMs that are not discussed [3,4]\n2. The advantages of specialist models improving scores over strong single-LLM models are covered in [1,2] so that aspect of the system is not very novel.\n3. The concepts of budget conscious LLM routing is covered in [3,4] so that aspect of the system is not very novel either.\n4. The only learning aspect of this work is the predictive model (Mistral7b with learned classification head). Without accuracy measurements, and comparisons with other models, there is little insight to be gained.\n\n[1] Large Language Model Routing with Benchmark Datasets\n\n[2] Harnessing the Power of Multiple Minds: Lessons Learned from LLM Routing\n\n[3] Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing\n\n[4] RouteLLM: Learning to Route LLMs with Preference Data"}},"nonreaders":[],"tmdate":1733101537908,"tcdate":1730507790715,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission11514/Reviewer_o5Db"],"signatures":["ICLR.cc/2025/Conference/Submission11514/Reviewer_o5Db"],"forum":"RQ9fQLEajC","number":1,"license":"CC BY 4.0","cdate":1730507790715,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission11514/-/Official_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1733101537908,"domain":"ICLR.cc/2025/Conference","replyto":"RQ9fQLEajC","id":"tY42Aysv6r","forumContent":{"TLDR":{"value":"An LLM-based Router trained to route to the most effective LLM optimizing performance, cost, and efficiency, creating a new state-of-the-art model by integrating knowledge of underlying models."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["LLM","efficiency","knowledge integration","generation"]},"supplementary_material":{"value":"/attachment/2e84b1f1c381960311d11bdd23f4d081c4e4d14a.zip"},"primary_area":{"value":"generative models"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"LLMs with superior response quality—particularly larger or closed-source models—often come with higher inference costs, making their deployment inefficient and costly. Meanwhile, developing foundational LLMs from scratch is becoming increasingly resource-intensive and impractical for many applications. To address the challenge of balancing quality and cost, we introduce Routoo, an architecture designed to optimize the selection of LLMs for specific prompts based on performance, cost, and efficiency. Routoo provides controllability over the trade-off between inference cost and quality, enabling significant reductions in inference costs for a given quality requirement.\nRoutoo comprises two key components: a performance predictor and cost-aware selector. The performance predictor is a lightweight LLM that estimates the expected performance of various underlying LLMs on a given prompt without executing them. The cost-aware selector module then selects the most suitable model based on these predictions and constraints such as cost and latency, significantly reducing inference costs for the same quality. \nWe evaluated Routoo using the MMLU benchmark across 57 domains employing open-source models. Our results show that Routoo matches the performance of the Mixtral 8x7b model while reducing inference costs by one-third. Additionally, by allowing increased costs, Routoo surpasses Mixtral's accuracy by over 5\\% at equivalent costs, achieving an accuracy of 75.9\\%. When integrating GPT4 into our model pool, Routoo nearly matches GPT4's performance at half the cost and exceeds it with a 25\\% cost reduction. \nThese outcomes highlight Routoo's potential to significantly reduce inference costs without compromising quality, and even to establish new state-of-the-art results by leveraging the collective capabilities of multiple LLMs."},"_bibtex":{"value":"@misc{\nmohammadshahi2025routoo,\ntitle={Routoo: Learning to Route to Large Language Models Effectively},\nauthor={Alireza Mohammadshahi and Arshad Rafiq Shaikh and Majid Yazdani},\nyear={2025},\nurl={https://openreview.net/forum?id=RQ9fQLEajC}\n}"},"title":{"value":"Routoo: Learning to Route to Large Language Models Effectively"},"pdf":{"value":"/pdf/454cdce2a12d959ded0bc50a6c5b05c494a2d4dd.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"mohammadshahi|routoo_learning_to_route_to_large_language_models_effectively"},"authorids":{"value":["~Alireza_Mohammadshahi1","~Arshad_Rafiq_Shaikh1","~Majid_Yazdani1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Alireza Mohammadshahi","Arshad Rafiq Shaikh","Majid Yazdani"]}},"version":2},{"content":{"venue":{"value":"Crossref"},"pdf":{"value":"https://link.springer.com/content/pdf/10.1007/978-3-032-37676-3_35.pdf"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"wang|uncertaintyaware_sequential_decision_rules_for_eventtriggered_llm_invocation_in_streaming_systems"},"html":{"value":"https://doi.org/10.1007/978-3-032-37676-3_35"},"abstract":{"value":"Streaming inference pipelines increasingly pair lightweight “fast models” with Large Language Models (LLMs) that provide rich semantic understanding at substantial cost. The central question of when to invoke the LLM has received limited formal treatment. We cast this as a risk-based sequential stopping problem, where a trigger policy \\(\\pi \\) fires when a risk functional \\(R(\\mathcal {H}_t)\\) exceeds a threshold \\(\\theta \\). Within this framework we prove six results: a minimum inter-event time bound excluding trigger chattering; optimality of threshold policies via smooth pasting; approximate SPRT guarantees under estimated parameters; \\(O(\\sqrt{T\\log T})\\) regret for stationary streams (extending to \\(O(\\sqrt{(C_T+1)\\,T\\log T})\\) under \\(C_T\\) changepoints); \\(O(1/\\sqrt{T})\\) convergence of online gradient descent for adaptive thresholds; and a calibration-to-miss-rate transfer inequality. Several classical trigger families, including event-triggered, optimal stopping, SPRT, CUSUM, and Bayesian triggers, can be expressed as special cases of this framework. On turbofan degradation data (CMAPSS) with real LLM calls, we empirically verify the theoretical assumptions, ablate the risk function design, compare against six baselines including a RouteLLM-style router and contextual bandits, and analyze cost sensitivity and LLM failure modes. The results confirm sublinear regret (\\(\\alpha {<}1\\) for all principled triggers), high diagnostic quality (92.9% of 1,600 LLM diagnoses reach grounding score \\({\\ge }0.75\\) under our rubric), and that anomaly-score-driven risk functions dominate alternatives by roughly an order of magnitude on the Pareto AUC."},"title":{"value":"Uncertainty-Aware Sequential Decision Rules for Event-Triggered LLM Invocation in Streaming Systems"},"authors":{"value":[{"fullname":"Zhaohui Wang","username":"~Zhaohui_Wang8"}]}},"tmdate":1790665451327,"pdate":1798761600000,"externalIds":["doi:10.1007/978-3-032-37676-3_35"],"tcdate":1790665448313,"writers":["~","OpenReview.net/Public_Article/ORCID.org","OpenReview.net/Support"],"signatures":["~Zhaohui_Geoffrey_Wang1"],"forum":"CdwsqivZni","license":"CC BY-SA 4.0","number":113054,"cdate":1788728055070,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/ORCID.org/-/Record","OpenReview.net/Public_Article/-/Edit"],"mdate":1790665451327,"domain":"OpenReview.net/Public_Article","id":"CdwsqivZni","version":2},{"content":{"comment":{"value":"We thank the reviewer for the follow-up questions and are glad our clarifications resolved the question on Corollary 2.2. We address the remaining concerns below.\n\n**Q1: SLM zero-cost.** SLM-LLM collaboration approaches, such as routing, cascading, and speculative decoding, are built on the premise that SLM inference costs are substantially lower than LLM inference costs. The larger this cost differential, the greater the gains in these approaches. Accordingly, the zero-cost SLM setting is widely adopted in the SLM-LLM collaboration literature:\n\n1. Hybrid LLM [1] measures cost as the fraction of queries routed to the small model, with no cost attributed to SLM inference.\n2. RouteLLM [2] measures cost as the percentage of calls to the strong model, excluding the weak model's cost entirely.\n3. ABC [3] acknowledges SLM ensemble cost but excludes it from their edge-to-cloud cost accounting.\n\nEven when SLMs are accessed via paid APIs, the cost gap is typically substantial, as illustrated by representative providers below:\n\n| Provider | SLM| Input ($/1M) | Output ($/1M) | LLM| Input ($/1M) | Output ($/1M) |\n|-|-|-|-|-|-|-|\n| Artificial Analysis| Llama-3.2-3B | 0.08 | 0.08| Llama-3.3-70B | 0.58 | 0.71 |\n| Together AI | Gemma-3n-E4B | 0.02 | 0.04 | Llama-3.3-70B | 0.88 | 0.88 |\n| Groq | Llama-3.1-8B | 0.05 | 0.08 | Llama-3.3-70B | 0.59 | 0.79 |\n\nThe SLM is 6-30× cheaper per token than the LLM and, in some cases, entirely free, reinforcing the zero-cost assumption as reasonable.\n\nFinally, we agree that nonzero SLM costs reduce the cost advantage of all cascading-based methods (including Reactive Shepherding, ABC, and FrugalGPT), as they invoke the SLM multiple times before escalation.This is not unique to the above approaches; it is inherent to all cascading-based methods. However, Proactive Shepherding is minimally affected: it makes the hint decision before SLM inference, so the SLM runs only once per query — the same as routing baselines — largely preserving its cost advantage.\n\nWe will include a sensitivity analysis with nonzero SLM costs in the revised paper and report updated Tables 1-4 to transparently show how each method's advantage changes.\n\n**Q2.** *Sensitivity to hyperparameter choices.* Our method is not brittle to hyperparameter variation. The Pareto front curves demonstrate the performance of Reactive Shepherding across a wide range of hyperparameter configurations ($\\eta_{\\\\text{hint}}$, K, $\\alpha$). Across all four datasets, Reactive Shepherding consistently achieves favorable cost-accuracy trade-offs over a broad region of the hyperparameter space, rather than at a single narrow operating point.\n\n*Generality across tasks*. We acknowledge that Table 5 reports different hyperparameter values across datasets, but highlight three points.  First, this is standard practice across all baselines in our comparison: RouteLLM requires tuning its routing threshold, FrugalGPT requires tuning its confidence threshold, and GraphRouter switches between modes, as shown in Table 5. Our approach is no more demanding in this regard. Second, each hyperparameter serves an intuitive and distinct role: $\\\\alpha$ controls the cost-accuracy trade-off, $\\\\eta_{\\\\text{hint}}$ determines when a query is too complex for hinting and should go directly to the LLM, and K governs the confidence level of the cascading decision. Third, and most importantly, our model itself generalizes across tasks without retraining. For example, the HumanEval and MBPP results in Tables 3-4 are obtained using the model trained solely on GSM8K. This demonstrates that the learned hint allocation captures task-general patterns of query difficulty, and only lightweight hyperparameter calibration (not model retraining) is needed for new tasks. \n\n**Q3: Reactive Shepherding exceeding LLM-only baseline on HumanEval and MBPP.** We thank the reviewer for this keen observation. This occurs because Reactive Shepherding's cascading mechanism introduces an additional source of correct answers that the LLM-only baseline lacks. When the SLM achieves majority agreement across three samples (N=2 per Table 5), its consensus output is used directly without invoking the LLM. On code generation tasks, the SLM's majority-voted solution is sometimes correct on queries where the LLM's single-pass output fails to compile or contains bugs. The system thus exceeds the LLM-only baseline by combining two complementary strengths: SLM consensus for queries where agreement signals reliability, and LLM hints for harder queries. We observed the same phenomenon with ABC, which also uses SLM agreement as a gating mechanism. We will add this explanation when we add these new figures to the revised paper.\n\n[1]Ding, Dujian, et al. \"Hybrid llm: Cost-efficient and quality-aware query routing.\" ICLR (2024)\n\n[2]Ong, Isaac, et al. \"Routellm: Learning to route llms with preference data.\" ICLR (2025)\n\n[3]Kolawole, Steven, et al. \"Agreement-based cascading for efficient inference.\" TMLR(2025)"}},"parentInvitations":"ICML.cc/2026/Conference/-/Reply_Rebuttal_Comment","tmdate":1782369955893,"tcdate":1775325279925,"writers":["ICML.cc/2026/Conference","ICML.cc/2026/Conference/Submission30175/Authors"],"signatures":["ICML.cc/2026/Conference/Submission30175/Authors"],"forum":"H8rywbFXuM","number":1,"license":"CC BY 4.0","cdate":1775325279925,"readers":["everyone"],"invitations":["ICML.cc/2026/Conference/Submission30175/Official_Review4/Rebuttal1/Rebuttal_Acknowledgement1/-/Reply_Rebuttal_Comment","ICML.cc/2026/Conference/-/Edit"],"mdate":1782369955893,"domain":"ICML.cc/2026/Conference","replyto":"7YEc5IoaxK","id":"7vAW1alhmG","forumContent":{"venue":{"value":"Submitted to ICML 2026"},"keywords":{"value":["LLM routing; LLM cascading; SLM-LLM collaboration; Cost-efficient inference"]},"_bibtex":{"value":"@misc{\nanonymous2026pay,\ntitle={Pay for Hints, Not Answers: {LLM} Shepherding for Cost-Efficient Inference},\nauthor={Anonymous},\nyear={2026},\nurl={https://openreview.net/forum?id=H8rywbFXuM}\n}"},"title":{"value":"Pay for Hints, Not Answers: LLM Shepherding for Cost-Efficient Inference"},"paperhash":{"value":"dong|pay_for_hints_not_answers_llm_shepherding_for_costefficient_inference"},"primary_area":{"value":"deep_learning->large_language_models"},"abstract":{"value":"Large Language Models (LLMs) deliver state-of-the-art performance on complex reasoning tasks, but their inference costs limit deployment at scale. Small Language Models (SLMs) offer dramatic cost savings yet lag substantially in accuracy. Existing approaches—routing and cascading—treat the LLM as an all-or-nothing resource: either the query bypasses the LLM entirely, or the LLM generates a complete response at full cost. We introduce **LLM Shepherding**, a framework that requests only a short prefix (a hint) from the LLM and provides it to SLM. This simple mechanism is surprisingly effective for math and coding tasks: even hints comprising 10--30\\% of the full LLM response improve SLM accuracy significantly. Shepherding generalizes both routing and cascading, and it achieves lower cost under oracle decision-making. We develop a two-stage predictor that jointly determines whether a hint is needed and how many tokens to request. On the widely-used mathematical reasoning (GSM8K, CNK12) and code generation (HumanEval, MBPP) benchmarks, Shepherding reduces costs by 42--94\\% relative to LLM-only inference. Compared to state-of-the-art routing and cascading baselines, shepherding delivers up to 2.8$\\times$ cost reduction while matching accuracy. To our knowledge, this is the first work to exploit token-level budget control for SLM-LLM collaboration. Code is available at https://anonymous.4open.science/r/LLM_Sheperding-77C3/."},"pdf":{"value":"/pdf/b5f1805f8847279424d1a89d1e99c69e9d93c924.pdf"},"venueid":{"value":"ICML.cc/2026/Conference/Rejected_Submission"},"authorids":{"value":["~Ziming_Dong1","~Hardik_Sharma3","~Evan_O'Toole1","~Jaya_Prakash_Champati1","~Kui_Wu1"]},"authors":{"value":["Ziming Dong","Hardik Sharma","Evan O'Toole","Jaya Prakash Champati","Kui Wu"]}},"version":2},{"content":{"summary":{"value":"This paper introduces a unified framework for routing among multiple LLMs and decoding strategies under both continual and non-continual settings. The main idea is to train modular predictors that estimate, for each model–decoding pair, its expected accuracy and computational cost on a given task. These predictors combine general text embeddings (from an off-the-shelf embedding model) with task-specific representations to capture both semantic and domain-specific information. Routing decisions are then formulated as an optimization problem that can be solved efficiently. Experimental results show that the proposed framework outperforms existing baselines in terms of both accuracy and efficiency."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"See the weakness above."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"- The paper addresses an important and practical problem. Effectively routing among multiple LLMs and decoding strategies under varying cost and performance constraints has become an increasingly relevant and widely discussed topic. This topic is relevant to the growing need for efficient deployment of large reasoning models, especially in settings where computational budgets and task distributions evolve over time. The formulation of both continual and non-continual routing scenarios adds further practical value to the work.\n\n- The proposed framework demonstrates consistent improvements over existing routing baselines across several reasoning and question-answering benchmarks. The experimental results suggest that the modular predictor design and optimization-based decision process lead to better accuracy–efficiency trade-offs, validating the effectiveness of the proposed approach in diverse routing settings."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- Although the paper repeatedly discusses routing in both continual and non-continual settings, the problem itself is never formally defined. It remains unclear what exactly constitutes a routing input and output, and how the optimization objective is formulated in mathematical terms. The framework is presented primarily at a conceptual level, which makes it difficult to precisely understand the problem scope and to reproduce the approach in other contexts. As a result, the paper may not be self-contained for readers who are not already familiar with prior routing works such as RouteLLM or RTR, which substantially limits accessibility and clarity.\n\n- The overall framework follows a structure that is already common in related areas such as task planning and model selection. The paper mainly adapts existing ideas to the LLM routing setting, without introducing a fundamentally new algorithmic or theoretical insight. \n\n- The paper lacks a systematic ablation study quantifying the contribution of each component in the proposed framework. Is there any ablation study or experimental analysis addressing this? For example, how would the performance change if only the general-purpose representation or the task-specific representation were used?"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762924865611,"tcdate":1761725804412,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission14463/Reviewer_dMBg"],"signatures":["ICLR.cc/2026/Conference/Submission14463/Reviewer_dMBg"],"forum":"gCUY6QIv8r","number":2,"license":"CC BY 4.0","cdate":1761725804412,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission14463/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762924865611,"domain":"ICLR.cc/2026/Conference","replyto":"gCUY6QIv8r","id":"CBGEtBOL6F","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["model routing","continual routing","constrained routing","unconstrained routing"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"AI tasks differ in complexity and are best addressed with different computation strategies (e.g., combinations of models and decoding methods). Hence, an effective routing system that maps tasks to the appropriate strategies is crucial.\nMost prior methods build the routing framework by training a *single* model across *all* strategies, which demands full retraining whenever new strategies appear and leads to high overhead. Attempts at such continual routing, however, often face difficulties with generalization.\nPrior models also typically use a *single* input representation, limiting their ability to capture the full complexity of the routing problem and leading to sub-optimal routing decisions.\nTo address these gaps, we propose CONCUR, a **con**tinual routing framework that supports both **c**onstrained and **u**nconstrained **r**outing (i.e., routing with or without a budget).\nOur *modular* design trains a separate predictor model for each strategy, enabling seamless incorporation of new strategies with low additional training cost.\nOur predictors also leverage *multiple* representations of both tasks and computation strategies to better capture overall problem complexity.\nExperiments on both in-distribution and out-of-distribution, knowledge- and reasoning-intensive tasks show that our method outperforms the best single strategy and strong existing routing techniques with higher end-to-end accuracy and lower inference cost in both continual and non-continual settings, while also reducing training cost in the continual setting."},"_bibtex":{"value":"@inproceedings{\nchen2026concur,\ntitle={{CONCUR}: A Framework for Continual Constrained and Unconstrained Routing},\nauthor={Peter Baile Chen and Weiyue Li and Dan Roth and Mike Cafarella and Samuel Madden and Jacob Andreas},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=gCUY6QIv8r}\n}"},"title":{"value":"CONCUR: A Framework for Continual Constrained and Unconstrained Routing"},"pdf":{"value":"/pdf/b59e1bbe2f0d9d26266b4e43191885655a9464f7.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"chen|concur_a_framework_for_continual_constrained_and_unconstrained_routing"},"authorids":{"value":["~Peter_Baile_Chen1","~Weiyue_Li1","~Dan_Roth3","~Mike_Cafarella1","~Samuel_Madden1","~Jacob_Andreas1"]},"authors":{"value":["Peter Baile Chen","Weiyue Li","Dan Roth","Mike Cafarella","Samuel Madden","Jacob Andreas"]}},"version":2},{"content":{"comment":{"value":"We appreciate the reviewers’ efforts in evaluating our paper. Below, we summarize the key points reviewers raised—items marked with \\*\\* indicate issues for which we provide additional experiments or clarifications, while unmarked items reflect strengths acknowledged by them. \"Action/Summary\" includes the highly summarized rebuttal content for each reviewer.\n\n| Dimension | Reviewer qr28 | Reviewer s59p | Reviewer L1od | Reviewer 8ann | Action/Summary |\n| :--- | :--- | :--- | :--- | :--- | :--- |\n| **Novelty & Design** | \"Heterogeneous graph construction... is convincing\" **\"Novelty over GraphRouter marginal..\"** | \"Practical, resource-efficient design\" **\"Key challenge unclear.. modules standard\"** | \"Captures complex relational dependencies\" **\"Personalization vs. Routing conceptual..\"** | \"Explicit node types... is a novel approach\" **\"Design is empirical.. lacks theoretical justification\"** | **Summary:** Reviewers generally find the graph design novel and practical. **Reviewer qr28 & s59p rebuttal:** We clarify the unique challenge of extracting preferences from scarce/noisy history and how GMTRouter differs from GraphRouter. **Reviewer 8ann rebuttal:** We added experiments to verify GMTRouter is a general framework effective across scenarios, rather than being designed for a specific GNN algorithm. |\n| **Evaluation Scope** | **\"Synthetic datasets quality.. multi-turn settings..\"** | **\"Missing Graph baselines (KGT)\"** **\"Missing Routed LLM (RouteLLM)\"** | **\"Comparison with memory-based methods (MA-GNN)..\"** **\"Comparison with personalized LLM..\"** | **\"Graph representation vs. Raw interaction..\"** | **Summary:** We significantly expanded the evaluation scope. **Reviewer qr28 rebuttal:** We added the LaMP dataset (personalized generation). **Reviewer s59p rebuttal:** We added RouteLLM and MA-GNN as baselines. **Reviewer L1od rebuttal:** We added IPA-RAG (Personalized LLM) and MA-GNN. **Reviewer 8ann rebuttal:** We added TIGER to compare graph-based vs. raw interaction learning. |\n| **Scale, Bias & Cost** | **\"Scale of Chatbot Arena (11 users) is small..\"** **\"Validate on scarce/noisy data..\"** | \"Tests on one real-world and three synthetic benchmarks\" | **\"Lacks analysis of graph construction cost..\"** | \"Inductive training... addresses cold-start\" **\"Multi-user graph clarification..\"** | **Reviewer qr28 rebuttal:** We scaled Chatbot Arena experiments to 161 users and conducted noise robustness tests (up to 20% noise). **Reviewer L1od rebuttal:** We provided a detailed table on graph encoding and construction time. **Reviewer 8ann rebuttal:** We clarified that per-user subgraph sampling ensures user independence. |\n| **Performance** | \"Outperforms existing router baselines\" \"Efficient, minimal computing\" | \"Consistently outperforms baselines\" \"Generalizes well to unseen users\" | \"Consistently outperforms baselines\" \"Adapts to new users with minimal data\" | \"Consistent improvements over baselines\" \"Lightweight and practical\" | **Summary:** All reviewers acknowledge GMTRouter consistently outperforms baselines, is computationally efficient, and generalizes well to new users. |\n| **Presentation** | **\"Reduce the margin.. formatting..\"** | **\"Writing is a little bit confusing (selection)..\"** | \"Presentation: Good\" | **\"Duplicated references..\"** **\"AUC not defined\"** | **Summary:** Most reviewers find the presentation clear. **Reviewer s59p rebuttal:** We revised Section 3 to clarify the link between GNN construction and routing selection. **Reviewer 8ann rebuttal:** We fixed all reference duplications and defined AUC. |\n\nIn addition, we expand several minor discussions: such as the specific mechanisms of user-conditioned subgraph sampling, the synergistic effect of combining routing with personalized generation (GMTRouter + IPA-RAG), and detailed runtime analysis. These clarifications do not affect the core contribution of the paper and are fully addressed in the rebuttal."},"title":{"value":"General Response by Authors"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764822640138,"tcdate":1764822640138,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission14657/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission14657/Authors"],"forum":"oRe9tHpSpn","number":10,"license":"CC BY 4.0","cdate":1764822640138,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission14657/-/Official_Comment"],"mdate":1764822640138,"domain":"ICLR.cc/2026/Conference","replyto":"oRe9tHpSpn","id":"GTBCk9BoP5","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Heterogeneous GNN","LLM routing","Preference Learning"]},"primary_area":{"value":"learning on graphs and other geometries & topologies"},"abstract":{"value":"Large Language Model (LLM) routing has demonstrated strong capability in balancing response quality with computational cost. As users exhibit diverse preferences, personalization has attracted increasing attention in LLM routing, since even identical queries may require different models to generate responses tailored to individual needs. However, existing approaches are not fully personalized and often fail to faithfully capture the complex interactions between specific users and LLMs. Moreover, user preference data is typically scarce, noisy, and inconsistent in format, which limits the effectiveness of methods that rely solely on user-specific data. To address these challenges, we propose GMTRouter, which represents multi-turn user–LLM interactions as a heterogeneous graph with four node types: user, LLM, query, and response, thereby maximally preserving the rich relational structure of the interaction. Through a tailored message-passing mechanism, GMTRouter learns to capture user preferences from few-shot data within a lightweight inductive graph learning framework, enabling effective personalization. Extensive experiments demonstrate that GMTRouter consistently outperforms the strongest baselines, achieving 0.9%–21.6% higher accuracy and 0.006–0.309 higher AUC across multiple datasets. More importantly, we further demonstrate that GMTRouter can adapt to new users and evolving preferences using only few-shot data, without extensive fine-tuning."},"_bibtex":{"value":"@misc{\nxie2026gmtrouter,\ntitle={{GMTR}outer: Personalized {LLM} Router over Multi-turn User Interactions},\nauthor={Encheng Xie and Yihang Sun and Tao Feng and Jiaxuan You},\nyear={2026},\nurl={https://openreview.net/forum?id=oRe9tHpSpn}\n}"},"title":{"value":"GMTRouter: Personalized LLM Router over Multi-turn User Interactions"},"pdf":{"value":"/pdf/b4f227b8ee59ab4f2a14d8fc028c0e06e9455c44.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"xie|gmtrouter_personalized_llm_router_over_multiturn_user_interactions"},"authorids":{"value":["~Encheng_Xie1","~Yihang_Sun2","~Tao_Feng5","~Jiaxuan_You2"]},"authors":{"value":["Encheng Xie","Yihang Sun","Tao Feng","Jiaxuan You"]}},"version":2},{"content":{"comment":{"value":"Dear Area Chair,\n\nWe first appreciate your time and effort in handling our submission.\n\nAs the discussion period is coming to an end, we would like to provide some summarized context to ensure you have as much relevant information as possible when making the final decision.\n\n> ### **Summary of Our Main Contribution**\n\nExisting cascade-based serving systems fail to address three critical challenges: \n- The heterogeneous resource requirements of different LLMs.\n- The varying characteristics of incoming workloads. \n- The need for joint optimization between system deployment and routing strategies. \n \nCascadia addresses these challenges through a **bi-level optimization approach** that co-optimizes resource allocation, parallelism strategy, and routing strategy. This system-algorithm co-design achieves up to 4$\\times$ tighter latency SLOs and 5$\\times$ higher throughput compared to both single-model and state-of-the-art cascade serving systems while maintaining target answer quality.\n\n\n> ### **Regarding Paper Reviews**\n\n**Three reviewers (HDzY, jBdC, m6H1)** rated soundness, presentation, and contribution as **good** (scores of 3/3/3) and gave overall ratings of **6**. All of them acknowledged the **significance and novelty** of our work. Their primary suggestions centered on:\n\n- Additional experiments: E.g., prefix caching evaluation, scheduling optimality analysis, cost efficiency analysis, latency results under fluctuating workloads, and ablation studies with uniform thresholds.\n- Further clarifications: E.g., query complexity measurement, judging overhead, re-scheduling mechanism details.\n\nOur comprehensive responses—comprising several new experiments detailed in the rebuttal and Appendices F-N, along with multiple clarifications—systematically responded to all their concerns.\n\n**Reviewer 11UR** gave an overall rating of **4** and acknowledged the **importance** of our work. The reviewer raised concerns about judging overhead, time-to-first-token (TTFT) latency, and insufficient baseline comparisons. We fully responded to these concerns by:\n\n- Additional experiments: Benchmarking GPT-4o judging overhead, baseline comparisons with RouteLLM, and BERT-based judging experiments across realistic and oracle scenarios.\n- Further clarifications: Streaming-based implementation for TTFT.\n\nWe believe our comprehensive responses address the reviewer's concerns.\n\n\n\n> ### **Revisions in the Updated Manuscript**\n\nWe added the following changes to our updated manuscript based on the reviewers' valuable suggestions:\n\n*Additional Experiments:*\n- Prefix caching evaluation (Appendix M) – Reviewer HDzY;\n- Comparison with RouteLLM showing 21.3% lower SLO scale and 18.8% higher throughput (Section 4.2, Appendix I) – Reviewers 11UR, jBdC;\n- Sensitivity analysis with weaker judges (GPT-4o-mini, Llama3.1-70B) (Appendix K) – Reviewer m6H1\n- Fine-tuned BERT for cascading experiments (Appendix N) – Reviewer 11UR;\n- Scheduling optimality analysis against exhaustive search (Appendix J) – Reviewer jBdC;\n- Cost efficiency analysis showing 20-39% cost reduction vs. CascadeServe (Section 4.2, Appendix L) – Reviewer m6H1;\n- Latency results under fluctuating workloads (Section 4.4, Appendix G) – Reviewer HDzY;\n- GPT-4o judging overhead benchmarks (Appendix H) – Reviewer 11UR.\n\n*Clarifications:*\n- Query complexity measurement methodology (footnote on page 9) – Reviewer HDzY;\n- Judging overhead (footnote on page 6) – Reviewers 11UR, m6H1;\n- CascadeServe citation (Section 4.1) – Reviewer HDzY;\n- Re-scheduling mechanism details (Section 4.4, Appendix G) – Reviewers HDzY, m6H1.\n\nWe believe these revisions significantly strengthen the paper and comprehensively respond to the core concerns raised by the reviewers.\n\nBest regards,\nThe Authors of Submission 12568"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764668099403,"tcdate":1764667890510,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission12568/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission12568/Authors"],"forum":"wkrWbrHTJQ","number":9,"license":"CC BY 4.0","cdate":1764667890510,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission12568/-/Official_Comment"],"mdate":1764668099403,"domain":"ICLR.cc/2026/Conference","replyto":"wkrWbrHTJQ","id":"qxs9vZIbDr","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Distributed","Parallel","and Cluster Computing"]},"primary_area":{"value":"infrastructure, software libraries, hardware, systems, etc."},"abstract":{"value":"Recent advances in large language models (LLMs) have intensified the need to deliver both rapid responses and high-quality outputs. More powerful models yield better results but incur higher inference latency, whereas smaller models are faster yet less capable. Recent work proposes balancing this latency–quality trade-off using model cascades, which route simpler queries to smaller models and more complex ones to larger models. However, enabling efficient cascade serving remains challenging. Current frameworks lack effective mechanisms for handling (i) the huge and varying resource demands of different LLMs, (ii) the inherent heterogeneity of LLM workloads, and (iii) the co-optimization of system deployment and routing strategy.\n\nMotivated by these observations, we introduce Cascadia, a novel cascade serving framework designed explicitly to schedule request routing and deploy model cascades for fast, quality-preserving LLM serving. Cascadia employs a bi-level optimization method: at the deployment level, it uses a mixed-integer linear program to select resource allocations and parallelism strategies based on LLM information and workload characteristics; at the routing level, it applies a Chebyshev-guided method to iteratively co-optimize the routing strategy and the system deployment produced by the deployment level. Our extensive evaluation on diverse workload traces and different model cascades (DeepSeek and the Llama series) demonstrates that Cascadia significantly outperforms both single-model deployments and the state-of-the-art cascade serving baseline, achieving up to 4$\\times$ (2.3$\\times$ on average) tighter latency SLOs and up to 5$\\times$ (2.4$\\times$ on average) higher throughput while maintaining target answer quality."},"_bibtex":{"value":"@inproceedings{\njiang2026cascadia,\ntitle={Cascadia: An Efficient Cascade Serving System for Large Language Models},\nauthor={Youhe Jiang and Fangcheng Fu and Wanru Zhao and Stephan Rabanser and Jintao Zhang and Nicholas D. Lane and Binhang Yuan},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=wkrWbrHTJQ}\n}"},"title":{"value":"Cascadia: An Efficient Cascade Serving System for Large Language Models"},"pdf":{"value":"/pdf/a1ec00676c2c3a4e0e9b432a8676721778d8cd3d.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"jiang|cascadia_an_efficient_cascade_serving_system_for_large_language_models"},"authorids":{"value":["~Youhe_Jiang3","~Fangcheng_Fu1","~Wanru_Zhao1","~Stephan_Rabanser1","~Jintao_Zhang2","~Nicholas_D._Lane2","~Binhang_Yuan1"]},"authors":{"value":["Youhe Jiang","Fangcheng Fu","Wanru Zhao","Stephan Rabanser","Jintao Zhang","Nicholas D. Lane","Binhang Yuan"]}},"version":2},{"content":{"venue":{"value":"Crossref"},"pdf":{"value":"https://www.techrxiv.org/doi/pdf/10.36227/techrxiv.177220388.80392106"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"sharma|phiguard_complianceaware_llm_routing_for_healthcare_with_distributionfree_safety_guarantees"},"html":{"value":"https://doi.org/10.36227/techrxiv.177220388.80392106/v1"},"abstract":{"value":"Healthcare organizations deploying Large Language Models face a regulatory tradeoff: route queries through cloud APIs for cost efficiency and risk exposing protected health information (PHI), or keep everything on-premises. Existing LLM routers (RouteLLM, FrugalGPT, PILOT, BaRP) optimize cost against quality-none enforce data-residency constraints. We propose PHI-GUARD, a framework that treats data sensitivity as a hard compliance constraint. We develop SafeTS (Safetyconstrained Thompson Sampling), which builds safe action sets from classifier posteriors, and CARES (Compliance-Aware Residual Envelope Scoring), which provides distribution-free safety guarantees via asymmetric conformal residuals-requiring only exchangeability, not calibration. On 30,000 MIMIC-IV clinical queries with 10,000-query MIMIC-III cross-validation, our Dis-tilBERT classifier achieves 98.4% accuracy with 97.4% T3 (PHI) recall, but 8 misclassifications exceed confidence 0.80 (max 0.997). Static routing leaks 0.92% of PHI queries. SafeTS reduces violations from 83 to 1-8 through safe action sets but cannot eliminate them. CARES is the only strategy achieving zero violations with a verified distribution-free bound (δ + 1/(n + 1) = 0.005), routing 35.6% to cloud. Classifier ablation confirms CARES maintains zero violations across all tested architectures-from DistilBERT (98.4% accuracy) to a rule-based heuristic (56.2% accuracy)-demonstrating that the guarantee adapts to classifier quality automatically. Cross-dataset evaluation confirms CARES maintains zero violations despite 11-point T3 recall degradation on MIMIC-III, while StaticILP violations increase 5.8×. End-toend validation with real API endpoints (GPT-4o-mini, Llama-3.1) confirms safety guarantees transfer to live execution."},"title":{"value":"PHI-GUARD: Compliance-Aware LLM Routing for Healthcare with Distribution-Free Safety Guarantees"},"authors":{"value":[{"fullname":"Aman Sharma","username":"~Aman_Sharma8"}]}},"tmdate":1789090137990,"pdate":1772150400000,"externalIds":["doi:10.36227/techrxiv.177220388.80392106/v1"],"tcdate":1774328534056,"writers":["~","OpenReview.net/Public_Article/ORCID.org","OpenReview.net/Support"],"signatures":["~Aman_Sharma8"],"forum":"s4IdSmvYM3","license":"CC BY-SA 4.0","number":58966,"cdate":1772255937381,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/ORCID.org/-/Record","OpenReview.net/Public_Article/-/Edit","OpenReview.net/-/Edit"],"mdate":1789090137990,"domain":"OpenReview.net/Public_Article","id":"s4IdSmvYM3","version":2},{"content":{"comment":{"value":"I thank the authors for their detailed response and for their efforts to improve the manuscript's formatting and presentation. I have read the rebuttal carefully. While some points are clarified, I still have concerns regarding the stability of the proposed method and the validation of the core concepts. I have decided to maintain my current score for now. Please find my follow-up comments and questions below:\n\n**1. Regarding Threshold Sensitivity and Stability (Major Concern)**\n\nIn the rebuttal, the authors state: *\"Only optimal range where both lightweight and strong experts were receiving a number of tokens to process was 0.478-0.5 depending on our architecture.\"* This admission raises a significant concern about the robustness of the proposed method. A functional hyperparameter range of only `0.022` (0.478 to 0.500) suggests that the router's output probabilities are extremely concentrated or that the system is highly brittle. In a robust MoE system, we would expect the router to learn a distribution that allows for a wider, more stable range of thresholds, or to use a top-k mechanism that avoids such collapse.\n*   **My Question is:** Why is the effective threshold range so narrow? Does this imply that the router is not effectively learning to separate \"easy\" and \"hard\" tokens with high confidence (i.e., most sigmoid outputs are hovering near 0.5)? Did you apply any auxiliary load-balancing loss (like the one mentioned in Eq. 4) specifically to encourage a wider variance in the router's output probabilities?\n\n**2. Regarding the Definition and Validation of Token Complexity**\n\nThe authors argue that Table 2 (perplexity reduction) validates the heuristic complexity labels. However, a reduction in overall model perplexity only proves that the *architecture* works, not that the *heuristic labels* (e.g., treating specific math symbols as \"complex\") correlate with actual statistical difficulty (surprisal/entropy).\n\n*   **My Question is:** I appreciate the honesty that you \"will work for creating a specific definition.\" However, for the current submission, is there any quantitative evidence *directly* correlating your heuristic labels with the model's internal uncertainty? For example, before training the router, do the tokens you labeled as \"complex\" actually have higher loss/entropy in the base model compared to \"simple\" tokens? Without this, the definition remains arbitrary.\n\n**3. Regarding Baselines (RouteLLM/RouterBench)**\n\nI accept the authors' clarification that RouteLLM and RouterBench operate at the model/query level rather than the token level. Excluding them is acceptable given the architectural differences. However, this places more burden on the comparison with DynaMoE and the internal ablation studies to prove the method's efficacy.\n\n**4. Regarding Novelty vs. DynaMoE**\n\nThe authors state that DynaMoE uses similarity heuristics while the proposed method uses a threshold-based complexity score. I am still not fully convinced that this represents a fundamental shift. Both approaches rely on heuristics to proxy difficulty. The primary difference appears to be the training signal (few-shot supervision vs. similarity). **Could you elaborate on why the few-shot supervision provides a superior signal for *routing* specifically? Does it lead to better expert specialization compared to similarity-based grouping?**\n\nI look forward to your response."},"title":{"value":"Thanks for Authors’ Rebuttal"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1770917692109,"tcdate":1764249421327,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission13468/Reviewer_3Bi6"],"signatures":["ICLR.cc/2026/Conference/Submission13468/Reviewer_3Bi6"],"forum":"miyzT0uhhF","number":5,"license":"CC BY 4.0","cdate":1764249421327,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission13468/-/Official_Comment","ICLR.cc/2026/Conference/-/Edit"],"mdate":1770917692109,"domain":"ICLR.cc/2026/Conference","replyto":"l3Fvbq9XJM","id":"7w52uyFfSi","forumContent":{"TLDR":{"value":"Mixture of Expert Architecture for Large Language Models to enhance scaling and performance"},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Mixture of Experts","Large Language Model","Router","Token Complexity Threshold."]},"primary_area":{"value":"generative models"},"abstract":{"value":"Mixture-of-Experts (MoE) architectures have emerged as a powerful technique for improving and scaling Large Language Models by conditionally activating Feed Forward subnetworks and distributing tokens through a routing system,  within the Transformer layers. However, existing MoE methods often rely on static top-k routing strategies that do not involve token-level variability in complexity, leading to suboptimal expert utilization. In this research, we propose a novel token-complexity-based routing framework that dynamically allocates tokens to either lightweight or strong feedforward networks (FFNs) based on their estimated token complexity. Our router is trained using a few-shot classification objective to distinguish between easy and complex tokens and surrogate neural network layer. The efficacy of the framework is evaluated while integrating the router with Mistral-7B and Llama-2-7B model. We evaluate our approach on several benchmarks from various fields, and our proposed MoE framework improves accuracy up to 12% compared to the state-of-the-art results using different MoE architecture, with reasonable computational cost."},"_bibtex":{"value":"@misc{\nrahman2026tokencomplexity,\ntitle={Token-Complexity based Routing Technique within Mixture of Experts Architecture for Large Language Model},\nauthor={Tasnia Rahman and Sathish A.P. Kumar},\nyear={2026},\nurl={https://openreview.net/forum?id=miyzT0uhhF}\n}"},"title":{"value":"Token-Complexity based Routing Technique within Mixture of Experts Architecture for Large Language Model"},"pdf":{"value":"/pdf/8daaf66a8305dcfe6a2ae91cf82adf0862008c1a.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"rahman|tokencomplexity_based_routing_technique_within_mixture_of_experts_architecture_for_large_language_model"},"authorids":{"value":["~Tasnia_Rahman1","~Sathish_A.P._Kumar1"]},"authors":{"value":["Tasnia Rahman","Sathish A.P. Kumar"]}},"version":2},{"content":{"summary":{"value":"This paper addresses the problem of selecting appropriate Large Language Models for multi-stage complex tasks to jointly optimize result quality while minimizing inference cost. The authors formalize this as a constraint-based optimization problem with the intention to guarantee lower bounds on accuracy and minimize LLM inference cost. They propose MixLLM, a training-free tree-based search framework that leverages exploration-exploitation principles with a two-phase greedy strategy to adaptively balance result quality and cost. The method includes automatic early termination through search-space pruning based on cost estimation and accuracy prediction."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"Q0: Answer the concerns raised in the Weaknesses section above.\nQ1: The candidate LLM pool in your experiments is limited to 5 models, yet you mention production environments may have 60+ candidate LLMs. Can you provide empirical analysis of: (a) wall-clock time for the search process across different pool sizes, (b) average number of examined nodes before termination, (c) how performance degrades with LLM pools of 10, 20, and 40+ models, and (d) what percentage of inference cost savings is consumed by exploration overhead?\nQ2: Can you provide: (a) theoretical guarantees on solution quality (e.g., approximation bounds showing the found solution is within X% of optimal), (b) convergence analysis or proof that the method terminates in finite time under all conditions, (c) worst-case complexity analysis for the number of examined nodes, and (d) conditions under which the method is guaranteed to find the global optimum?\nQ3: For practitioners deploying MixLLM, how should they determine: (a) the baseline accuracy $a$ when there is no clear \"best same-LLM\" combination or when different same-LLM combinations perform similarly, (b) the relaxation parameter $\\theta$, (c) thresholds for pruning nodes in the cost estimation and accuracy prediction mechanisms."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"1. The problem formulation is technically sound and practical. The constraint-based single-objective optimization approach with the relaxation parameter $\\theta$ elegantly handles LLM stochasticity, which is empirically validated through multi-round experiments showing stable fluctuation ranges.\n2. The technical contributions are substantial. The two-phase greedy selection strategy that switches from accuracy prioritization to cost minimization once the accuracy constraint is satisfied is intuitive yet effective.\n3. Unlike competing methods (FrugalGPT, GraphRouter, HybridLLM, RouteLLM) that require training routers on task-specific data, MixLLM is training-free and automatically terminates when no promising nodes remain. This is a significant practical advantage, reducing deployment barriers and making the approach more generalizable across different domains."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. My major concern is, while the paper claims the method is efficient, there is insufficient analysis of computational overhead and scalability. The candidate LLM pool is limited to 5 models, and for a k-stage task, the combinatorial search space is |L|^k. It is not clear how the proposed approach can scale to thousands of LLMs. \n2. The accuracy prediction mechanism shows concerning limitations. For MAC-SQL(Spider), the prediction success rate is only 40%, attributed to marginal accuracy differences (≤2%) and limited empirical runtime data. This suggests the prediction mechanism may fail precisely when fine-grained distinctions matter most. The paper does not provide: (a) detailed description of the prediction algorithm beyond high-level concepts about \"runtime experience knowledge\" and \"node lineage,\" (b) ablation studies isolating the accuracy prediction component's contribution, (c) analysis of failure modes and when the predictor can be trusted, or (d) comparison with standard supervised learning baselines for accuracy prediction.\n3. The related work section omits several relevant areas that diminish the paper's positioning. Missing categories include  (a) Mixture-of-Experts (MoE) literature for routing between specialized models, which addresses similar selection problems, (b) LLM selection [1] and routing works [2], and more. \n4.  The paper lacks systematic analysis of when and why the method works. Key missing elements include: (a) characterization of task properties (e.g., stage dependencies, error propagation patterns, intermediate result quality distributions) that influence LLM selection effectiveness, (b) theoretical analysis or convergence guarantees for the search procedure. \n\n\n[1] RELM: https://openreview.net/pdf?id=gWi4ZcPQRl\n[2] Smoothie: https://openreview.net/pdf?id=pPSWHsgqRp"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762922269322,"tcdate":1760688046584,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission11091/Reviewer_Wf1K"],"signatures":["ICLR.cc/2026/Conference/Submission11091/Reviewer_Wf1K"],"forum":"jVjlpauPD5","number":1,"license":"CC BY 4.0","cdate":1760688046584,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission11091/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762922269322,"domain":"ICLR.cc/2026/Conference","replyto":"jVjlpauPD5","id":"cjzdRZjaaJ","forumContent":{"TLDR":{"value":"We study the problem of selecting LLMs for multi-stage complex tasks to preserve high-quality results with minimum LLM inference cost."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["LLM selection","multi-stage tasks","high-quality results","minimum inference cost","quality-cost trade-off"]},"primary_area":{"value":"other topics in machine learning (i.e., none of the above)"},"abstract":{"value":"We study the problem of selecting LLMs for multi-stage complex tasks to jointly\noptimize final result quality and minimize LLM inference cost. Existing approaches primarily target simple tasks or optimize for improving result quality\nor reducing cost only, overlooking the trade-off between them. We address this\ngap by systematically investigating LLM performance in multi-stage workflows.\nInspired by our findings from real-world applications, we formalize the LLM\nselection problem as a constraint-based optimization task with good properties:\nguarantee lower bounds on accuracy, minimize LLM inference cost and tolerate\nperformance fluctuations caused by LLM stochasticity, making it more practical\nfor users. We further introduce MixLLM, a search framework that leverages the\nexploration–exploitation principle to adaptively balance result quality and LLM\ninference cost. MixLLM is carefully designed to efficiently identify a (near-)optimal solution with minimal exploration and to terminate automatically and\nearly via search-space pruning. Experimental results demonstrate that, compared\nto using a single powerful commercial or open-source LLM, or selecting LLMs\nvia existing state-of-the-art methods, our approach not only improves result quality (by 1% – 16%) but also significantly reduces inference cost (by 18% – 92%). In addition, our approach efficiently adapts to different tasks, methods, and datasets,\ndemonstrating its practicality and robustness for multi-stage complex tasks."},"_bibtex":{"value":"@misc{\nding2026mixllm,\ntitle={Mix{LLM}: Selecting Large Language Models with High-Quality Results and Minimum Inference Cost for Multi-Stage Complex Tasks},\nauthor={Mengsu Ding and Rong Zhu and Bolin Ding and Jingren Zhou},\nyear={2026},\nurl={https://openreview.net/forum?id=jVjlpauPD5}\n}"},"title":{"value":"MixLLM: Selecting Large Language Models with High-Quality Results and Minimum Inference Cost for Multi-Stage Complex Tasks"},"pdf":{"value":"/pdf/0ae5042abe7f52c2dcdc6ec0992d7eece6849a12.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"ding|mixllm_selecting_large_language_models_with_highquality_results_and_minimum_inference_cost_for_multistage_complex_tasks"},"authorids":{"value":["~Mengsu_Ding1","~Rong_Zhu2","~Bolin_Ding3","~Jingren_Zhou1"]},"authors":{"value":["Mengsu Ding","Rong Zhu","Bolin Ding","Jingren Zhou"]}},"version":2},{"content":{"comment":{"value":"Thanks for the detailed responses.\n\nI still feel that relying solely on cumulative regret is insufficient to evaluate the efficacy of the proposed method, especially given the extensive literature on LLM routing. At a minimum, the authors should report accuracy (or other relevant metrics) for the proposed pipeline and use the strong LLM’s performance on the same evaluation set (or random selection as in [1]) as a reference point. In addition, the other works [1, 2, 3] cited in this paper also utilize preference data; some of their techniques may be adapted as baselines for the setting studied here, thus also addressing concerns from other reviewers regarding the need for comparisons with more modern baselines.\n\n[1] Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M. Waleed Kadous, and Ion Stoica. RouteLLM: Learning to Route LLMs from Preference Data. In International Conference on Learning Representations, pp. 1–14, 2025.   \n[2] Zesen Zhao, Shuowei Jin, and Zhuoqing Morley Mao. Eagle: Efficient Training-Free Router for\nMulti-LLM Inference. In Proceedings of the Workshop on Machine Learning for Systems at\nNeurIPS 2024, pp. 1–8, 2024.  \n[3] Xinyuan Wang, Yanchi Liu, Wei Cheng, Xujiang Zhao, Zhengzhang Chen, Wenchao Yu, Yanjie Fu,\nand Haifeng Chen. MixLLM: Dynamic Routing in Mixed Large Language Models. In Proceed-\nings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Compu-\ntational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 10912–10922,\n2025."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764131964259,"tcdate":1764131964259,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission16481/Reviewer_RdwD"],"signatures":["ICLR.cc/2026/Conference/Submission16481/Reviewer_RdwD"],"forum":"zf3gNrp3jp","number":13,"license":"CC BY 4.0","cdate":1764131964259,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission16481/-/Official_Comment"],"mdate":1764131964259,"domain":"ICLR.cc/2026/Conference","replyto":"RAqlF9PR8P","id":"o5WN5KQjAb","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"TLDR":{"value":"This submission presents a contextual dueling bandit approach for LLM routing under preference feedback."},"keywords":{"value":["LLM routing","feel good thompson sampling","contextual bandits"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"We study LLM routing, the problem of selecting the best model for each query while balancing user satisfaction, model expertise, and inference cost. We formulate routing as contextual dueling bandits, learning from pairwise preference feedback rather than absolute scores, thereby yielding label-efficient and dynamic adaptation. Building on this formulation, we introduce Category-Calibrated Fine-Tuning (CCFT), a representation-learning method that derives model embeddings from offline data using contrastive fine-tuning with categorical weighting. These embeddings enable the practical instantiation of Feel-Good Thompson Sampling for Contextual Dueling Bandits (FGTS.CDB), a theoretically grounded posterior-sampling algorithm. We propose four variants of the categorical weighting that explicitly integrate model quality and cost, and we empirically evaluate the proposed methods on the RouterBench and MixInstruct datasets. Across both benchmarks, our methods achieve lower cumulative regret and faster convergence, with better robustness and performance-cost balance than strong baselines built with a general-purpose OpenAI embedding model."},"_bibtex":{"value":"@misc{\nchiang2026llm,\ntitle={{LLM} Routing with Dueling Feedback},\nauthor={Chao-Kai Chiang and Takashi Ishida and Masashi Sugiyama},\nyear={2026},\nurl={https://openreview.net/forum?id=zf3gNrp3jp}\n}"},"title":{"value":"LLM Routing with Dueling Feedback"},"pdf":{"value":"/pdf/6b80ce072b321fe2c98ba0445fd9a04491ebbf17.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"chiang|llm_routing_with_dueling_feedback"},"authorids":{"value":["~Chao-Kai_Chiang1","~Takashi_Ishida1","~Masashi_Sugiyama1"]},"authors":{"value":["Chao-Kai Chiang","Takashi Ishida","Masashi Sugiyama"]}},"version":2},{"content":{"comment":{"value":"Q1: Comparison with recent routing methods. A1: We will expand Section 2 to explicitly contrast with recent works like RouteLLM and AutoMix. The key distinction is that Bench-CoE's Subject-Level router does not require training a router on model outputs or preference data (which is expensive). Instead, it trains a classifier on public benchmark questions, effectively transferring the \"routing knowledge\" from the benchmark to the user task.\n\nQ2: Handling ties in expert performance. A2: As briefly mentioned in Section 3.2 (Eq 5 context), when experts tie on a specific query (or subject), we break ties by selecting the model with the highest overall average score across the entire benchmark. We will make this heuristic more explicit in the method section.\n\nQ3: Training details. A3: We apologize for the omission.\n\nClassifiers: We fine-tuned BERT-base-uncased (for text) and TinyLLaVA (for multimodal) using standard cross-entropy loss on the subject labels.\n\nHyperparameters: Learning rate 2e-5, batch size 32, trained for 3 epochs.\n\nWe will add a \"Reproducibility\" section in the Appendix with full hyperparameters.\n\nQ4: Adding new subjects. A4: To add a new subject, one simply adds a new output head (class) to the router and fine-tunes it on a small set of examples for that subject. Alternatively, if the new subject falls under an existing broader category, no retraining is needed—only the Subject-Expert Map needs to be updated in the lookup table."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763706478876,"tcdate":1763706478876,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission16362/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission16362/Authors"],"forum":"vnlsxFbWSB","number":5,"license":"CC BY 4.0","cdate":1763706478876,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission16362/-/Official_Comment"],"mdate":1763706478876,"domain":"ICLR.cc/2026/Conference","replyto":"AuaYAetibj","id":"KIsUwGpbB9","forumContent":{"TLDR":{"value":"We propose the Bench-CoE framework that enables Collaboration of Experts (CoE) by effectively leveraging benchmark evaluations to achieve optimal performance across various tasks."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Bench-CoE","Framework","Collaboration of Experts","Benchmark","LLMs"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"Large Language Models (LLMs) are key technologies that drive intelligent systems to handle multiple tasks. To meet the demands of various tasks, an increasing number of LLMs-driven experts with diverse capabilities have been developed, spreading from language to visual understanding and generalization, accompanied by corresponding benchmarks to evaluate their performance. This paper proposes the Bench-CoE framework, which enables Collaboration of Experts (CoE) by effectively leveraging benchmark evaluations to achieve optimal performance across various tasks. Bench-CoE consists of a set of specialized expert models, a router for assigning tasks to corresponding experts, and a benchmark dataset for training the router. Based on this framework, we first formulate Query-Level Bench-CoE that is an abstraction of existing CoE methods exploiting the benchmark dataset. We further propose Subject-Level Bench-CoE, a new method that effectively addresses the potential issues of Query-Level Bench-CoE in poor generalization and labeling costs during training the router. Experiments show that the Query-Level Bench-CoE excels in in-distribution tasks, while the Subject-Level Bench-CoE demonstrates stronger out-of-distribution generalization. Our proposed Bench-CoE achieves efficient expert collaboration with minimal training label costs, improving adaptability in multi-task and cross-domain scenarios."},"_bibtex":{"value":"@misc{\nwang2026benchcoe,\ntitle={Bench-CoE: A Framework for Collaboration of Experts from Benchmark},\nauthor={Yuanshuai Wang and Xingjian Zhang and Jinkun Zhao and Siwei Wen and Peilin Feng and Shuhao Liao and Lei Huang and wenjun wu},\nyear={2026},\nurl={https://openreview.net/forum?id=vnlsxFbWSB}\n}"},"title":{"value":"Bench-CoE: A Framework for Collaboration of Experts from Benchmark"},"pdf":{"value":"/pdf/6511239f5f3769c1d5ff62d9986cc51ec0e06c44.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"wang|benchcoe_a_framework_for_collaboration_of_experts_from_benchmark"},"authorids":{"value":["~Yuanshuai_Wang1","~Xingjian_Zhang2","~Jinkun_Zhao1","~Siwei_Wen5","~Peilin_Feng1","~Shuhao_Liao2","~Lei_Huang1","~wenjun_wu3"]},"authors":{"value":["Yuanshuai Wang","Xingjian Zhang","Jinkun Zhao","Siwei Wen","Peilin Feng","Shuhao Liao","Lei Huang","wenjun wu"]}},"version":2},{"content":{"additional_comments_on_reviewer_discussion":{"value":"Authors have sufficiently engaged with reviewers during the rebuttal phase and Reviewers V25N and GuJg have agreed that their concerns have been addressed. Although Reviewer 7nQc has not engaged with the authors despite nudge from me, his comments also seem positive."},"metareview":{"value":"The paper proposes RouteLLM to dynamically choose between a stronger and weaker LLM during inference, achieving 2x cost savings on 3 benchmarks with minimal impact on response quality. Empirical results show that data augmentation is important to letting the routers outperform a random baseline and generalise across domains without needing to be retrained.\n\n\nStrengths:\n* Demonstrates significant cost savings (over 2x) without compromising response quality [Reviewer V25N, 7nQc, GuJg]\n* Introduces APGR to quantify performance relative to cost, effectively balancing quality and cost constraints [Reviewer 7nQc and V25N]\n* Demonstrates robustness and adaptability, as the router generalizes effectively to unseen LLM pairs and domains without retraining [all reviewers]\n* Authors also provide a process of calibration for estimating the right cost threshold to calibrate based on both the target data distribution and router used.\n\nWeakness\n* The model only considers  binary routing of strong and weak models, but does not consider other binary differences between models such as coding-specific versus generalist models or English-only versus multilingual models. To reviewer V25N comment on this, authors claim that it is applicable to these other binary differences between models, without any substantiations."}},"nonreaders":[],"tmdate":1738733183604,"tcdate":1734586664012,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission13083/Senior_Area_Chairs","ICLR.cc/2025/Conference/Submission13083/Area_Chair_YxWN"],"signatures":["ICLR.cc/2025/Conference/Submission13083/Area_Chair_YxWN"],"forum":"8sSqNntaMr","number":1,"license":"CC BY 4.0","cdate":1734586664012,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission13083/-/Meta_Review","ICLR.cc/2025/Conference/-/Edit"],"mdate":1738733183604,"domain":"ICLR.cc/2025/Conference","replyto":"8sSqNntaMr","id":"sdKMWI8E9h","forumContent":{"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Large language models","query routing"]},"supplementary_material":{"value":"/attachment/21a6352b733fa8b803857c0be57176fe75cfc5f6.zip"},"primary_area":{"value":"generative models"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Large language models (LLMs) excel at a wide range of tasks, but choosing the right model often involves balancing performance and cost. Powerful models offer better results but are expensive, while smaller models are more cost-effective but less capable. To address this trade-off, we introduce a training framework for learning efficient router models that dynamically select between a stronger and weaker LLM during inference. Our framework leverages human preference data and employs data augmentation techniques to enhance performance. Evaluations on public benchmarks show that our approach can reduce costs by over 2 times without sacrificing response quality. Moreover, our routers exhibit strong generalization capabilities, maintaining performance even when routing between LLMs not included in training. This highlights the potential of our framework to deliver cost-effective, high-performance LLM solutions."},"_bibtex":{"value":"@inproceedings{\nong2025routellm,\ntitle={Route{LLM}: Learning to Route {LLM}s from Preference Data},\nauthor={Isaac Ong and Amjad Almahairi and Vincent Wu and Wei-Lin Chiang and Tianhao Wu and Joseph E. Gonzalez and M Waleed Kadous and Ion Stoica},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=8sSqNntaMr}\n}"},"title":{"value":"RouteLLM: Learning to Route LLMs from Preference Data"},"pdf":{"value":"/pdf/900c2f5b71fd9d4dc34d8f299cce9a11336d30b4.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"ong|routellm_learning_to_route_llms_from_preference_data"},"authorids":{"value":["~Isaac_Ong1","~Amjad_Almahairi1","~Vincent_Wu1","~Wei-Lin_Chiang1","~Tianhao_Wu1","~Joseph_E._Gonzalez1","~M_Waleed_Kadous1","~Ion_Stoica1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Isaac Ong","Amjad Almahairi","Vincent Wu","Wei-Lin Chiang","Tianhao Wu","Joseph E. Gonzalez","M Waleed Kadous","Ion Stoica"]}},"version":2},{"content":{"comment":{"value":"Dear Reviewer RdwD,\n\nThank you for raising these points, and for the helpful suggestions. We are currently deriving the accuracy metrics, and hope to add it before the discussion period ends.\n\nWe ran RouterBench experiments with the suggested random selection strategy. We also compare with an optimal selection strategy (that selects two identical LLMs that are optimal every time). The results below show that our e5b_E4 and mpnet_E4 performs close to the optimal with respect to the score, and is much better than the random strategy.\n\n| Method   | Mean over 3 shuffles | Std              |\n| -------- | -------------------- | ---------------- |\n| optimal  | 0.7663               | 0.0085           |\n| e5b_E4   | 0.7536               | 0.0276           |\n| mpnet_E4 | 0.7416               | 0.0291           |\n| random   | 0.5855               | 0.0190           |\n\nEagle [2], by design (Section 2.2, page 3 in [2]), ranks LLMs only based on empirical scores and does not include any exploration mechanism. As a result, it cannot manage the exploration–exploitation trade-off and is vulnerable to distribution shift in online settings, an issue that our Thompson-sampling-based method is explicitly designed to handle. Because Eagle’s code is not publicly available, conducting full experimental comparisons is difficult. However, in our simple implementation, the cumulative-regret curve grows roughly linearly, consistent with our conjecture regarding its limited ability to adapt online data. If accepted, we will complete the implementation and add the comparison to the camera ready version.\n\nRouteLLM [1] and MixLLM [3] operate in different regimes: [1] uses preference labels similar to our case but handles only 2‑model routing with an offline setup, and [3] utilizes point‑wise supervision. In contrast, we study online routing over >2 models from pairwise feedback. We will clarify these differences in the paper."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764800814225,"tcdate":1764209666945,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission16481/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission16481/Authors"],"forum":"zf3gNrp3jp","number":15,"license":"CC BY 4.0","cdate":1764209666945,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission16481/-/Official_Comment"],"mdate":1764800814225,"domain":"ICLR.cc/2026/Conference","replyto":"Rycin8aVgl","id":"UvrVM8pxmd","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"TLDR":{"value":"This submission presents a contextual dueling bandit approach for LLM routing under preference feedback."},"keywords":{"value":["LLM routing","feel good thompson sampling","contextual bandits"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"We study LLM routing, the problem of selecting the best model for each query while balancing user satisfaction, model expertise, and inference cost. We formulate routing as contextual dueling bandits, learning from pairwise preference feedback rather than absolute scores, thereby yielding label-efficient and dynamic adaptation. Building on this formulation, we introduce Category-Calibrated Fine-Tuning (CCFT), a representation-learning method that derives model embeddings from offline data using contrastive fine-tuning with categorical weighting. These embeddings enable the practical instantiation of Feel-Good Thompson Sampling for Contextual Dueling Bandits (FGTS.CDB), a theoretically grounded posterior-sampling algorithm. We propose four variants of the categorical weighting that explicitly integrate model quality and cost, and we empirically evaluate the proposed methods on the RouterBench and MixInstruct datasets. Across both benchmarks, our methods achieve lower cumulative regret and faster convergence, with better robustness and performance-cost balance than strong baselines built with a general-purpose OpenAI embedding model."},"_bibtex":{"value":"@misc{\nchiang2026llm,\ntitle={{LLM} Routing with Dueling Feedback},\nauthor={Chao-Kai Chiang and Takashi Ishida and Masashi Sugiyama},\nyear={2026},\nurl={https://openreview.net/forum?id=zf3gNrp3jp}\n}"},"title":{"value":"LLM Routing with Dueling Feedback"},"pdf":{"value":"/pdf/6b80ce072b321fe2c98ba0445fd9a04491ebbf17.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"chiang|llm_routing_with_dueling_feedback"},"authorids":{"value":["~Chao-Kai_Chiang1","~Takashi_Ishida1","~Masashi_Sugiyama1"]},"authors":{"value":["Chao-Kai Chiang","Takashi Ishida","Masashi Sugiyama"]}},"version":2},{"content":{"comment":{"value":">**W1:** The framework introduces significant complexity. The experiments are limited to ≤15 agents. \n\n**A1:** Although Agora introduces an expressive economic structure, its runtime complexity remains modest because the system enforces **strict sparsity** rather than activating all agents.\n\n* **Theoretical Convergence (No Oscillation):** As proven in **Proposition 3.3.1** (Appendix C), the trading protocol performs a deterministic greedy descent on the global cost. Since the total cost is bounded, the trading sequence is **mathematically guaranteed to terminate** in finite steps, preventing infinite loops or high-frequency trading risks.\n\n* **Sparsity via Economic Rationality:** The profitability constraint (Eq. 5, $\\Delta \\mathcal{C} < 0$) acts as a strict gatekeeper. In a hypothetical pool of 100 agents, the vast majority will lack a \"comparative advantage\" for a specific query and will be automatically filtered out. The system maintains a small **active set** of experts regardless of the total pool size.\n\n* **Empirical Evidence:** Our experiments in **Appendix D** confirm this self-regulation: tripling the pool size ($N=5 \\to 15$) only marginally increased trade frequency ($0.8 \\to 1.2$ trades/task), demonstrating that communication overhead hits a saturation point and does not scale linearly with $N$.\n\n* **Negligible Overhead:** The Broker's selection complexity is linear $O(N)$ and computationally trivial compared to the VLM inference costs (TFLOPs) it saves.\n\n\n---\n\n\n>**W2:** The paper overlooks key transaction costs and may not be viable for real-time applications.\n\n**A2:** The coordination cost in Agora is intentionally lightweight and negligible relative to VLM inference. The Broker evaluation and the ( $\\Delta\\mathcal{C}$ ) profitability checks consist of lightweight arithmetic operations that run in ($O(N)$) time over the naturally sparse active set enforced by Eq. 5. Communication is similarly minimal, as only small uncertainty vectors are exchanged, and the system performs on average just **0.8–1.2 trades per task** (Table 4).\n\nImportantly, the **“Average Time ($s$)”** in Fig. 4 and the runtime measurements in Appendix I (8.7–22.8s) already include all coordination steps end-to-end. These results show that Agora matches or outperforms router baselines in wall-clock time while achieving higher accuracy, confirming that coordination overhead does not hinder real-time viability.\n\n---\n\n\n>**W3:** While numerous baselines are included, the adaptation of language-model-focused routers (like FrugalGPT, RouteLLM) to the multi-modal, multi-agent setting is only briefly mentioned in the appendix.\n\n**A3:** We appreciate this opportunity to clarify. In the revised paper, we have explicitly detailed the multimodal adaptation in Section 4.1. Specifically:\n\n1.  **Unified Multimodal Wrapper:** As text-oriented routers (FrugalGPT, RouteLLM, etc.) cannot natively process images, we implemented a unified wrapper (detailed in Appendix K) that encodes visual inputs using the visual encoder of the strongest available VLM in the pool. This visual embedding is concatenated with the textual query, enabling these routers to apply their original selection logic to multimodal contexts without structural modification.\n2.  **Rigorous Fairness:** All baselines—including these adapted routers—operate on the **identical heterogeneous agent pool ($N=6$)** as Agora, using consistent prompts and decoding settings. This ensures that any performance gap arises purely from the coordination strategy, not from the underlying model capabilities.\n\nTo address the concern regarding cross-dataset validation, we have extended our head-to-head comparison beyond MMBench. **Table R2** below demonstrates that Agora consistently outperforms the strongest aggregation (MoA) and routing (KABB-VLM) baselines across reasoning-heavy (MathVision) and perception-heavy (InfoVQA, CC-OCR) benchmarks. Notably, while baselines struggle to significantly surpass the best single expert in perception tasks due to hallucination risks, Agora’s uncertainty trading mechanism effectively mitigates this, securing clear gains.\n\n---\n\n\n**Table R2: Head-to-Head Strategy Comparison on Additional Benchmarks (Accuracy, %)**\n*Comparison using the exact same heterogeneous agent pool ($N=6$) and protocol as Table 1.*\n\n| Dataset | MoA | KABB-VLM | **Agora (Ours)** |\n| :--- | :---: | :---: | :---: |\n| **MathVision** | 42.1 | 42.5 | **44.3** |\n| **InfoVQA** | 87.6 | 87.8 | **88.9** |\n| **CC-OCR** | 80.2 | 80.4 | **81.2** |"},"title":{"value":"Part I"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763926818347,"tcdate":1763568083435,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission582/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission582/Authors"],"forum":"zeqCjGQB4U","number":1,"license":"CC BY 4.0","cdate":1763568083435,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission582/-/Official_Comment"],"mdate":1763926818347,"domain":"ICLR.cc/2026/Conference","replyto":"aj5Eea9564","id":"6DxdiTQcro","forumContent":{"TLDR":{"value":"We presents a framework that reframes coordination as a decentralized market for uncertainty."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["agent; Vision Language Model; Uncernity"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Vision-Language Models (VLMs) enable powerful multi-agent systems, but scaling them is economically unsustainable: coordinating heterogeneous agents under information asymmetry often spirals costs. Existing paradigms, such as Mixture-of-Agents and knowledge-based routers, rely on heuristic proxies that ignore costs and collapse uncertainty structure, leading to provably suboptimal coordination.\nWe introduce Agora, a framework that reframes coordination as a decentralized market for uncertainty. Agora formalizes epistemic uncertainty into a structured, tradable asset (perceptual, semantic, inferential), and enforces profitability-driven trading among agents based on rational economic rules. A market-aware broker, extending Thompson Sampling, initiates collaboration and guides the system toward cost-efficient equilibria. Experiments on five multimodal benchmarks (MMMU, MMBench, MathVision, InfoVQA, CC-OCR) show that Agora outperforms strong VLMs and heuristic multi-agent strategies, e.g., achieving +8.5% accuracy over the best baseline on MMMU while reducing cost by over 3×. These results establish market-based coordination as a principled and scalable paradigm for building economically viable multi-agent visual intelligence systems."},"_bibtex":{"value":"@inproceedings{\nzhang2026why,\ntitle={Why Keep Your Doubts to Yourself? Trading Visual Uncertainties among Vision-Language Models},\nauthor={Jusheng Zhang and Yijia Fan and Kaitong Cai and Jing Yang and Jiawei Yao and Jian Wang and Guanlong Qu and Ziliang Chen and Keze Wang},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=zeqCjGQB4U}\n}"},"title":{"value":"Why Keep Your Doubts to Yourself? Trading Visual Uncertainties among Vision-Language Models"},"pdf":{"value":"/pdf/03f4bead771e441ade14013c814de209cd0630ea.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"zhang|why_keep_your_doubts_to_yourself_trading_visual_uncertainties_among_visionlanguage_models"},"authorids":{"value":["~Jusheng_Zhang3","~Yijia_Fan1","~Kaitong_Cai1","~Jing_Yang39","~Jiawei_Yao3","~Jian_Wang10","~Guanlong_Qu2","~Ziliang_Chen1","~Keze_Wang1"]},"authors":{"value":["Jusheng Zhang","Yijia Fan","Kaitong Cai","Jing Yang","Jiawei Yao","Jian Wang","Guanlong Qu","Ziliang Chen","Keze Wang"]}},"version":2},{"content":{"comment":{"value":"We sincerely thank you for your feedback and suggestions! There exist many misunderstandings. We address all your concerns below:\n\n> W1: This paper introduces a quite complex framework based on different components proposed by past works, while it lacks really interesting or important insights/ findings.\n\nIndeed, there are some misunderstandings about the actual contribution of our method. Our method consists of two critical component: **thinking via CCoT** and **judging difficluty via CCoT**. While these components build upon previous work, they both introduce novel designs tailored to address the specific challenge. \n\nFor the **thinking via CCoT** component, traditional CCoT-based methods such as Coconut, CODI, and CompressCoT suffer from suboptimal fine-tuning performance due to **imperfect alignment targets**, as illustrated in Figure 1. To overcome this limitation, we introduce **synthetic CCoTs, which are generated before fine-tuning and used as enhanced alignment targets**. These synthetic CCoTs effectively capture reasoning information, thereby achieving better fine-tuning performance. As demonstrated in Table 1, this design allows our method to substantially outperform all other CCoT-based approaches.\n\nFor the **judging difficulty via CCoT** component, traditional methods (e.g., Seq_PPL, PromptLLM, RouteLLM, and Probe_Q) **only rely on superficial cues** from the question to judge difficulty, such as perplexity and lexical patterns (Lines 399–406). However, as shown in Figure 6, difficult questions may appear similar to easy ones, sharing short length and similar lexical structures. Relying solely on such superficial cues is insufficient. To overcome this limitation, we **introduce a difficulty classifier that leverages both the question and the reasoning information encoded in the CCoT**. As demonstrated in Table 2, incorporating additional information from CCoTs enables our method to **more effectively identify hard questions**. Furthermore, due to this improved identification capability, our method achieves **better downstream task performance** compared to other hard-question identification baselines, as shown in Figures 3(c) and 3(d)."},"title":{"value":"Authors' Response (1/5)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763361612990,"tcdate":1763361612990,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission6628/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission6628/Authors"],"forum":"cTzmLqlQZv","number":2,"license":"CC BY 4.0","cdate":1763361612990,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission6628/-/Official_Comment"],"mdate":1763361612990,"domain":"ICLR.cc/2026/Conference","replyto":"XaNQ4Oyhel","id":"Nh85ZupbG9","forumContent":{"TLDR":{"value":"We propose an novel efficient reasoning framework for LLM by using synthetic continuous Chain-of-Thought to provide better alignment target."},"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Large Language Models","Large Reasoning Models","Efficient Reasoning","Implicit Reasoning"]},"supplementary_material":{"value":"/attachment/7a8843c4449321667483712986decbd228b6635f.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"While Chain-of-Thought (CoT) reasoning improves model performance, it incurs significant time costs due to the generation of discrete CoT tokens (DCoT). Continuous CoT (CCoT) offers a more efficient alternative, but existing CCoT methods are hampered by indirect fine-tuning, limited alignment, or inconsistent targets. To overcome these limitations, we propose ***SynAdapt***, an innovative efficient reasoning framework. Specifically, *SynAdapt* generates the synthetic CCoT to serve as a precise and effective alignment target for LLMs. This synthetic CCoT explicitly guides the LLM to learn CCoT and derive accurate answers directly. Furthermore, relying solely on CCoT is insufficient for solving hard questions. To address this, *SynAdapt* integrates a difficulty classifier that leverages both question context and CCoT to identify hard questions. CCoT can effectively help identify hard questions after some brief reasoning. We then adaptively prompt the LLM to re-think these hard questions for improved performance. Extensive experimental results across various benchmarks from different difficulty levels strongly demonstrate the effectiveness of our method, achieving the best accuracy-efficiency trade-off."},"_bibtex":{"value":"@misc{\nwang2025synadapt,\ntitle={SynAdapt: Learning Adaptive Reasoning in Large Language Models via Synthetic Continuous Chain-of-Thought},\nauthor={Jianwei Wang and Ziming Wu and Fuming Lai and ShaobingLian and Ziqian Zeng},\nyear={2025},\nurl={https://openreview.net/forum?id=cTzmLqlQZv}\n}"},"title":{"value":"SynAdapt: Learning Adaptive Reasoning in Large Language Models via Synthetic Continuous Chain-of-Thought"},"pdf":{"value":"/pdf/bb2b718d09de28e8a9df263f0bc8d733bfe2909a.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"wang|synadapt_learning_adaptive_reasoning_in_large_language_models_via_synthetic_continuous_chainofthought"},"authorids":{"value":["~Jianwei_Wang2","~Ziming_Wu1","~Fuming_Lai1","~ShaobingLian1","~Ziqian_Zeng1"]},"authors":{"value":["Jianwei Wang","Ziming Wu","Fuming Lai","ShaobingLian","Ziqian Zeng"]}},"version":2},{"content":{"comment":{"value":"We thank the reviewer for the careful reading and constructive suggestions. We are glad that the reviewer recognizes the Bayesian perspective of our method and the role of semantic priors as both initializers and regularizers. We respond to the main points below.\n\n---\n\n**R1: Model description quality (W1, Q1)**\n\nWe agree that the reliability of the semantic prior depends on the quality of the underlying model metadata. In the current manuscript, we mitigate this issue by using an auxiliary LLM to standardize raw model descriptions into unified, routing-oriented capability summaries, prioritizing official documentation, model cards, and paper information while avoiding unsupported claims (Sec. 4.1 and App. B).\n\nHowever, standardization cannot fully address outdated, incomplete, promotional, or incorrect metadata. We have therefore added these failure modes to the Limitations section and identified more reliable prior construction as an important direction for future work (Sec. 6).\n\n---\n\n**R2: Gaussian assumption (W2, Q2)**\n\nWe agree that the Gaussian distribution itself has unbounded support. Our main motivation is to provide a unified formulation across heterogeneous utility signals. Specifically, RouterBench, MMR-Bench, and RouterEval use discrete correctness-based utilities, whereas MixInstruct provides continuous utility scores. The Gaussian formulation allows us to model uncertainty over the expected utility in both cases using the local empirical mean and variance, while retaining efficient closed-form Bayesian fusion. In contrast, Beta-Bernoulli formulations are naturally suited to binary outcomes but do not directly cover continuous utilities such as those in MixInstruct.\n\nImportantly, we do not assume that the raw utility observations themselves are literally Gaussian. Rather, Gaussian fusion serves as a practical approximation for uncertainty over utility estimates. Moreover, all utilities are normalized to [0,1] (Sec. 3.1); the prior mean is bounded by the sigmoid, the likelihood mean is a weighted average of historical utilities, and Eq. (19) forms a precision-weighted average of these two means. Therefore, the posterior mean actually used for routing remains within [0,1].\n\nWe have clarified this modeling choice in Sec. 4.3 and its limitation in Sec. 6.\n\n---\n\n**R3: Non-random / structured missingness (W3, Q3)**\n\nWe thank the reviewer for this valuable suggestion. We added two structured missingness settings on RouterEval: (1) **cost-dependent missingness**, where higher-cost models have higher missing rates; and (2) **new-model missingness**, where only 1% of the historical records of the highest-cost model are retained.\n\n| Setting        | Method       |       Ps ↑ |      QNC ↓ |\n| -------------- | ------------ | ---------: | ---------: |\n| Cost-dependent | BayesRouter  | **0.7906** | **0.8850** |\n|                | RouteLLM-MF  |     0.7850 |     1.0000 |\n|                | Avengers-Pro |     0.7531 |        inf |\n|                | EmbedLLM     |     0.7680 |        inf |\n| New-model      | BayesRouter  | **0.8028** | **0.8062** |\n|                | RouteLLM-MF  |     0.7850 |     1.0000 |\n|                | Avengers-Pro |     0.7760 |        inf |\n|                | EmbedLLM     |     0.7818 |        inf |\n\nBayesRouter consistently outperforms the representative baselines under both settings. These results are now included in Sec. 5.2 (Table 1), with the construction of the two missingness settings and the method-specific handling of missing entries detailed in App. D.2.\n\n---\n\n**R4: Moving latency comparison to the main paper (Q4)**\n\nWe appreciate this suggestion. We agree that the latency comparison provides important evidence for the practical efficiency of BayesRouter, and we have therefore moved this result to the main paper (Sec. 5.2, Q3; Fig. 4).\n\n---\n\n**R5: Query-dependent cost (Q5)**\n\nWe thank the reviewer for pointing this out. This was an error in the manuscript description. In all experiments, inference cost is **query-dependent rather than a fixed model-level average cost**. Specifically, we use a lightweight MLP cost head to predict the cost for each query-model pair. Thus, the cost model can capture systematic query-model-dependent variations in expected inference cost, including those associated with different output lengths. We have corrected the routing formulation in Sec. 3.1 (Eqs. 2--3), added the implementation details of the cost predictor in Sec. 5.1, and revised the performance--cost evaluation in App. C accordingly.\n\n---\n\n**R6: Posterior uncertainty and exploration (Q6)**\n\nWe agree with this observation. The current BayesRouter is primarily exploitative, while posterior precision provides a natural relative uncertainty signal. Using this uncertainty for exploration, e.g., through Thompson Sampling or UCB-style strategies, is a promising extension. We have added this direction to the Future Work discussion (Sec. 6)."},"title":{"value":"Reviewer Response and Revision Summary"}},"parentInvitations":"TMLR/-/Official_Comment","tmdate":1791270103012,"tcdate":1786428984762,"writers":["TMLR","TMLR/Paper9506/Authors"],"signatures":["TMLR/Paper9506/Authors"],"forum":"K3xNTJOM1j","number":1,"license":"CC BY 4.0","cdate":1786428984762,"readers":["everyone"],"invitations":["TMLR/Paper9506/-/Official_Comment","TMLR/Paper9506/-/Comment_Release"],"mdate":1791270103012,"domain":"TMLR","replyto":"Ar0Z5cTubM","id":"SSEnfXSX0C","forumContent":{"submission_length":{"value":"Regular submission (no more than 12 pages of main content)"},"venue":{"value":"Under review for TMLR"},"abstract":{"value":"LLM routing is commonly treated as a data-driven utility estimation problem, where routing decisions are learned primarily from historical interactions. In this work, we argue that such a likelihood-dominant view is fundamentally incomplete. The key challenge in LLM routing is not only how to fit utility from data, but also how to make stable decisions when available evidence is insufficient or unreliable. In these regimes, prior information is indispensable: it initializes utility estimation when observations are scarce, and regularizes it when feedback is noisy or incomplete. Based on this insight, we propose BayesRouter, a lightweight posterior utility inference framework that integrates semantic priors from standardized model descriptions with empirical evidence from retrieved historical interactions via closed-form Bayesian fusion. Experiments on multiple public benchmarks demonstrate that BayesRouter achieves a better accuracy–cost trade-off than strong routing baselines, improves performance in low-data settings, and exhibits stronger robustness under noisy and missing supervision. Our results show that priors are not merely auxiliary in LLM routing, but a key ingredient for stable utility estimation."},"_bibtex":{"value":"@article{\nanonymous2026priors,\ntitle={Priors in {LLM} Routing: From Initialization to Regularization},\nauthor={Anonymous},\njournal={Submitted to Transactions on Machine Learning Research},\nyear={2026},\nurl={https://openreview.net/forum?id=K3xNTJOM1j},\nnote={Under review}\n}"},"title":{"value":"Priors in LLM Routing: From Initialization to Regularization"},"pdf":{"value":"/pdf/18a58a6c75a4fc82810c51f04364eb5ba138a7bc.pdf"},"venueid":{"value":"TMLR/Under_Review"},"assigned_action_editor":{"value":"~Arto_Klami1"}},"version":2},{"content":{"comment":{"value":"## **W2. Difficulty grading & LLM-as-judge bias ------ Empirical difficulty and human validation**\n\n> W2. Difficulty grading misalignment with real-world scenarios. Difficulty is determined solely via an LLM-as-Judge setup, which may introduce model-induced bias; human annotation should be incorporated.\n\nWe have revised Section 3 to reflect the role of the LLM-as-judge and our notion of \"difficulty\". **Bloom's Taxonomy in our benchmark is used only to characterize the cognitive skill of each query and to guide stratified sampling**; it is not used as the definition of difficulty in our analyses. **Difficulty is defined purely empirically**: for each query, we compute the number of models (out of 42) that answer it correctly and sort queries by this empirical difficulty. We then group queries into three bands—hard (≤4/42 models correct), medium (5–19/42), and easy (>19/42)—yielding 23.4%, 29.1%, and 47.5% of the dataset, respectively. As shown in Fig. 3–4, the overall and per-domain difficulty curves are smooth from the hardest to the easiest queries, indicating that the final 8.4k-query subset spans a broad and well-populated difficulty range for LLM routers.\n\nWe also quantified the reliability of LLM as judges via **human annotation**. We conduct a human validation study on 5.4% of the data (450 queries), stratified by category and Bloom level. Eighteen volunteers (undergraduate and graduate students) label Bloom levels under the same protocol as the LLM. We observe 54.9% exact agreement and 76.7% agreement within ±1 Bloom level with the human majority vote, indicating that most disagreements are between adjacent levels in an inherently subjective taxonomy. Since Bloom labels only guide dataset construction, while all router evaluations rely on empirical accuracy across 42 LLMs, any model-induced bias from the LLM-as-judge has **a limited impact** on our reported difficulty grading or conclusions.\n\n---\n\n### **W2.2. More routing demands ------ Agents out-of-scope; coding included; LongBench-v2 added**\n> W2.2. Moreover, the benchmark should expand to more realistic routing demands, such as long-context generation, tool use, code agents, and deep research. In real deployments, achieving good performance often involves orchestrating multiple models collaboratively rather than invoking a single model.\n\n\nThank you for the suggestion. We fully agree that tool use and multi-step \"deep research\" workflows are important aspects when using LLMs. However, **routers in such domains are still underexplored**. Today's routers primarily focus on mapping a user query to one backbone model in a model pool. Therefore, our design is tailored for the mainstream existing routers, but our framework **can be easily extended** when such routers become popular.\n\n\n**Coding agent is included** in our benchmark via queries from LiveCodeBench, which focus on program synthesis and code understanding. These tasks naturally differentiate routers that can reliably select strong coding models from those that cannot, providing a good backbone model in coding-agent settings.\n\nAt the same time, we agree that **long-context generation** is a reasonable and important capability for LLM routers. To address this, we have **incorporated LongBench-v2** into our evaluation suite and report router performance in Table 1. We observe that GPT-5 achieves the highest accuracy on long-context tasks, while Azure-Router and NotDiamond offer slightly lower accuracy at substantially different cost points. Lightweight routers such as NIRT-BERT and MIRT-BERT also remain competitive in accuracy at much lower cost, whereas MLP and KNN attain reasonable accuracy only at very high cost, illustrating that naive over-use of large models is inefficient in the long-context regime. Overall, these results demonstrate that RouterArena can capture meaningful cost–accuracy trade-offs for long-context routing, complementing our core short-context evaluations.\n\n---\n\n### Table 1: RouterArena routers — long-context performance and cost.\n\n| Router       | Acc (%) | Cost / 1K Queries (USD) |\n|-------------|---------|--------------------------|\n| GPT-5       | 71      | 45.70                   |\n| NotDiamond  | 70      | 59.30                   |\n| Azure-Router| 67      | 9.54                    |\n| NIRT-BERT   | 60      | 6.60                    |\n| RouteLLM    | –       | –                       |\n| Graphrouter | 42      | 23.60                   |\n| vLLM-SR     | 42      | 64.80                   |\n| CARROT      | 25      | 4.10                    |\n| RouterDC    | 12      | 3.60                    |\n| MIRT-BERT   | 60      | 7.50                    |\n| MLP         | 65      | 286.70                  |\n| KNN         | 69      | 300.30                  |\n\nNote: RouteLLM couldn't process long context queries due to its text encoder only supports context lengths of 8192 tokens."},"title":{"value":"Thank you! (2/4)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763965947679,"tcdate":1763965425117,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission13871/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission13871/Authors"],"forum":"9HsaIi4ngF","number":8,"license":"CC BY 4.0","cdate":1763965425117,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission13871/-/Official_Comment"],"mdate":1763965947679,"domain":"ICLR.cc/2026/Conference","replyto":"8iXIX7wkBw","id":"0WANBKaMwV","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"TLDR":{"value":"We proposed a comprehensive LLM router evaluation framework, with a new dataset, metrics, and ranking system."},"keywords":{"value":["LLM Router","Evaluation"]},"primary_area":{"value":"datasets and benchmarks"},"abstract":{"value":"Today's LLM ecosystem comprises a wide spectrum of models that differ in size, capability, and cost. No single model is optimal for all scenarios; hence, LLM routers have become essential for selecting the most appropriate model under varying circumstances.\nHowever, the rapid emergence of various routers has led to fragmented evaluation practices and inconsistent metrics, making it difficult to systematically assess progress in this space. To address this problem, we need a comprehensive router comparison and a standardized leaderboard, similar to those available for models. In this work, we introduce RouterArena, the first open platform enabling comprehensive comparison of LLM routers. RouterArena has (1) a principally constructed dataset with broad knowledge domain coverage, (2) distinguishable difficulty levels for each domain, (3) an extensive list of evaluation metrics, and (4) an automated framework for evaluation and leaderboard updates. Leveraging this framework, we have produced the initial leaderboard with detailed metrics comparison.\nFigure1 provides a preview of the leaderboard. The complete framework and the latest router leaderboard are publicly available at https://routeworks.github.io/"},"_bibtex":{"value":"@inproceedings{\nlu2026routerarena,\ntitle={RouterArena: An Open Platform for Comprehensive Comparison of {LLM} Routers},\nauthor={Yifan Lu and Rixin Liu and Jiayi Yuan and Xingqi Cui and Shenrun Zhang and Hongyi Liu and Jiarong Xing},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=9HsaIi4ngF}\n}"},"title":{"value":"RouterArena: An Open Platform for Comprehensive Comparison of LLM Routers"},"pdf":{"value":"/pdf/03e5d4326116c387dbd69c38e194211db0736cda.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"lu|routerarena_an_open_platform_for_comprehensive_comparison_of_llm_routers"},"authorids":{"value":["~Yifan_Lu8","~Rixin_Liu1","~Jiayi_Yuan1","~Xingqi_Cui1","~Shenrun_Zhang1","~Hongyi_Liu5","~Jiarong_Xing1"]},"authors":{"value":["Yifan Lu","Rixin Liu","Jiayi Yuan","Xingqi Cui","Shenrun Zhang","Hongyi Liu","Jiarong Xing"]}},"version":2},{"content":{"comment":{"value":"**Questions:**\n\n1. **What is the computational overhead compared to baselines (training and inference)? Can you quantify the trade-off?**\n\n    The baseline considered in our work is RouteLLM (Ong et al. (2024)), which could be viewed as the PB-only router and the pooled GS-PB router, trained in our numerical experiments.  Compared with these baselines, our meta-router only has one additional step for the CATE estimation through meta-learners. As we have discussed in the first part of our reply to your W4, the overall complexity of meta-router, is essentially the same order as training the corresponding ML algorithm for estimation. Thus the computational complexities of meta-router and baseline routers are of the same order. \n\n\n2. **Can you empirically measure and report the actual bias in preference data vs. gold-standard benchmarks used?**\n\n    Thank you. For HealthBench, We report in the new Figure~1 the histogram of the PB--GS quality differences $(r_i - y_i)_{i=1}^{5000}$.   The sample mean of these differences is substantially below zero, as confirmed by a two-sided t-test yielding a p-value smaller than $2.2 \\times 10^{-16}$, which motivates the training of debiased meta-router. \n\n3. **How does performance scale with model pool size—do you have results for routing among 8–10 models?**\n\n    Thank you. Please see our response to W2 in the first part of our reply.\n\n4. **Could you compare to modern LLM routing baselines, such as confidence-based or learned reward model routing?**\n\n     Thank you. In this work, we focus on comparing Meta-Router with RouteLLM (Ong et al., 2024), which is the method that aligns most directly with our problem formulation and shares the same routing objective. Confidence-based routers and learned reward-model routers address more complex or different real-world settings, making direct, controlled comparisons less appropriate. Nevertheless, we acknowledge these state-of-the-art approaches in the introduction and position our contribution relative to this broader line of work.\n\n5. **Have you performed error analysis outlining types of queries or inputs where Meta-Router fails?**\n\n   Thank you. Please see our response to W3 in the first part of our reply.\n\n6. **Why does T-learner outperform X-learner in Table 2 occasionally—are simpler meta-learners sometimes preferable?**\n\n   Thank you. We do not include T-learner or X-learner in our numerical experiments, as our study focuses on DR-learner and R-learner throughout, and we do not use table-style summaries for the empirical results. Nevertheless, for the general question, the newly added ablation studies in Section 4.1 and the new PRBench experiments in Section 4.2, do show that DR-learner and R-learner–based Meta-Routers often achieve comparable performance, with each occasionally outperforming the other depending on the underlying ML algorithm and the data regime. These findings suggest that the best choice of meta-learner and ML method may be data- and domain-dependent. We therefore recommend that practitioners test multiple combinations of ML models and meta-learners to obtain the strongest routing performance in their specific setting.\n\n7. **How is out-of-distribution or universally poor query detection handled by the method?**\n    \n    Thank you for this important question. Our current work focuses on the in-distribution setting, where deployment queries are drawn from the same population as the GS and PB data used to train Meta-Router. For truly out-of-distribution (OOD) queries, a practical platform may collect responses from both models and obtain PB or GS evaluations for these new queries. This naturally forms an online-learning process in which the system gradually expands the coverage of the in-distribution domain. Integrating such OOD-aware data collection into Meta-Router is an interesting direction for future work, and we have noted this in the revised manuscript.\n\n    On the other hand, “universally poor” queries $q$ correspond to cases where the gold-standard quality gain between the primary and alternative models, i.e., $\\psi(q)$, is close to zero. For such queries, the optimal policy is to route to the cheaper model, and Meta-Router indeed learns to do so by trading off the estimated gain $\\psi(q)$ against the cost parameter $w > 0$."},"title":{"value":"Response to Reviewer uL2P (Part II)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763770178491,"tcdate":1763770178491,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission18138/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission18138/Authors"],"forum":"r0BFucF2dH","number":9,"license":"CC BY 4.0","cdate":1763770178491,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission18138/-/Official_Comment"],"mdate":1763770178491,"domain":"ICLR.cc/2026/Conference","replyto":"CsULiMnpE2","id":"nGLrzjAiuf","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Causal learning","Meta-learner","Large Language Model","query routing"]},"supplementary_material":{"value":"/attachment/9164932b61c808ca473aa568fa79122a32342bda.pdf"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"In language tasks requiring extensive  human-model interaction, the inference cost of large language models (LLMs) can be substantial. To reduce expenses while preserving the quality of the responses, an LLM router selects among candidate models to balance between the expected response quality and the inference cost. A central challenge in router training is the accuracy and accessibility of reliable supervision. Gold-standard data, obtained from domain experts or benchmark labels, provide accurate quality evaluations of LLM responses but are costly and difficult to scale. In contrast, preference-based data, collected via crowdsourcing or LLM-as-a-judge systems, are cheaper and more scalable, yet often biased in reflecting the true quality of responses. We cast the problem of LLM router training with combined Gold-standard and preference-based data into a causal inference framework by viewing the response evaluation mechanism as the treatment assignment. This perspective further reveals that the bias in preference-based data corresponds to the well-known causal estimand: the conditional average treatment effect (CATE). Based on this new perspective, we develop an integrative causal router training framework that corrects preference-data bias, addresses imbalances between two data sources, and improves routing robustness and efficiency.  Numerical experiments demonstrate that our approach delivers more accurate routing and improves the trade-off between cost and quality. Illustrative code to reproduce our main experiment is available at https://github.com/yichistat/Meta-router."},"_bibtex":{"value":"@inproceedings{\nzhang2026metarouter,\ntitle={Meta-Router: Bridging Gold-standard and Preference-based Evaluations in {LLM} Routing},\nauthor={Yichi Zhang and Fangzheng Xie and Shu Yang and Chong Wu},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=r0BFucF2dH}\n}"},"title":{"value":"Meta-Router: Bridging Gold-standard and Preference-based Evaluations in LLM Routing"},"pdf":{"value":"/pdf/b5b19f14eb1393098b973a891e7dde44f0e0ef87.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"zhang|metarouter_bridging_goldstandard_and_preferencebased_evaluations_in_llm_routing"},"authorids":{"value":["~Yichi_Zhang14","~Fangzheng_Xie1","~Shu_Yang4","~Chong_Wu1"]},"authors":{"value":["Yichi Zhang","Fangzheng Xie","Shu Yang","Chong Wu"]}},"version":2},{"content":{"comment":{"value":"Dear reviewer ZgLP,\n\nThanks for your thorough and positive evaluation, as well as the constructive suggestions to further improve the clarity and accessibility of our paper.\n\n---\n\n**Q1**: Could you explain why choosing just SLM for CITER-S ablates on long-term influence of the trajectory chosen by the policy? Is this just hoping that LLM has better representation even if the token prediction is wrong?\n\n**R1**: Thank you for this insightful question. In CITER-S, we ablate the long-term influence by assigning the SLM whenever both the SLM and LLM predict the next token incorrectly, rather than evaluating the impact of this choice on the final output. By selecting the SLM in this scenario, our primary goal is to reduce inference cost. Since the only available information is that both models have made incorrect predictions, we do not assume that the LLM’s output is inherently better than the SLM’s when both are wrong; therefore, there is no justification for preferring the LLM in such cases, and we default to using the SLM.\n\n---\n\n**Q2**: Suggestions to Improve\n\n**R2**: We appreciate the reviewer for suggesting the following improvements and will address them in the revised version.\n\n> More thorough setup of background for readers without a RL background would be helpful. (Eg. Why this specific RL policy method? Most of this setup is clear from the DPO paper, but more setup is helpful along w/ citing DPO earlier in 2.1, though you do this in 2.2)\n\nWe will add an introductory explanation clarifying our use of a reinforcement learning (RL) framework. Specifically, we adopt RL because the router functions as a decision policy model, and this approach enables the router to learn routing strategies that minimize inference cost while maintaining high performance. We will emphasize this motivation in Section 2.1, as mentioned at the beginning of Section 2, to make the rationale behind our approach clearer for readers.\n\n> More description of how you get data for RouteLLM and Co-LLM would be helpful\n\nIn our experiments, all datasets used contain gold-standard labels. Therefore, we adopt the same procedure as RouteLLM for the MMLU dataset to generate ground-truth preference labels across all our datasets. Specifically, we use both the pure LLM and SLM to generate complete responses for each query in the training set, then apply a rule-based approach to extract the final answer from these responses. The preference label is subsequently determined by the correctness of the extracted final answer.\n\nCo-LLM operates as a token-level router, requiring token-level ground-truth next-token labels for training. We follow the same generation trajectory as CITER, which first produces a rationale for query analysis followed by the final answer to the question. For datasets such as GSM8k, MATH, and MMLU-PP, which already provide reasoning steps, we directly use the provided chain-of-thought (CoT) rationales. For datasets like CSQA and ARC-Challenge, which lack such rationales, we employ ChatGPT 3.5 turbo to generate reasoning steps by prompting it to produce a reasoning process for the original question, given the ground-truth answer. We also instruct ChatGPT 3.5 turbo to provide a predicted answer based on the generated rationale, and filter out any rationales whose predicted answers do not match the ground-truth answers.\n\n> More description of (data transformation) x-axis definition in Figure 2-5. Details in Appendix are good, but a 1-2 line high level definition in paper would be good.\n\nWe agree that providing a high-level explanation in the main text will help readers. We will add a one- to two-sentence summary clarifying the definition of data transformation amount. Specifically, it refers to the volume of data, including both model weights and the KV cache, that must be transferred from GPU HBM to the on-chip cache.\n\n> Move Section 3.7 to Appendix if you need space for more clear descriptions. More extensive analysis on this could have been nice.\n\nThank you for the space-saving suggestion. If additional clarification or description is required elsewhere in the paper, we will move Section 3.7 (case study analysis) to the Appendix to accommodate more essential content or further analysis."},"title":{"value":"Response to Reviewer ZgLP (1/2)"}},"parentInvitations":"colmweb.org/COLM/2025/Conference/-/Official_Comment","tmdate":1756226122716,"tcdate":1748920203373,"writers":["colmweb.org/COLM/2025/Conference","colmweb.org/COLM/2025/Conference/Submission1200/Authors"],"signatures":["colmweb.org/COLM/2025/Conference/Submission1200/Authors"],"forum":"nqX9UYW9Af","number":6,"license":"CC BY 4.0","cdate":1748920203373,"readers":["everyone"],"invitations":["colmweb.org/COLM/2025/Conference/Submission1200/-/Official_Comment","colmweb.org/COLM/2025/Conference/-/Edit"],"mdate":1756226122716,"domain":"colmweb.org/COLM/2025/Conference","replyto":"oZNcnrUyYf","id":"lRGkoQZeGE","forumContent":{"TLDR":{"value":"A novel Collaborative Inference with Token-lEvel Routing (CITER) framework that introduces a token-level routing mechanism, enabling efficient collaboration between small and large language models (SLMs & LLMs)."},"venue":{"value":"COLM 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the COLM Code of Ethics on https://colmweb.org/CoE.html"},"keywords":{"value":["collaborative inference","efficient inference","token-level routing","large language model"]},"abstract":{"value":"Large language models have achieved remarkable success in various tasks but suffer from high computational costs during inference, limiting their deployment in resource-constrained applications. To address this issue, we propose a novel Collaborative Inference with Token-lEvel Routing (CITER) framework that enables efficient collaboration between small and large language models (SLMs & LLMs) through a token-level routing strategy. Specifically, CITER routes non-critical tokens to an SLM for efficiency and routes critical tokens to an LLM for generalization quality. We formulate router training as a policy optimization, where the router receives rewards based on both the quality of predictions and the inference costs of generation. This allows the router to learn to predict token-level routing scores and make routing decisions based on both the current token and the future impact of its decisions. To further accelerate the reward evaluation process, we introduce a shortcut which significantly reduces the costs of the reward estimation and improving the practicality of our approach. Extensive experiments on five benchmark datasets demonstrate that CITER reduces the inference costs while preserving high-quality generation, offering a promising solution for real-time and resource-constrained applications."},"_bibtex":{"value":"@inproceedings{\nzheng2025citer,\ntitle={{CITER}: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routing},\nauthor={Wenhao Zheng and Yixiao Chen and Weitong Zhang and Souvik Kundu and Yun Li and Zhengzhong Liu and Eric P. Xing and Hongyi Wang and Huaxiu Yao},\nbooktitle={Second Conference on Language Modeling},\nyear={2025},\nurl={https://openreview.net/forum?id=nqX9UYW9Af}\n}"},"title":{"value":"CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routing"},"pdf":{"value":"/pdf/44adc116c02c1f30777bdd97bc65accd40cb593a.pdf"},"venueid":{"value":"colmweb.org/COLM/2025/Conference"},"paperhash":{"value":"zheng|citer_collaborative_inference_for_efficient_large_language_model_decoding_with_tokenlevel_routing"},"authorids":{"value":["~Wenhao_Zheng4","~Yixiao_Chen2","~Weitong_Zhang2","~Souvik_Kundu2","~Yun_Li7","~Zhengzhong_Liu1","~Eric_Xing1","~Hongyi_Wang1","~Huaxiu_Yao1"]},"author_guide":{"value":"I certify that this submission complies with the submission instructions as described on https://colmweb.org/AuthorGuide.html"},"authors":{"value":["Wenhao Zheng","Yixiao Chen","Weitong Zhang","Souvik Kundu","Yun Li","Zhengzhong Liu","Eric P. Xing","Hongyi Wang","Huaxiu Yao"]}},"version":2},{"content":{"comment":{"value":"For the baseline in vision models, we follow the original paper and use the same DeiT model as the large model. In addition, we introduce a much smaller TinyViT model as the lightweight model for cascading. Notably, we observe that combining two models with individual accuracies in the 81–82% range produces a cascading accuracy of 90%, while simultaneously reducing the total number of activated parameters during inference by a substantial margin, as shown in Table 5. \n[Table 5] Performance comparison of model cascading methods on language tasks.\n| Method                               | Active Param | Top-1 ACC |\n| :-----------------------------------: | :----------: | :--------: |\n| Single model (DeiT-B)      |   86.57      |   81.98    |\n| Single model (TinyViT)      |   11.00               |   81.53          | \n| MoEE (DeiT-B)       |   58.55      |   80.12    |\n| Calibrated confidence cascading (DeiT-B + TinyViT) (Temperature Scaling, 15 bins K=10) |   39.85   |   90.46   |\n| Calibrated confidence cascading (DeiT-B + TinyViT) (Histogram Binning, Training-Free, 15 bins K=10) |   39.85   |   90.46   |\n\n**Q7: “Why use different calibrations for vision and language tasks? Temperature scaling is often used in LLM calibration…”**\n\n**A7**: The proposed framework is universal and compatible with any post-hoc calibration method. We previously adopted Platt scaling for language tasks mainly for practicality, as MMLU and coding tasks provide verifiable binary rewards. In fact, any calibration technique can be applied to both vision and language models within the proposed framework.\nAs shown in Table 1 and 2, we evaluate three calibration methods: temperature scaling, Platt scaling, and histogram binning, in terms of both ECE and cascading performance. All three methods exhibit comparable performance, which we attribute to the favorable properties of model confidence (approximate monotonicity and generalization).\n\n[1] RouteLLM: Learning to Route LLMs with Preference Data. ICLR 2025\n\n[2] Confident Learning: Estimating Uncertainty in Dataset Labels. JAIR\n\n[3] Efficient Data Driven Mixture-of-Expert Extraction from Trained Networks. CVPR 2025\n\n[4] WAIC, but Why? Generative Ensembles for Robust Anomaly Detection\n\n[5] Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers. ICML 2001"},"title":{"value":"Author Response to Reviewer nfCf (3/3)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764383344447,"tcdate":1764383105096,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission8659/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission8659/Authors"],"forum":"imzDS4N8CW","number":12,"license":"CC BY 4.0","cdate":1764383105096,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission8659/-/Official_Comment"],"mdate":1764383344447,"domain":"ICLR.cc/2026/Conference","replyto":"riJwWXcv7u","id":"OAgCPSGy52","forumContent":{"TLDR":{"value":"a general framework that enables a model to know when it does not know and to make use of this property"},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Model Calibration","Model Cascades","Data Cleaning"]},"primary_area":{"value":"alignment, fairness, safety, privacy, and societal considerations"},"abstract":{"value":"When a model knows when it does not know, many possibilities emerge. The first question is how to enable a model to recognize that it does not know. A promising approach is to use confidence, computed from the model’s internal signals, to reflect its ignorance. Prior work in specific domains has shown that calibration can provide reliable confidence estimates. In this work, we propose a simple, effective, and universal training-free method that applies to both vision and language models, performing model calibration, cascading, and data cleaning to better exploit a model's ability to recognize when it does not know. We first highlight two key empirical observations: higher confidence corresponds to higher accuracy within a single model, and models calibrated on the validation set remain calibrated on a held-out test set. These findings empirically establish the reliability and comparability of calibrated confidence. Building on this, we introduce two applications: 1. Model cascading with calibrated advantage routing and 2. Data cleaning based on mixture of experts. Using the routing signal derived from the comparability of calibrated confidences, we cascade large and small models to improve efficiency with almost no compromise in accuracy, and we further cascade two large state-of-the-art models to achieve performance beyond either model alone. Leveraging multiple experts and their calibrated confidences, we design a simple yet effective data-cleaning method that balances precision and detection rate to identify mislabeled samples in ImageNet and Massive Multitask Language Understanding (MMLU) datasets. Our results demonstrate that enabling models to recognize when they do not know is a practical step toward more efficient, reliable, and trustworthy AI."},"_bibtex":{"value":"@misc{\nhao2026when,\ntitle={When Models Know When They Do Not Know: Calibration, Cascading and Cleaning},\nauthor={Chenjie Hao and Weyl Lu and Yuko Ishiwaka and ZENGYI LI and Weier Wan and Yubei Chen},\nyear={2026},\nurl={https://openreview.net/forum?id=imzDS4N8CW}\n}"},"title":{"value":"When Models Know When They Do Not Know: Calibration, Cascading and Cleaning"},"pdf":{"value":"/pdf/15f153d32eedccd6d5def2e3c98e8ad6d40eaffb.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"hao|when_models_know_when_they_do_not_know_calibration_cascading_and_cleaning"},"authorids":{"value":["~Chenjie_Hao1","~Weyl_Lu1","~Yuko_Ishiwaka1","~ZENGYI_LI1","~Weier_Wan1","~Yubei_Chen1"]},"authors":{"value":["Chenjie Hao","Weyl Lu","Yuko Ishiwaka","ZENGYI LI","Weier Wan","Yubei Chen"]}},"version":2},{"content":{"comment":{"value":"### 5.Calibration methods. \n\nReviewers (R1, R2, R4) question the use of task-specific calibration methods, and suggest a more thorough exploration of different calibration techniques.\n\nThe proposed framework is universal and compatible with any post-hoc calibration method. We previously adopted Platt scaling for language tasks mainly for practicality, as MMLU and coding tasks provide verifiable binary rewards. In fact, any calibration technique can be applied to both vision and language models within the proposed framework.\n\nTo make this point clearer, we can further include a training-free, non-parametric calibration method—histogram binning [6] — as an additional comparison. The following results evaluate three calibration methods: temperature scaling, Platt scaling, and histogram binning, in terms of both ECE and cascading performance. All three methods exhibit comparable performance, which we attribute to the favorable properties of model confidence (approximate monotonicity and generalization).\n\n[Table 8] ECE of different calibration methods.\n| ECE / Dataset         |  MMLU  | MBPP |  GSM8K  |\n| :------------------- | :----: | :-----------: | :-----: |\n| Histogram Binning    | 0.0239 |    0.0719     | 0.0536  |\n| Temperature Scaling  | 0.0768 |    0.1635     | 0.1054  |\n| Platt Scaling        | 0.0203 |    0.0696     | 0.0731  |\n\n[Table 9] APGR of model cascading of different calibration methods.\n| AGPR / Dataset        |  MMLU  |  MBPP  |  GSM8K |\n| :------------------- | :----: | :----: | :----: |\n| Histogram Binning    | 0.7006 | 0.6609 | 0.7489 |\n| Temperature Scaling  | 0.7013 | 0.6772 | 0.7593 |\n| Platt Scaling        | 0.7013 | 0.6897 | 0.7496 |\n\n\n\n[1] RouteLLM: Learning to Route LLMs with Preference Data. ICLR 2025\n\n[2] Confident Learning: Estimating Uncertainty in Dataset Labels. JAIR\n\n[3] Efficient Data Driven Mixture-of-Expert Extraction from Trained Networks. CVPR 2025\n\n[4] Do Deep Generative Models Know What They Don't Know? ICLR 2019\n\n[5] A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. ACM Transactions on Information Systems Jan 2024\n\n[6] Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers. ICML 2001\n\n[7] Why Normalizing Flows Fail to Detect Out-of-Distribution Data? NeurIPS 2020\n\n[8] WAIC, but Why? Generative Ensembles for Robust Anomaly Detection\n\n[9] Input complexity and out-of-distribution detection with likelihood-based generative models. ICLR 2020\n\n**Next, we reply to each reviewer to address the specific questions.**"},"title":{"value":"General reply (5/5)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764364458914,"tcdate":1764364378883,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission8659/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission8659/Authors"],"forum":"imzDS4N8CW","number":5,"license":"CC BY 4.0","cdate":1764364378883,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission8659/-/Official_Comment"],"mdate":1764364458914,"domain":"ICLR.cc/2026/Conference","replyto":"vm9Mzzey21","id":"bVi93fdw04","forumContent":{"TLDR":{"value":"a general framework that enables a model to know when it does not know and to make use of this property"},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Model Calibration","Model Cascades","Data Cleaning"]},"primary_area":{"value":"alignment, fairness, safety, privacy, and societal considerations"},"abstract":{"value":"When a model knows when it does not know, many possibilities emerge. The first question is how to enable a model to recognize that it does not know. A promising approach is to use confidence, computed from the model’s internal signals, to reflect its ignorance. Prior work in specific domains has shown that calibration can provide reliable confidence estimates. In this work, we propose a simple, effective, and universal training-free method that applies to both vision and language models, performing model calibration, cascading, and data cleaning to better exploit a model's ability to recognize when it does not know. We first highlight two key empirical observations: higher confidence corresponds to higher accuracy within a single model, and models calibrated on the validation set remain calibrated on a held-out test set. These findings empirically establish the reliability and comparability of calibrated confidence. Building on this, we introduce two applications: 1. Model cascading with calibrated advantage routing and 2. Data cleaning based on mixture of experts. Using the routing signal derived from the comparability of calibrated confidences, we cascade large and small models to improve efficiency with almost no compromise in accuracy, and we further cascade two large state-of-the-art models to achieve performance beyond either model alone. Leveraging multiple experts and their calibrated confidences, we design a simple yet effective data-cleaning method that balances precision and detection rate to identify mislabeled samples in ImageNet and Massive Multitask Language Understanding (MMLU) datasets. Our results demonstrate that enabling models to recognize when they do not know is a practical step toward more efficient, reliable, and trustworthy AI."},"_bibtex":{"value":"@misc{\nhao2026when,\ntitle={When Models Know When They Do Not Know: Calibration, Cascading and Cleaning},\nauthor={Chenjie Hao and Weyl Lu and Yuko Ishiwaka and ZENGYI LI and Weier Wan and Yubei Chen},\nyear={2026},\nurl={https://openreview.net/forum?id=imzDS4N8CW}\n}"},"title":{"value":"When Models Know When They Do Not Know: Calibration, Cascading and Cleaning"},"pdf":{"value":"/pdf/15f153d32eedccd6d5def2e3c98e8ad6d40eaffb.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"hao|when_models_know_when_they_do_not_know_calibration_cascading_and_cleaning"},"authorids":{"value":["~Chenjie_Hao1","~Weyl_Lu1","~Yuko_Ishiwaka1","~ZENGYI_LI1","~Weier_Wan1","~Yubei_Chen1"]},"authors":{"value":["Chenjie Hao","Weyl Lu","Yuko Ishiwaka","ZENGYI LI","Weier Wan","Yubei Chen"]}},"version":2},{"content":{"comment":{"value":"We deeply appreciate your insightful comment and prompt response. We completely agree that the approach taken by prior work is cleaner. That said, we believe there are still some important practical reasons for jointly considering fine-tuning and draft learning, as detailed below.\n\n- **Mismatch between fine-tuned LLM and pre-trained drafter**. After fine-tuning a LLM, the pre-trained drafter may become outdated, especially when there is a significant distribution shift in the fine-tuned data (for example, consider a private dataset in finance). As highlighted in [r1], the gap between the acceptance rate of a static drafter and a domain-specific drafter can be as large as 30% or more. This serves as a strong motivation for simultaneously fine-tuning the LLM and the drafter;\n- **Practices in modern LLM serving**. In many industrial LLM systems, requests are typically not handled by a single centralized pre-trained model. Instead, they are routed to multiple domain-adapted models in the backend to ensure optimal response quality (via a request router similar to [r2]). To accelerate these domain-specific LLMs, jointly training a domain-specific drafter that aligns well with the LLM seems to be a natural choice;\n- **Expanding the applicability of speculative decoding methods**. Not all researchers or practitioners have the resources to train a drafter on a pre-trained dataset. Our work on successfully applying speculative decoding in fine-tuning scenarios makes this technique  more accessible to users with limited resource (e.g. 1~2 A10 GPUs).\n\nIn summary, the joint consideration of fine-tuning and efficient drafting has important implications for both *industry* and *academia*. We hope these explanation address your concerns. At the same time, we would be eager to hear any further thoughts or insights you may have on this. Thank you!\n\n\n*References*\n\n[r1] liu et. al. Online Speculative Decoding. ICML 2024.\n\n[r2] Ong et.al. RouteLLM: Learning to Route LLMs with Preference Data. arxiv 2406.18665."},"title":{"value":"Clarifying the Motivation for Jointly Considering Fine-Tuning and Drafter Training"}},"tmdate":1732868269911,"tcdate":1732797037714,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission10295/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission10295/Authors"],"forum":"gfDbD1MRYk","number":7,"license":"CC BY 4.0","cdate":1732797037714,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission10295/-/Official_Comment"],"mdate":1732868269911,"domain":"ICLR.cc/2025/Conference","replyto":"2qrtZzFUEd","id":"tdfbgfWaYf","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"TLDR":{"value":"A semi-autoregressive paradigm for accelerating LLM inference, offering both efficiency and accuracy"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["efficient inference","dependence modeling","non-autoregressive models","LLMs","speculative decoding"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"Inference in large language models (LLMs) is often slow due to their autoregressive nature. \nIn this work, we formulate a semi-autoregressive decoding paradigm for LLMs that delegates part of the expensive computation from the original large model to a smaller, more efficient autoregressive model. The core of our design lies in the separate modeling of token dependencies, where the large model handles long-term dependencies on distant tokens, while the smaller model addresses short-term dependencies on recent tokens. When employed as a draft model in speculative decoding, our method allows for substantial reuse of computation in the LLM without missing any token dependencies, thereby striking a good balance between draft quality and drafting speed. Experiments on text summarization, medical QA, code generation, and mathematical reasoning tasks demonstrates the efficacy of our method."},"_bibtex":{"value":"@misc{\nchen2025semiautoregressive,\ntitle={Semi-autoregressive Decoding for Efficient {LLM} Inference},\nauthor={Yanzhi Chen and Menglin Xia and Wenbo Gong and Ankur Mallick and Srikant Bharadwaj and Metod Jazbec and Shoaib Ahmed Siddiqui and Adrian Weller and Victor R{\\\"u}hle},\nyear={2025},\nurl={https://openreview.net/forum?id=gfDbD1MRYk}\n}"},"title":{"value":"Semi-autoregressive Decoding for Efficient LLM Inference"},"pdf":{"value":"/pdf/dcca731cb3cb692e090d222b9b55ce466c0a6ae8.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"chen|semiautoregressive_decoding_for_efficient_llm_inference"},"authorids":{"value":["~Yanzhi_Chen1","~Menglin_Xia1","~Wenbo_Gong1","~Ankur_Mallick1","~Srikant_Bharadwaj1","~Metod_Jazbec1","~Shoaib_Ahmed_Siddiqui1","~Adrian_Weller1","~Victor_Rühle1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Yanzhi Chen","Menglin Xia","Wenbo Gong","Ankur Mallick","Srikant Bharadwaj","Metod Jazbec","Shoaib Ahmed Siddiqui","Adrian Weller","Victor Rühle"]}},"version":2},{"content":{"summary":{"value":"This paper proposes a reinforcement learning framework for collaborative device-cloud LLM inference where lightweight on-device models learn to autonomously decide when to offload queries to powerful cloud LLMs. Unlike existing approaches using external routers, the method integrates routing capability directly into post-training through a reward maximization problem with hierarchical rewards. The authors develop a Group-Adaptive Policy Gradient (GAPG) algorithm featuring unbiased group-level gradient estimation and adaptive prompt filtering to balance local problem-solving with cloud assistance. Experiments across mathematical benchmarks demonstrate the approach consistently outperforms baselines, maintains stable training, and significantly narrows the performance gap to full cloud LLM usage while respecting offloading constraints."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"Please see weaknesses."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"+ This work elegantly integrates routing optimization directly into the post-training process, eliminating the need for separate external routers. This unified approach allows the on-device LLM to simultaneously improve its problem-solving capabilities while learning judicious offloading strategies.\n+ The group-level policy gradient addresses GRPO's limitations in collaborative settings, while adaptive prompt filtering effectively enforces cloud usage constraints, preventing the model from over-relying on external assistance during training.\n+ The evaluation demonstrates performance across multiple model sizes (1B to 3B parameters) and diverse mathematical benchmarks."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- The work lacks clear motivation and validation for why routing capability should be embedded in the on-device LLM with a first-device-then-cloud cascade way rather than using separate routers for directly choosing which LLM. The optimization goal appears vague—it achieves significantly lower accuracy than cloud LLM alone (cf. Table 1), yet doesn't explicitly quantify or optimize the cost trade-off between on-device computation and cloud invocation, making it difficult to assess whether the approach provides genuine practical benefits. In particular, accuracy should be the first objective in real-world deployment.\n\n- The core technical contribution is incremental. The group-level policy gradient (Proposition 3.1) closely resembles existing GRPO with minor modifications. The only novel component—adaptive prompt filtering—is a simple heuristic that selects training prompts based on response correctness ratios. This lacks theoretical justification and appears to be an ad-hoc empirical design without principled foundations for why this specific filtering strategy is optimal.\n\n- Evaluation is restricted exclusively to mathematical reasoning tasks (Countdown and MATH benchmarks). The paper completely ignores other important domains like general question answering, commonsense reasoning, coding, summarization, or multi-turn dialogue. This severely limits claims about generalizability, as mathematical problems may have unique characteristics that don't transfer to other collaborative inference scenarios where routing decisions differ fundamentally.\n\n- The paper compares against only one routing method (Task-Tuning&Router using DeBERTa) and naive random offloading, missing numerous recent works in LLM routing and cascade systems. Notable omissions include FrugalGPT, RouteLLM variants, context-aware cascading policies, and ensemble-based routing methods cited in related work but never experimentally compared, weakening claims of superiority.\n\n- The paper provides no ablation analysis to isolate the contribution of individual components. Critical questions remain unanswered: How much does hierarchical reward design contribute versus the gradient estimator? What's the impact of different reward weights (αₐ, αc, αf)? How sensitive is performance to the filtering ratio ρ? Without ablations, it's impossible to understand which design choices actually matter or whether simpler alternatives might work equally well."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762919107366,"tcdate":1761797215310,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission6850/Reviewer_Xpht"],"signatures":["ICLR.cc/2026/Conference/Submission6850/Reviewer_Xpht"],"forum":"SwHnok3JtD","number":2,"license":"CC BY 4.0","cdate":1761797215310,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission6850/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762919107366,"domain":"ICLR.cc/2026/Conference","replyto":"SwHnok3JtD","id":"7ReKF9CHhr","forumContent":{"TLDR":{"value":"We propose a unified post-training framework that integrates routing optimization, enabling the on-device LLM to improve its problem-solving ability while learning routing strategies."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Large Language Models","Device-Cloud Collaboration","Post-Training","Routing Optimization","Reinforcement Learning"]},"supplementary_material":{"value":"/attachment/0abec1d957d5dfe99e2938019c58c05fb162a7f5.zip"},"primary_area":{"value":"other topics in machine learning (i.e., none of the above)"},"abstract":{"value":"Device-cloud collaboration has emerged as a promising paradigm for deploying large language models (LLMs), combining the efficiency of lightweight on-device inference with the superior performance of powerful cloud LLMs. An essential problem in this scenario lies in deciding whether a given query is best handled locally or delegated to the cloud. Existing approaches typically rely on external routers, implemented as binary classifiers, which often struggle to determine task difficulty from the prompt's surface pattern. To address these limitations, we propose a framework where the on-device LLM makes routing decisions at the end of its solving process, with this capability instilled through post-training. In particular, we formulate a reward maximization problem with carefully designed rewards that encourage effective problem solving and judicious offloading to the cloud. To solve this problem, we develop a group-adaptive policy gradient algorithm, featuring a group-level policy gradient, designed to yield an unbiased gradient estimator of the reward, and adaptive prompt filtering, developed to provide complementary learning signals. Extensive experiments across models and benchmarks show that the proposed methodology consistently outperforms existing baselines and significantly narrows the gap to full cloud LLM performance."},"_bibtex":{"value":"@misc{\nfang2026collaborative,\ntitle={Collaborative Device-Cloud {LLM} Inference through Reinforcement Learning},\nauthor={Wenzhi Fang and Dong-Jun Han and Liangqi Yuan and Christopher Brinton},\nyear={2026},\nurl={https://openreview.net/forum?id=SwHnok3JtD}\n}"},"title":{"value":"Collaborative Device-Cloud LLM Inference through Reinforcement Learning"},"pdf":{"value":"/pdf/b0a3698c6181d5c8e4245b48f5ea8700da6b0ff7.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"fang|collaborative_devicecloud_llm_inference_through_reinforcement_learning"},"authorids":{"value":["~Wenzhi_Fang1","~Dong-Jun_Han1","~Liangqi_Yuan1","~Christopher_Brinton1"]},"authors":{"value":["Wenzhi Fang","Dong-Jun Han","Liangqi Yuan","Christopher Brinton"]}},"version":2},{"content":{"summary":{"value":"Authors propose a method to aggregate pretrained LLM decoders, Mixture of Thoughts (MoT). The idea is to let a primary expert “look at” the hidden states of other active experts and use that information thought cross attention. For each query, a router selects the top-K experts and designates a primary expert. The authors insert interaction layers that project each model’s hidden states into a shared latent space; the primary then performs cross-attention over the selected peers’ latents. All experts are frozen; only the router and interaction layers are trained with a joint objective. \n\nWhile I like the idea of “merging” pertained models, there are several drawbacks in the proposed implementation as well as in the evaluation setup (see Weaknesses). I think that the marginal improvements on downstream tasks do not justify the overhead and complexity of the proposed approach."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"Inference is reported to be conducted on an RTX A6000 GPU with 48 GB of memory. The authors state that $k = 3$, which implies that three $7$B models (approximately $21$B active parameters) are used simultaneously. Assuming bfloat16 precision, the model weights alone would require roughly 42GB of memory, leaving very limited space for the rest (KV caches, router, etc). This suggests that inference must have been distributed across multiple devices or involved CPU offloading.\nCould the authors please clarify the inference setup, specifically:\n- whether experts parallelism was used,\n- whether any activation checkpointing, quantization, or CPU offloading was employed \n\nto fit the models into 48 GB?"},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"- The paper introduces a new way to enable communication between pretrained LLM decoders by exchanging latent representations rather than final outputs. This latent-level interaction is conceptually interesting and bridges ideas from model ensembles and mixture-of-experts architectures. \n- It is well written and easy to follow."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The main drawback is that injecting interaction layers makes inference synchronous across experts. One of the main benefits of using a pool of models is the ability to aggregate their results asynchronously, but this design removes that advantage.\n\nLet us consider two cases:\n1. On device inference with limited memory (we can fit only one model)\nIn prior works such as RouteLLM and Averagers, we can process the prompt, select experts, and then sequentially: load an expert, perform the computation, save the output, and repeat this process k times. In the proposed approach, however, this becomes problematic because intermediate hidden states must be stored to allow the primary expert to attend to them later. This introduces additional memory reads and writes, which can further stress an already memory-bound inference setup.\n\n2. All experts placed on different devices:\nIn this scenario, interaction layers require a gather operation by the primary expert. While the exchanged hidden states are small and the bandwidth cost is minimal (even over Ethernet I suppose), this design introduces a synchronization point. In heterogeneous setups, where devices or models have varying latencies, these synchronization points could accumulate and lead to noticeable overhead.\n\nI would appreciate seeing a table similar to Table 10 in the Appendix, but focusing on overhead rather than parameter counts. In particular, it would be helpful to include:\n\n- communication overhead when models are placed on different devices with different type of interconnect\n- memory overhead\n- latency for generation ( authors provide average time across downstream tasks but I would be more interested in latency for prompts of different length)\n\nA table summarizing these measurements would make it much easier to understand the trade-offs.\nIf the authors can explicitly demonstrate that generation speed is not hindered by the additional communication and provide a detailed analysis of memory and bandwidth requirements (see Questions), the proposed method would be significantly more convincing."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762918624697,"tcdate":1760698039483,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission6329/Reviewer_Q4vY"],"signatures":["ICLR.cc/2026/Conference/Submission6329/Reviewer_Q4vY"],"forum":"x9tSyvnD8o","number":1,"license":"CC BY 4.0","cdate":1760698039483,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission6329/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762918624697,"domain":"ICLR.cc/2026/Conference","replyto":"x9tSyvnD8o","id":"9K30upPGMn","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"TLDR":{"value":"Multi-LLM collaboration via latent-space interactions"},"keywords":{"value":["Large Language Models","LLM","LLM routing","LLM aggregation","LLM ensemble"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Open-source Large Language Models (LLMs) increasingly specialize by domain (e.g., math, code, general reasoning), motivating systems that leverage complementary strengths across models. Prior multi-LLM approaches either (i) route a query to one or a few experts and generate independently, (ii) aggregate outputs from each model via costly multi-turn exchanges, or (iii) fuse weights into a single model—typically requiring architectural homogeneity. We introduce Mixture of Thoughts (MoT), a simple method for latent-level collaboration among heterogeneous experts under a global routing scheme. For each query, a lightweight router selects top-$K$ experts and designates a primary expert; uniformly placed interaction layers project hidden states into a shared latent space where the primary expert performs cross-attention over its active (selected) peers. Pre-trained experts remain frozen; only the router and the lightweight interaction layers are trained with a novel joint training objective that improves both the expert selection and inter-expert collaboration. Across five in-distribution (ID) and three out-of-distribution (OOD) benchmarks, MoT surpasses the current routing and aggregation-based state-of-the-art, Avengers, by $+0.38\\%$ and $+2.92\\%$, respectively. Further, MoT significantly outperforms the best-performing single model. It achieves this with single-pass inference, runtime comparable to routing baselines, and none of the overheads of iterative aggregation. MoT offers a simple latent-space mechanism for combining heterogeneous LLMs, a practical step toward broader multi-LLM collaboration. Our code is publicly available at \\url{https://anonymous.4open.science/r/mot-5B4B/}."},"_bibtex":{"value":"@misc{\nfein-ashley2026mixture,\ntitle={Mixture of Thoughts: Learning to Aggregate What Experts Think, Not Just What They Say},\nauthor={Jacob Fein-Ashley and Dhruv Parikh and Rajgopal Kannan and Viktor Prasanna},\nyear={2026},\nurl={https://openreview.net/forum?id=x9tSyvnD8o}\n}"},"title":{"value":"Mixture of Thoughts: Learning to Aggregate What Experts Think, Not Just What They Say"},"pdf":{"value":"/pdf/a857f7bf89f375d5f31ac949d50eea649304f16b.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"feinashley|mixture_of_thoughts_learning_to_aggregate_what_experts_think_not_just_what_they_say"},"authorids":{"value":["~Jacob_Fein-Ashley1","~Dhruv_Parikh1","~Rajgopal_Kannan1","~Viktor_Prasanna1"]},"authors":{"value":["Jacob Fein-Ashley","Dhruv Parikh","Rajgopal Kannan","Viktor Prasanna"]}},"version":2},{"content":{"comment":{"value":"[Table 6] APGR comparison of model cascading methods on language tasks.\n| Method                               |   MMLU   |  GSM8K  |\n| :-----------------------------------: | :------: | :-----: |\n| Random Baseline                      |  0.5000  | 0.5000  |\n| Matrix Factorization (Pretrained)     |  0.5418  | 0.5672  |\n| Matrix Factorization (Trained)        |  0.5267  | 0.6234  |\n| Matrix Factorization (Finetuned)      |  0.5497  | 0.5718  |\n| Calibrated confidence cascading (Histogram Binning, 15 bins)      |  0.7006  | 0.7489  |\n| Calibrated confidence cascading (Temperature Scaling, 15 bins)    |  0.7013  | 0.7593  |\n| Calibrated confidence cascading (Platt Scaling, 15 bins)          |  0.7013  | 0.7496  |\n\n[Table 7] Performance comparison of model cascading methods on language tasks.\n| Method                               | Active Param | Top-1 ACC |\n| :-----------------------------------: | :----------: | :--------: |\n| Single model (DeiT-B)      |   86.57      |   81.98    |\n| Single model (TinyViT)      |   11.00               |   81.53          | \n| MoEE (DeiT-B)       |   58.55      |   80.12    |\n| Calibrated confidence cascading (DeiT-B + TinyViT) (Temperature Scaling, 15 bins K=10) |   39.85   |   90.46   |\n| Calibrated confidence cascading (DeiT-B + TinyViT) (Histogram Binning, Training-Free, 15 bins K=10) |   39.85   |   90.46   |\n\n**Q4: “Why don't try some other calibration methods?”**\n\n**A4**: Good suggestion! We further include a training-free, non-parametric calibration method—histogram binning [6] — as an additional comparison. The following results evaluate three calibration methods: temperature scaling, Platt scaling, and histogram binning, in terms of both ECE and cascading performance. All three methods exhibit comparable performance, which we attribute to the favorable properties of model confidence (approximate monotonicity and generalization).\n[Table 8] ECE of different calibration methods.\n| ECE / Dataset         |  MMLU  | MBPP |  GSM8K  |\n| :------------------- | :----: | :-----------: | :-----: |\n| Histogram Binning    | 0.0239 |    0.0719     | 0.0536  |\n| Temperature Scaling  | 0.0768 |    0.1635     | 0.1054  |\n| Platt Scaling        | 0.0203 |    0.0696     | 0.0731  |\n[Table 9] APGR of model cascading of different calibration methods.\n| AGPR / Dataset        |  MMLU  |  MBPP  |  GSM8K |\n| :------------------- | :----: | :----: | :----: |\n| Histogram Binning    | 0.7006 | 0.6609 | 0.7489 |\n| Temperature Scaling  | 0.7013 | 0.6772 | 0.7593 |\n| Platt Scaling        | 0.7013 | 0.6897 | 0.7496 |\n\n[1] RouteLLM: Learning to Route LLMs with Preference Data. ICLR 2025\n\n[2] Confident Learning: Estimating Uncertainty in Dataset Labels. JAIR\n\n[3] Efficient Data Driven Mixture-of-Expert Extraction from Trained Networks. CVPR 2025\n\n[4] Do Deep Generative Models Know What They Don't Know? ICLR 2019\n\n[5] A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. ACM Transactions on Information Systems Jan 2024\n\n[6] Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers. ICML 2001\n\n[7] Why Normalizing Flows Fail to Detect Out-of-Distribution Data? NeurIPS 2020\n\n[8] WAIC, but Why? Generative Ensembles for Robust Anomaly Detection\n\n[9] Input complexity and out-of-distribution detection with likelihood-based generative models. ICLR 2020"},"title":{"value":"Author Response to Reviewer tASX (3/3)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764387023396,"tcdate":1764386941279,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission8659/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission8659/Authors"],"forum":"imzDS4N8CW","number":17,"license":"CC BY 4.0","cdate":1764386941279,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission8659/-/Official_Comment"],"mdate":1764387023396,"domain":"ICLR.cc/2026/Conference","replyto":"9jFncytBG5","id":"vCNGHHdaVk","forumContent":{"TLDR":{"value":"a general framework that enables a model to know when it does not know and to make use of this property"},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Model Calibration","Model Cascades","Data Cleaning"]},"primary_area":{"value":"alignment, fairness, safety, privacy, and societal considerations"},"abstract":{"value":"When a model knows when it does not know, many possibilities emerge. The first question is how to enable a model to recognize that it does not know. A promising approach is to use confidence, computed from the model’s internal signals, to reflect its ignorance. Prior work in specific domains has shown that calibration can provide reliable confidence estimates. In this work, we propose a simple, effective, and universal training-free method that applies to both vision and language models, performing model calibration, cascading, and data cleaning to better exploit a model's ability to recognize when it does not know. We first highlight two key empirical observations: higher confidence corresponds to higher accuracy within a single model, and models calibrated on the validation set remain calibrated on a held-out test set. These findings empirically establish the reliability and comparability of calibrated confidence. Building on this, we introduce two applications: 1. Model cascading with calibrated advantage routing and 2. Data cleaning based on mixture of experts. Using the routing signal derived from the comparability of calibrated confidences, we cascade large and small models to improve efficiency with almost no compromise in accuracy, and we further cascade two large state-of-the-art models to achieve performance beyond either model alone. Leveraging multiple experts and their calibrated confidences, we design a simple yet effective data-cleaning method that balances precision and detection rate to identify mislabeled samples in ImageNet and Massive Multitask Language Understanding (MMLU) datasets. Our results demonstrate that enabling models to recognize when they do not know is a practical step toward more efficient, reliable, and trustworthy AI."},"_bibtex":{"value":"@misc{\nhao2026when,\ntitle={When Models Know When They Do Not Know: Calibration, Cascading and Cleaning},\nauthor={Chenjie Hao and Weyl Lu and Yuko Ishiwaka and ZENGYI LI and Weier Wan and Yubei Chen},\nyear={2026},\nurl={https://openreview.net/forum?id=imzDS4N8CW}\n}"},"title":{"value":"When Models Know When They Do Not Know: Calibration, Cascading and Cleaning"},"pdf":{"value":"/pdf/15f153d32eedccd6d5def2e3c98e8ad6d40eaffb.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"hao|when_models_know_when_they_do_not_know_calibration_cascading_and_cleaning"},"authorids":{"value":["~Chenjie_Hao1","~Weyl_Lu1","~Yuko_Ishiwaka1","~ZENGYI_LI1","~Weier_Wan1","~Yubei_Chen1"]},"authors":{"value":["Chenjie Hao","Weyl Lu","Yuko Ishiwaka","ZENGYI LI","Weier Wan","Yubei Chen"]}},"version":2},{"content":{"comment":{"value":"Dear reviewer xyvj,\n\nWe sincerely appreciate it for your thoughtful feedback and suggestions. We address the main concerns and questions below.\n\n---\n\n**Q1**: The reliance on in-domain ground-truth data for the tuning of the router model makes CITER less generalizable compared to methods like speculative decoding.\n\n**R1**: We appreciate the reviewer’s observation regarding the generalizability of CITER. We acknowledge that our current approach relies on in-domain ground-truth data for training the router, which may limit its direct applicability to entirely new domains compared to approaches like speculative decoding that do not require explicit supervision. However, the router’s input features, specifically, the hidden states from the SLM, are generic representations, which suggests the potential for generalization across different domains. To explore this, we conducted additional experiments where we trained our router on a mixture of all the datasets used in our study, including CSQA, ARC-Challenge, MMLU-PP, GSM8k, and MATH, and then evaluated its performance on the OpenBookQA dataset. The results are presented in the table below.\n\nDue to the limitations of the rebuttal system, we are unable to include figures or plots in our response. Therefore, we present our results in table format, where each row corresponds to a different method, each column represents a specific value of the x-axis variable (e.g., data transfer amount), and each cell shows the accuracy achieved under that setting. To facilitate a fair and direct comparison between methods, we performed linear interpolation on the original data to obtain accuracy values for all methods at the same set of x-axis points. This approach ensures that our comparisons are consistent and transparent across all methods and conditions.\n\n**Table R1**: Performance comparison between CITER and our baselines on the OpenBookQA dataset. Higher values indicate better performance. The best result is **bolded**.\n| Accuracy (%) \\ Data Transfer Amount (GB) | 7.1  | 13.4     | 50.0     | 100.0    | 150.0    | 179.3    | 191.1 |\n| ---------------------------------------- | ---- | -------- | -------- | -------- | -------- | -------- | ----- |\n| SLM                                      | 56.6 | /        | /        | /        | /        | /        | /     |\n| LLM                                      | /    | /        | /        | /        | /        | /        | 91.1  |\n| Speculative Decoding                     | /    | /        | /        | /        | /        | **91.1** | /     |\n| RouteLLM                                 | /    | 58.0     | 64.7     | 73.8     | 83.6     | 88.8     | /     |\n| Co-LLM                                   | /    | 59.6     | 66.9     | 76.7     | 86.2     | 89.4     | /     |\n| CITER (Ours)                             | /    | **60.0** | **67.3** | **79.1** | **89.8** | 90.6     | /     |\n\nThese results demonstrate that CITER continues to outperform all baselines on the new dataset, highlighting the generalizability of our approach."},"title":{"value":"Response to Reviewer xyvj (1/2)"}},"parentInvitations":"colmweb.org/COLM/2025/Conference/-/Official_Comment","tmdate":1756226122340,"tcdate":1748920534604,"writers":["colmweb.org/COLM/2025/Conference","colmweb.org/COLM/2025/Conference/Submission1200/Authors"],"signatures":["colmweb.org/COLM/2025/Conference/Submission1200/Authors"],"forum":"nqX9UYW9Af","number":11,"license":"CC BY 4.0","cdate":1748920534604,"readers":["everyone"],"invitations":["colmweb.org/COLM/2025/Conference/Submission1200/-/Official_Comment","colmweb.org/COLM/2025/Conference/-/Edit"],"mdate":1756226122340,"domain":"colmweb.org/COLM/2025/Conference","replyto":"4hPsqQCDlu","id":"5FFZW2Qi9q","forumContent":{"TLDR":{"value":"A novel Collaborative Inference with Token-lEvel Routing (CITER) framework that introduces a token-level routing mechanism, enabling efficient collaboration between small and large language models (SLMs & LLMs)."},"venue":{"value":"COLM 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the COLM Code of Ethics on https://colmweb.org/CoE.html"},"keywords":{"value":["collaborative inference","efficient inference","token-level routing","large language model"]},"abstract":{"value":"Large language models have achieved remarkable success in various tasks but suffer from high computational costs during inference, limiting their deployment in resource-constrained applications. To address this issue, we propose a novel Collaborative Inference with Token-lEvel Routing (CITER) framework that enables efficient collaboration between small and large language models (SLMs & LLMs) through a token-level routing strategy. Specifically, CITER routes non-critical tokens to an SLM for efficiency and routes critical tokens to an LLM for generalization quality. We formulate router training as a policy optimization, where the router receives rewards based on both the quality of predictions and the inference costs of generation. This allows the router to learn to predict token-level routing scores and make routing decisions based on both the current token and the future impact of its decisions. To further accelerate the reward evaluation process, we introduce a shortcut which significantly reduces the costs of the reward estimation and improving the practicality of our approach. Extensive experiments on five benchmark datasets demonstrate that CITER reduces the inference costs while preserving high-quality generation, offering a promising solution for real-time and resource-constrained applications."},"_bibtex":{"value":"@inproceedings{\nzheng2025citer,\ntitle={{CITER}: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routing},\nauthor={Wenhao Zheng and Yixiao Chen and Weitong Zhang and Souvik Kundu and Yun Li and Zhengzhong Liu and Eric P. Xing and Hongyi Wang and Huaxiu Yao},\nbooktitle={Second Conference on Language Modeling},\nyear={2025},\nurl={https://openreview.net/forum?id=nqX9UYW9Af}\n}"},"title":{"value":"CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routing"},"pdf":{"value":"/pdf/44adc116c02c1f30777bdd97bc65accd40cb593a.pdf"},"venueid":{"value":"colmweb.org/COLM/2025/Conference"},"paperhash":{"value":"zheng|citer_collaborative_inference_for_efficient_large_language_model_decoding_with_tokenlevel_routing"},"authorids":{"value":["~Wenhao_Zheng4","~Yixiao_Chen2","~Weitong_Zhang2","~Souvik_Kundu2","~Yun_Li7","~Zhengzhong_Liu1","~Eric_Xing1","~Hongyi_Wang1","~Huaxiu_Yao1"]},"author_guide":{"value":"I certify that this submission complies with the submission instructions as described on https://colmweb.org/AuthorGuide.html"},"authors":{"value":["Wenhao Zheng","Yixiao Chen","Weitong Zhang","Souvik Kundu","Yun Li","Zhengzhong Liu","Eric P. Xing","Hongyi Wang","Huaxiu Yao"]}},"version":2},{"content":{"comment":{"value":"To Reviewer vj6u: We greatly appreciate your constructive comments, which we found especially insightful concerning the integration of a world model into SLAM. We wholeheartedly echo the view that world models are poised to become a cornerstone of next-generation research in SLAM and embodied intelligence.\n1. The core value proposition of our multi-agent orchestration is the seamless integration of diverse models for SLAM, primarily aimed at boosting operational efficiency and adaptability, not at directly enhancing localization accuracy. We think the available method to ablate this contribution is constructing a traditional, monolithic implementation that directly calls the identical core components (such as InstructIR for restoration and RouteLLM for depth model routing) as a baseline for comparison.\n2. The introduction of the Orchestrator, inspired by framework Magentic-One, represents an architectural decision in implementing the multi-agent system. This component is theoretically optional, as the core choice lies between a centralized orchestrator and distributed architecture. While a monolithic LLM with function-calling capabilities could, in principle, fulfill this role, smaller-scale LLMs tend to exhibit higher function-calling error rates. A sufficiently robust and capable LLM could indeed render a dedicated orchestrator redundant.\n3. In Co-Instruct, we employ the prompt = \"USER: The image: <|image|> Which happens in this image: motion-blur, rain, haze, fog, or low light? ASSISTANT:\" to ensure that each chat output is standardized. The response following \"ASSISTANT:\" is then extracted. These answers are concise tokens that can be approximately regarded as definitive outputs, similar to those produced by a classification head. If none of the listed conditions is present in the image, the model replies with \"None of the above.\"\n4. The labels for blurry/low-light frames were generated through a hybrid approach, combining the intrinsic assessment of the Assessor model with subsequent human verification.\n5. Under our SLAM framework, the KITTI dataset was not utilized for evaluating the overall system accuracy as it does not directly provide depth maps. We have completed the image restoration experiments on the TartanAir dataset, while the training of the LLM-Route for depth estimation is still ongoing. We are actively searching for other suitable datasets that meet our requirements. Ultimately, the most robust solution would be to create a dedicated, task-specific dataset of our own."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1770917755173,"tcdate":1763624697070,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7039/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission7039/Authors"],"forum":"duIZUIRKJB","number":6,"license":"CC BY 4.0","cdate":1763624697070,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7039/-/Official_Comment","ICLR.cc/2026/Conference/-/Edit"],"mdate":1770917755173,"domain":"ICLR.cc/2026/Conference","replyto":"Gm7EIk1q8B","id":"NHqZ7aKVTZ","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Deep SLAM; Multi-agent; Deep Estimation ;3DGS"]},"primary_area":{"value":"applications to robotics, autonomy, planning"},"abstract":{"value":"Visual Simultaneous Localization and Mapping (SLAM) reconstructs the metric structure of the physical world from sensor imagery, enabling precise robotic pose estimation. However, environmentally induced image degradation and varied image processing strategie significantly compromise localization accuracy. Intelligent SLAM systems address this challenge by autonomously perceiving dynamic perturbations and formulating adaptive processing strategies, further identifying and deploying optimal methodologies to achieve target localization objectives with enhanced metric precision. This paper introduces MAV-SLAM, a novel Multi-LLM-Agent-Orchestrated visual SLAM framework that proactively identifies and compensates for suboptimal image quality while autonomously selecting optimal depth estimation models. Specifically, we integrate a visual-language model that performs autonomous image restoration guided by image quality assessment, significantly enhancing SLAM localization performance. Furthermore, we implement a routing large language model for adaptive depth estimation, which consequently elevates the quality of 3D reconstruction via 3D Gaussian Splatting (3DGS). Rigorous evaluation across multiple benchmarks demonstrates that MAV-SLAM exhibits superior performance in both localization accuracy and 3DGS-based reconstruction fidelity, validating its effectiveness in real-world scenarios."},"_bibtex":{"value":"@misc{\nyao2026mavslam,\ntitle={{MAV}-{SLAM}: Multi-{LLM}-Agent Crew for Visual {SLAM} with 3D Gaussian Splatting},\nauthor={Jian Yao and Lisha Tang and Yang Li and Danping Zou},\nyear={2026},\nurl={https://openreview.net/forum?id=duIZUIRKJB}\n}"},"title":{"value":"MAV-SLAM: Multi-LLM-Agent Crew for Visual SLAM with 3D Gaussian Splatting"},"pdf":{"value":"/pdf/8fccc6924a5fddfae91e0c152e944d9a70c4b58f.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"yao|mavslam_multillmagent_crew_for_visual_slam_with_3d_gaussian_splatting"},"authorids":{"value":["~Jian_Yao9","~Lisha_Tang1","~Yang_Li26","~Danping_Zou1"]},"authors":{"value":["Jian Yao","Lisha Tang","Yang Li","Danping Zou"]}},"version":2},{"content":{"comment":{"value":"## **Q2: Is the true \"routing overhead\" underestimated? There may be multiple levels of consecutive rejections on the cascading path (the experiment used a 5-model approach, and the worst case was only 4 rejections to reach 14 bytes). Could you please provide the end-to-end latency of the first token for comparison with other methods?**\n\nWe appreciate the reviewer's scrutiny regarding the **practical latency** of our cascading architecture. We share the concern that multiple consecutive rejections could theoretically accumulate delay. However, our new comprehensive analysis in **Appendix A.1** empirically demonstrates that this overhead is **statistically negligible**.\n\n### **1. Negligible Cumulative Overhead (<5%)**\n\nOur data confirms that the \"routing overhead\" (time spent on rejections) is marginal compared to the cost of generation.\n\n- **Worst-Case Analysis:** Even in the worst-case path—where a query is rejected by all four smaller agents before reaching the final 14B model—the cumulative time consumed by these rejections constitutes **less than 5%** of the time required for the final LLM to generate the answer (*see* **Table 7 in the paper**).\n- **Reasoning:** This efficiency stems from the **brevity of the rejection signal**. A rejection typically involves generating only a few tokens (e.g., \"I don't know!\"), whereas generating a valid solution involves a lengthy CoT sequence. Thus, the cost of “saying no\" is a fraction of the cost of solving.\n\n### **2. Time-to-First-Token Analysis**\n\nTo provide the requested comparison, we benchmarked the **System Latency (Time to First Token)** of DiSRouter against router baselines (detailed in **Table R4**).\n\n*R4: Comparison of Time to First Token (s) between routing systems.*\n\n|  | RouterLLM | FrugalGPT | Automix | FORC | GraphRouter | DiSRouter |\n| --- | --- | --- | --- | --- | --- | --- |\n| Average | 0.048 | 2.620 | 3.475 | 0.070 | 0.024 | 0.081 |\n| Worst Case | 0.063 | 9.565 | 11.34 | 0.076 | 0.028 | 0.241 |\n- **vs. Predictive Routers:** DiSRouter incurs slightly higher latency than predictive routers (e.g., *RouteLLM*, *GraphRouter*) that utilize lightweight external classifiers, as our method requires LLM inference.\n- **vs. Generative Baselines:** Crucially, DiSRouter is **significantly faster** than other generative or verification-based routers (e.g., *FrugalGPT*, *AutoMix*). These baselines often require generating **full-length responses** from LLMs before performing verification or cascading, whereas DiSRouter aborts generation immediately upon detecting incompetence.\n\n**Conclusion:** Based on this analysis, we conclude that the \"routing overhead\" introduced by the rejection mechanism is **computationally minor** and does not impede the system's practical utility or user experience."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764155406128,"tcdate":1764155406128,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7635/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission7635/Authors"],"forum":"KDcwXKr0NU","number":7,"license":"CC BY 4.0","cdate":1764155406128,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7635/-/Official_Comment"],"mdate":1764155406128,"domain":"ICLR.cc/2026/Conference","replyto":"6wZj9Kgstr","id":"l2iofs9f9X","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"TLDR":{"value":"We introduce DiSRouter, a distributed system where LLMs leverage \"self-awareness\" to route queries among themselves, outperforming conventional centralized routers."},"keywords":{"value":["query routing","model selection","distributed system","self-awareness of LLM"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"The proliferation of Large Language Models (LLMs) has created a diverse ecosystem of models with highly varying performance and costs, necessitating effective query routing to balance performance and expense. Current routing systems often rely on a centralized external router trained on a fixed set of LLMs, making them inflexible and prone to poor performance since the small router can not fully understand the knowledge boundaries of different LLMs. We introduce DiSRouter (Distributed Self-Router), a novel paradigm that shifts from centralized control to distributed routing. In DiSRouter, a query traverses a network of LLM agents, each independently deciding whether to answer or route to other agents based on its own self-awareness—its ability to judge its competence. This distributed design offers superior flexibility, scalability, and generalizability. To enable this, we propose a two-stage Self-Awareness Training pipeline that enhances each LLM's self-awareness. Extensive experiments demonstrate that DiSRouter significantly outperforms existing routing methods in utility across various scenarios, effectively distinguishes between easy and hard queries, and shows strong generalization to out-of-domain tasks. Our work validates that leveraging an LLM's intrinsic self-awareness is more effective than external assessment, paving the way for more modular and efficient multi-agent systems."},"_bibtex":{"value":"@inproceedings{\nzheng2026disrouter,\ntitle={Di{SR}outer: Distributed Self-Routing for {LLM} Selections},\nauthor={Hang Zheng and Hongshen Xu and Yongkai.lin and Shuai Fan and Lu Chen and Kai Yu},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=KDcwXKr0NU}\n}"},"title":{"value":"DiSRouter: Distributed Self-Routing for LLM Selections"},"pdf":{"value":"/pdf/0190f98d94890b700d03febdee678e22a2a5e380.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"zheng|disrouter_distributed_selfrouting_for_llm_selections"},"authorids":{"value":["~Hang_Zheng3","~Hongshen_Xu1","~Yongkai.lin1","~Shuai_Fan1","~Lu_Chen3","~Kai_Yu3"]},"authors":{"value":["Hang Zheng","Hongshen Xu","Yongkai.lin","Shuai Fan","Lu Chen","Kai Yu"]}},"version":2},{"content":{"comment":{"value":"## **W5/Q4. Analysis by difficulty ------ Comprehensive breakdown added**\n> W5. The paper spends substantial space describing the construction of a dataset with distinct difficulty levels but does not subsequently analyze results based on these levels.\n> Q4. The paper builds a dataset with three difficulty levels, but all evaluations are aggregated. Why not provide router performance broken down by difficulty level? \n\nThank you for pointing this out. In the revision, we now **provide a comprehensive breakdown of router performance by difficulty level** (Easy / Medium / Hard), including both accuracy and average cost per 1K queries (Table 6 in the paper and we paste it below).\n\nThis analysis confirms that our empirical difficulty labels behave as intended: all routers achieve very high accuracy on Easy queries but suffer substantial degradation on Hard queries. On Easy examples, lightweight routers such as MIRT-BERT, CARROT, and vLLM-SR nearly match GPT-5 while being much cheaper, whereas on hard queries, GPT-5 remains the strongest, with Azure-Router and NotDiamond offering the next-best cost–accuracy trade-offs. We also observe that some routers (e.g., GPT-5, Azure-Router, MIRT-BERT, NotDiamond) allocate significantly more budget to Hard than Easy queries, while others show almost flat cost across difficulty, suggesting room for more difficulty-aware routing strategies.\n\n---\n\n### Table 2: Router performance by difficulty level (accuracy and cost per 1k queries).\n\n| Router        | Overall             | Easy                | Medium              | Hard                 |\n|--------------|---------------------|---------------------|---------------------|----------------------|\n| GPT-5        | 74.0% ($14.02)      | 95.1% ($5.68)       | 68.6% ($14.80)      | 27.5% ($35.73)       |\n| Azure-Router | 68.1% ($0.54)       | 93.3% ($0.30)       | 59.5% ($0.63)       | 17.9% ($1.05)        |\n| NotDiamond   | 68.0% ($9.34)       | 92.5% ($6.34)       | 61.9% ($10.29)      | 15.1% ($15.16)       |\n| vLLM-SR      | 67.3% ($1.67)       | 95.3% ($1.56)       | 57.9% ($1.70)       | 8.7% ($1.87)         |\n| CARROT       | 67.2% ($2.06)       | 95.0% ($1.79)       | 58.0% ($2.26)       | 9.0% ($2.36)         |\n| MIRT-BERT    | 66.9% ($0.15)       | 95.9% ($0.10)       | 56.8% ($0.17)       | 7.1% ($0.26)         |\n| NIRT-BERT    | 62.0% ($0.69)       | 93.7% ($0.53)       | 47.0% ($0.78)       | 4.7% ($0.90)         |        |\n| MLP          | 61.6% ($4.83)       | 93.2% ($3.54)       | 46.6% ($5.74)       | 4.6% ($6.52)         |\n| KNN          | 58.7% ($4.27)       | 89.2% ($3.08)       | 43.6% ($5.13)       | 4.5% ($5.77)         |\n| Graphrouter  | 57.0% ($0.34)       | 89.7% ($0.25)       | 39.3% ($0.39)       | 2.4% ($0.46) \n| RouteLLM     | 47.0% ($0.27)       | 79.8% ($0.20)       | 24.4% ($0.33)       | 2.0% ($0.32)         |\n| RouterDC     | 32.0% ($0.07)       | 54.9% ($0.06)       | 15.3% ($0.08)       | 2.2% ($0.09)         |\n\n---\n\n## **Q3. Figure naming ------ Renamed to Normalized Scatter Plot**\n> Q3. Figure 6 looks more like a scatter plot than a curve plot. Would the authors consider renaming it for accuracy? \n\nThanks for the suggestion! We have changed its name to \"Normalized Scatter Plot\"."},"title":{"value":"Thanks! (3/3)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763965996428,"tcdate":1763964956020,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission13871/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission13871/Authors"],"forum":"9HsaIi4ngF","number":6,"license":"CC BY 4.0","cdate":1763964956020,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission13871/-/Official_Comment"],"mdate":1763965996428,"domain":"ICLR.cc/2026/Conference","replyto":"LHAfoFB9LF","id":"18PHtnQbmH","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"TLDR":{"value":"We proposed a comprehensive LLM router evaluation framework, with a new dataset, metrics, and ranking system."},"keywords":{"value":["LLM Router","Evaluation"]},"primary_area":{"value":"datasets and benchmarks"},"abstract":{"value":"Today's LLM ecosystem comprises a wide spectrum of models that differ in size, capability, and cost. No single model is optimal for all scenarios; hence, LLM routers have become essential for selecting the most appropriate model under varying circumstances.\nHowever, the rapid emergence of various routers has led to fragmented evaluation practices and inconsistent metrics, making it difficult to systematically assess progress in this space. To address this problem, we need a comprehensive router comparison and a standardized leaderboard, similar to those available for models. In this work, we introduce RouterArena, the first open platform enabling comprehensive comparison of LLM routers. RouterArena has (1) a principally constructed dataset with broad knowledge domain coverage, (2) distinguishable difficulty levels for each domain, (3) an extensive list of evaluation metrics, and (4) an automated framework for evaluation and leaderboard updates. Leveraging this framework, we have produced the initial leaderboard with detailed metrics comparison.\nFigure1 provides a preview of the leaderboard. The complete framework and the latest router leaderboard are publicly available at https://routeworks.github.io/"},"_bibtex":{"value":"@inproceedings{\nlu2026routerarena,\ntitle={RouterArena: An Open Platform for Comprehensive Comparison of {LLM} Routers},\nauthor={Yifan Lu and Rixin Liu and Jiayi Yuan and Xingqi Cui and Shenrun Zhang and Hongyi Liu and Jiarong Xing},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=9HsaIi4ngF}\n}"},"title":{"value":"RouterArena: An Open Platform for Comprehensive Comparison of LLM Routers"},"pdf":{"value":"/pdf/03e5d4326116c387dbd69c38e194211db0736cda.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"lu|routerarena_an_open_platform_for_comprehensive_comparison_of_llm_routers"},"authorids":{"value":["~Yifan_Lu8","~Rixin_Liu1","~Jiayi_Yuan1","~Xingqi_Cui1","~Shenrun_Zhang1","~Hongyi_Liu5","~Jiarong_Xing1"]},"authors":{"value":["Yifan Lu","Rixin Liu","Jiayi Yuan","Xingqi Cui","Shenrun Zhang","Hongyi Liu","Jiarong Xing"]}},"version":2},{"content":{"comment":{"value":"### Weakness 2: \"Related work inadequately contextualizes the approach.\"\n\n**Response**: We revised the paper to include a more detailed explanation of model routing and ensemble-learning methods. These approaches are not applicable to DNA extraction because they rely on supervised training for specific downstream tasks, and thus their learned representations are task-dependent and do not generalize across tasks.\n\n\n\n> **Section 2: Related Work**\n>\n>  **Representation of LLMs.** This direction manages large LLM collections by encoding each LLM into a representation. Most methods are task-specific: Ong et al. (2025); Wang et al. (2024) learn representations for model routing, while Huang et al. (2024); Fang et al. (2024) encode models for ensemble learning. For instance, HybridLLM (Ding et al., 2024) trains a binary “easy-hard” classifier to route queries to a small or a larger model. RouteLLM (Ong et al., 2025) learns a supervised router that selects between stronger and weaker models based on query features. MetaLLM (Nguyen et al., 2024) formulates routing as a multi-armed bandit, adaptively trading off expected answer quality against invocation cost via online feedback. LLM-ensemble (Fang et al., 2024) learns the weights for different LLMs to predict a specifc attribute value. **These task-specific approaches rely on supervised training for particular tasks, making their learned representations not generalizable across tasks.** Zhuang et al. (2025) propose EmbedLLM, the first compact embedding supporting multiple tasks (e.g., model routing and benchmark prediction), but it is defined over a fixed model set; adding models requires retraining and can shift the learned embeddings. Yax et al. (2025) propose PhyloLM, which constructs a phylogenetic tree using a novel distance between LLMs. However, this distance is token-based and does not generalize well to heterogeneous LLMs. In contrast, our LLM DNA is a stable and generalizable intrinsic property extractable independently for each LLM.\n\n---\n\n### Weakness 3: \"Baseline comparisons are insufficient.\"\n\n**Response**: We include a comparison with the state-of-the-art approach PhyloLM [1] as suggested by Reviewer `ZryQ`, which similarly constructs phylogenetic-style visualizations from a distance matrix derived from token distributions. Our experiments show that LLM DNA consistently outperforms PhyloLM on the LLM relationship prediction task (Table 1) and demonstrates better generalizability. Please see the **updated Table 1** in the revised paper, as well as our **response to “Weakness 1,”** for further details."},"title":{"value":"Author Response (Part 2)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763890719703,"tcdate":1763890719703,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission8961/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission8961/Authors"],"forum":"UIxHaAqFqQ","number":10,"license":"CC BY 4.0","cdate":1763890719703,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission8961/-/Official_Comment"],"mdate":1763890719703,"domain":"ICLR.cc/2026/Conference","replyto":"inU5yJ7BnR","id":"OSplHGHdBd","forumContent":{"venue":{"value":"ICLR 2026 Oral"},"TLDR":{"value":"We introduce LLM DNA, a low-dimensional representation of LLMs, uncovers undocumented relations and constructs phylogenetic tree for LLM."},"keywords":{"value":["Large Language Models","Functional Representations","Model Fingerprinting","Semantic Embeddings","Model Evolution","Model Lineage","Phylogenetic Trees","Dimensionality Reduction","Black-box Model Analysis"]},"primary_area":{"value":"unsupervised, self-supervised, semi-supervised, and supervised representation learning"},"abstract":{"value":"The explosive growth of large language models (LLMs) has created a vast but opaque landscape: millions of models exist, yet their evolutionary relationships through fine-tuning, distillation, or adaptation are often undocumented or unclear, complicating LLM management. Existing methods are limited by task specificity, fixed model sets, or strict assumptions about tokenizers or architectures. Inspired by biological DNA, we address these limitations by mathematically defining *LLM DNA* as a low-dimensional, bi-Lipschitz representation of functional behavior. We prove that LLM DNA satisfies *inheritance* and *genetic determinism* and establish its existence. Building on this theory, we derive a general, scalable, training-free pipeline for DNA extraction. In experiments across 305 LLMs, DNA aligns with prior studies on limited subsets and achieves superior or competitive performance on various tasks. Beyond these tasks, DNA comparisons uncover previously undocumented relationships among LLMs. We further construct the evolutionary tree of LLMs using phylogenetic algorithms, which align with shifts from encoder-decoder to decoder-only architectures, reflect temporal progression, and reveal distinct evolutionary speeds across LLM families."},"_bibtex":{"value":"@inproceedings{\nwu2026llm,\ntitle={{LLM} {DNA}: Tracing Model Evolution via Functional Representations},\nauthor={Zhaomin Wu and Haodong Zhao and Ziyang Wang and Jizhou Guo and Qian Wang and Bingsheng He},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=UIxHaAqFqQ}\n}"},"title":{"value":"LLM DNA: Tracing Model Evolution via Functional Representations"},"pdf":{"value":"/pdf/f00716835fd981059a4c5cc8d79dcc5c455ad828.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"wu|llm_dna_tracing_model_evolution_via_functional_representations"},"authorids":{"value":["~Zhaomin_Wu1","~Haodong_Zhao2","~Ziyang_Wang16","~Jizhou_Guo1","~Qian_Wang25","~Bingsheng_He1"]},"authors":{"value":["Zhaomin Wu","Haodong Zhao","Ziyang Wang","Jizhou Guo","Qian Wang","Bingsheng He"]}},"version":2},{"content":{"summary":{"value":"This paper proposes a token‑complexity‑based dynamic routing mechanism for improving Mixture‑of‑Experts (MoE) architectures in large language models. Unlike conventional fixed top‑k routing, the method employs a lightweight neural router trained with few‑shot supervision to estimate each token’s complexity and dynamically assign it to either “lightweight” or “strong” feedforward expert networks (FFNs) based on a learned threshold α. Integrated with models such as Mistral‑7B, Mixtral‑8×7B, and LLaMA‑2‑7B, the framework achieves up to 12% accuracy improvements across benchmarks including GSM8K, MMLU, and HellaSwag, while maintaining stable computational cost. Overall, this work provides an efficient, scalable approach to enhance expert utilization and inference efficiency in transformer‑based large language models."},"soundness":{"value":2},"confidence":{"value":2},"questions":{"value":"- Could the authors clarify how their token‑complexity‑based router fundamentally differs from prior adaptive MoE systems such as DynaMoE or Flextron?  \n- How was the heuristic definition of token complexity validated, and does it correlate with quantitative measures like entropy or surprisal?  \n- What is the sensitivity of the model’s performance to the chosen routing threshold \\( \\alpha \\), and how was this value determined?  \n- Why were recent baselines like RouteLLM and RouterBench excluded from the experimental comparisons?  \n- Can the authors provide a breakdown of the additional inference cost introduced by the routing mechanism?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":1},"contribution":{"value":2},"strengths":{"value":"- The paper explores an interesting direction by integrating token‑level complexity awareness into MoE routing, which conceptually extends prior adaptive expert frameworks like DynaMoE and Flextron toward a more fine‑grained per‑token perspective.\n\n- The idea of using a few‑shot supervised router with a tunable complexity threshold is a creative attempt to make routing both data‑efficient and flexible, showing potential for broader applicability in resource‑constrained LLM deployment.\n\n- The authors conduct comprehensive empirical evaluations across diverse benchmarks, demonstrating consistent, though moderate, improvements that illustrate the promise of their complexity‑based routing strategy for enhancing expert utilization efficiency."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- The overall writing quality of the paper requires substantial improvement. First, the use of citations is incorrect and inconsistent — the authors do not properly distinguish among \\cite, \\citep, and \\citet, as seen for example at lines 131 and 140. Second, the formatting of the tables does not follow the ICLR template guidelines, since the table borders should open on both sides. In addition, some figures, such as Figure 2, are not visually appealing — there is excessive spacing between subfigures, and the font size is too small. Moreover, table references (e.g., “Table. 3.” at line 357) are not hyperlinked, making them unclickable. Overall, the writing and presentation quality of the manuscript still need significant improvement.\n\n- The proposed token-complexity-based routing concept overlaps substantially with prior adaptive MoE systems such as DynaMoE [1] and Flextron [2], which also route tokens or features based on difficulty or task semantics. The few-shot supervised router and complexity threshold provide some variation, but the paper does not sufficiently explain what theoretical gap this fills beyond existing “dynamic routing” approaches (e.g., Token-level Load Balancing). The paper should clarify how its method differs in the design of gating functions or optimization objectives, ideally supported by quantitative or theoretical evidence—such as improved load balance entropy, lower variance in expert utilization, or convergence guarantees—to substantiate novelty claims.\n\n- The definition of token complexity relies on heuristics—for instance, labeling mathematical or programming symbols as “complex” and common words as “simple.” This rule-based labeling can cause poor generalization across domains because it ignores probabilistic uncertainty or contextual diversity. A more rigorous quantitative measure such as token entropy, surprisal, contextual dependency length, or prediction variance should replace heuristic rules to better reflect real difficulty. Moreover, verifying the correlation between estimated complexity C(t) and empirical uncertainty measures (e.g., perplexity) would strengthen the validity of the metric. If such signals were integrated into the router, the routing decisions could become more consistent across tasks.\n\n- The experiments evaluate a wide set of benchmarks but lack the necessary depth to isolate where the performance improvements originate. The study omits comparisons to recent dynamic router baselines such as RouteLLM and RouterBench.\n\n\n**Reference:**\n\n[1] From dense to dynamic: Token-difficulty driven\nmoefication of pre-trained llms.\n\n[2]  Flextron: Many-in-one flexible large language model."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762924083803,"tcdate":1761974688321,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission13468/Reviewer_3Bi6"],"signatures":["ICLR.cc/2026/Conference/Submission13468/Reviewer_3Bi6"],"forum":"miyzT0uhhF","number":4,"license":"CC BY 4.0","cdate":1761974688321,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission13468/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762924083803,"domain":"ICLR.cc/2026/Conference","replyto":"miyzT0uhhF","id":"WQ9cBds31i","forumContent":{"TLDR":{"value":"Mixture of Expert Architecture for Large Language Models to enhance scaling and performance"},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Mixture of Experts","Large Language Model","Router","Token Complexity Threshold."]},"primary_area":{"value":"generative models"},"abstract":{"value":"Mixture-of-Experts (MoE) architectures have emerged as a powerful technique for improving and scaling Large Language Models by conditionally activating Feed Forward subnetworks and distributing tokens through a routing system,  within the Transformer layers. However, existing MoE methods often rely on static top-k routing strategies that do not involve token-level variability in complexity, leading to suboptimal expert utilization. In this research, we propose a novel token-complexity-based routing framework that dynamically allocates tokens to either lightweight or strong feedforward networks (FFNs) based on their estimated token complexity. Our router is trained using a few-shot classification objective to distinguish between easy and complex tokens and surrogate neural network layer. The efficacy of the framework is evaluated while integrating the router with Mistral-7B and Llama-2-7B model. We evaluate our approach on several benchmarks from various fields, and our proposed MoE framework improves accuracy up to 12% compared to the state-of-the-art results using different MoE architecture, with reasonable computational cost."},"_bibtex":{"value":"@misc{\nrahman2026tokencomplexity,\ntitle={Token-Complexity based Routing Technique within Mixture of Experts Architecture for Large Language Model},\nauthor={Tasnia Rahman and Sathish A.P. Kumar},\nyear={2026},\nurl={https://openreview.net/forum?id=miyzT0uhhF}\n}"},"title":{"value":"Token-Complexity based Routing Technique within Mixture of Experts Architecture for Large Language Model"},"pdf":{"value":"/pdf/8daaf66a8305dcfe6a2ae91cf82adf0862008c1a.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"rahman|tokencomplexity_based_routing_technique_within_mixture_of_experts_architecture_for_large_language_model"},"authorids":{"value":["~Tasnia_Rahman1","~Sathish_A.P._Kumar1"]},"authors":{"value":["Tasnia Rahman","Sathish A.P. Kumar"]}},"version":2},{"content":{"comment":{"value":"We thank the reviewer for the thoughtful feedback. We appreciate the reviewer highlighting the practical importance of efficient routing across LLMs and decoding strategies, as well as the relevance of addressing both continual and non-continual deployment scenarios. We also appreciate that the reviewer recognized the consistent empirical gains achieved by our modular predictor framework and optimization-based routing procedure. We hope the clarifications and additional context provided below reinforce the soundness of our method and help address the concerns raised.\n\n> W1: Although the paper repeatedly discusses routing in both continual and non-continual settings, the problem itself is never formally defined and how the optimization objective is formulated in mathematical terms.\n\nOur work considers two routing problems: constrained routing and unconstrained routing. The continual/non-continual settings describe the temporal context in which these routing tasks are performed. Specifically, in the continual setting, the framework is trained on an initial set of strategies (a strategy refers to a specific language model and a decoding method) at time X and must later incorporate additional strategies that become available at a later time Y. In contrast, in the non-continual setting, the set of strategies remains fixed over time.\n\nThe formal definitions of the two routing problems are provided in Section 2.2. For unconstrained routing, the optimization objective is to maximize accuracy $a_{ij}$ and minimize computational cost $c_{ij}$, expressed in line 210 as $\\max a_{ij} \\min c_{ij}$, where $i$ represents the task and $j$ represents the strategy. For constrained routing, the objective is defined in line 218 as $\\max_j a_{ij}$  such that $c_{ij} ≤ B$ where $B$ is the cost budget.\n\n> W1: It remains unclear what exactly constitutes a routing input and output.\n\nAs shown in Figure 1 and implied by the optimization problems above, the input consists of (i) a downstream task and (ii) a candidate set of computational strategies (a strategy refers to a specific language model and a decoding method). The output is the selected strategy from the input set that best satisfies the corresponding routing objective (either the unconstrained or constrained formulation). We will make it clearer in the revised version of our paper.\n\n> W1: The framework is presented primarily at a conceptual level, which makes it difficult to precisely understand the problem scope and to reproduce the approach in other contexts. As a result, the paper may not be self-contained for readers who are not already familiar with prior routing works such as RouteLLM or RTR, which substantially limits accessibility and clarity.\n\nThank you for your suggestions. We will improve the presentation of the paper."},"title":{"value":"Author response 1 (A1)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764806914556,"tcdate":1764803025001,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission14463/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission14463/Authors"],"forum":"gCUY6QIv8r","number":10,"license":"CC BY 4.0","cdate":1764803025001,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission14463/-/Official_Comment"],"mdate":1764806914556,"domain":"ICLR.cc/2026/Conference","replyto":"CBGEtBOL6F","id":"fbN3jijcHB","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["model routing","continual routing","constrained routing","unconstrained routing"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"AI tasks differ in complexity and are best addressed with different computation strategies (e.g., combinations of models and decoding methods). Hence, an effective routing system that maps tasks to the appropriate strategies is crucial.\nMost prior methods build the routing framework by training a *single* model across *all* strategies, which demands full retraining whenever new strategies appear and leads to high overhead. Attempts at such continual routing, however, often face difficulties with generalization.\nPrior models also typically use a *single* input representation, limiting their ability to capture the full complexity of the routing problem and leading to sub-optimal routing decisions.\nTo address these gaps, we propose CONCUR, a **con**tinual routing framework that supports both **c**onstrained and **u**nconstrained **r**outing (i.e., routing with or without a budget).\nOur *modular* design trains a separate predictor model for each strategy, enabling seamless incorporation of new strategies with low additional training cost.\nOur predictors also leverage *multiple* representations of both tasks and computation strategies to better capture overall problem complexity.\nExperiments on both in-distribution and out-of-distribution, knowledge- and reasoning-intensive tasks show that our method outperforms the best single strategy and strong existing routing techniques with higher end-to-end accuracy and lower inference cost in both continual and non-continual settings, while also reducing training cost in the continual setting."},"_bibtex":{"value":"@inproceedings{\nchen2026concur,\ntitle={{CONCUR}: A Framework for Continual Constrained and Unconstrained Routing},\nauthor={Peter Baile Chen and Weiyue Li and Dan Roth and Mike Cafarella and Samuel Madden and Jacob Andreas},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=gCUY6QIv8r}\n}"},"title":{"value":"CONCUR: A Framework for Continual Constrained and Unconstrained Routing"},"pdf":{"value":"/pdf/b59e1bbe2f0d9d26266b4e43191885655a9464f7.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"chen|concur_a_framework_for_continual_constrained_and_unconstrained_routing"},"authorids":{"value":["~Peter_Baile_Chen1","~Weiyue_Li1","~Dan_Roth3","~Mike_Cafarella1","~Samuel_Madden1","~Jacob_Andreas1"]},"authors":{"value":["Peter Baile Chen","Weiyue Li","Dan Roth","Mike Cafarella","Samuel Madden","Jacob Andreas"]}},"version":2},{"content":{"comment":{"value":"## **W4. Additional metrics (Length/Time) ------ Length added; Time is unreliable**\n> W4. Evaluation dimensions remain limited. Additional aspects—such as response time or response length—should be considered. For example, if the pool contains a cheaper reasoning model and a more expensive instruction model, then under the cost formulation cost = C_in * N_in + C_out * N_out, two routing strategies might achieve the same cost and same accuracy, yet differ meaningfully in behavior; how should their relative quality be judged?\n\n\nThank you for the suggestion. We agree that response length and response time are important behavioral characteristics. Following your comment, we **added average response length (in output tokens)** to the RouterArena leaderboard and show the updated table at the end. This metric reflects how frequently a router selects more reasoning-heavy models, e.g., GPT-5 and Not Diamond often yield longer outputs. We do not treat longer answers as better; the metric simply provides another axis for users to choose between concise and detailed responses.\n\nBy contrast, although we initially considered wall-clock response time, we ultimately decided not to include it because it is extremely hard to measure fairly and reproducibly. RouterArena evaluates heterogeneous routers that depend on proprietary APIs (e.g., Claude Sonnet-4, GPT-5) running on different hardware over the public internet. In this setting, observed latency is dominated by provider infrastructure and network conditions, not the router logic itself. We therefore focus on router-intrinsic latency, the overhead introduced before issuing the LLM call, which we report as our “router latency” metric.\n\nRegarding your concrete example where two routers that achieve the same cost and accuracy but differ in output behavior, our view is that such routers are **Pareto-equivalent on the primary axes**. In practice, users adopt routers either (i) to reduce inference cost by using cheaper models, or (ii) to improve accuracy by selecting domain-specialized models. RouterArena provides tunable $\\beta$ to let users adjust the cost-accuracy trade-off.\n\n---\n\n### Table 2: RouterArena routers — average generation length and cost\n\n| Router       | Avg Gen Length | Cost/M Tokens |\n|-------------|----------------|---------------|\n| GPT-5       | 970.65         | $10.00        |\n| NotDiamond  | 390.05         | $9.18         |\n| Azure-Router| 270.90         | $1.79         |\n| NIRT-BERT   | 215.82         | $1.33         |\n| RouteLLM    | 140.24         | $0.68         |\n| Graphrouter | 132.24         | $0.81         |\n| vLLM-SR     | 109.27         | $9.03         |\n| CARROT      | 105.56         | $12.58        |\n| RouterDC    | 102.83         | $0.20         |\n| MIRT-BERT   | 89.16          | $1.00         |\n| MLP         | 81.06          | $29.75        |\n| KNN         | 79.80          | $26.31        |\n\n---\n\n## **Notation**\n> All displayed formulas lack numbering, which reduces ease of reference and clarity in discussion. \n\nThanks for the suggestions! We have added the number labels for each equation in the paper."},"title":{"value":"Thank you! (4/4)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763965908014,"tcdate":1763965870213,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission13871/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission13871/Authors"],"forum":"9HsaIi4ngF","number":10,"license":"CC BY 4.0","cdate":1763965870213,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission13871/-/Official_Comment"],"mdate":1763965908014,"domain":"ICLR.cc/2026/Conference","replyto":"8iXIX7wkBw","id":"i5EDcAqwlB","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"TLDR":{"value":"We proposed a comprehensive LLM router evaluation framework, with a new dataset, metrics, and ranking system."},"keywords":{"value":["LLM Router","Evaluation"]},"primary_area":{"value":"datasets and benchmarks"},"abstract":{"value":"Today's LLM ecosystem comprises a wide spectrum of models that differ in size, capability, and cost. No single model is optimal for all scenarios; hence, LLM routers have become essential for selecting the most appropriate model under varying circumstances.\nHowever, the rapid emergence of various routers has led to fragmented evaluation practices and inconsistent metrics, making it difficult to systematically assess progress in this space. To address this problem, we need a comprehensive router comparison and a standardized leaderboard, similar to those available for models. In this work, we introduce RouterArena, the first open platform enabling comprehensive comparison of LLM routers. RouterArena has (1) a principally constructed dataset with broad knowledge domain coverage, (2) distinguishable difficulty levels for each domain, (3) an extensive list of evaluation metrics, and (4) an automated framework for evaluation and leaderboard updates. Leveraging this framework, we have produced the initial leaderboard with detailed metrics comparison.\nFigure1 provides a preview of the leaderboard. The complete framework and the latest router leaderboard are publicly available at https://routeworks.github.io/"},"_bibtex":{"value":"@inproceedings{\nlu2026routerarena,\ntitle={RouterArena: An Open Platform for Comprehensive Comparison of {LLM} Routers},\nauthor={Yifan Lu and Rixin Liu and Jiayi Yuan and Xingqi Cui and Shenrun Zhang and Hongyi Liu and Jiarong Xing},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=9HsaIi4ngF}\n}"},"title":{"value":"RouterArena: An Open Platform for Comprehensive Comparison of LLM Routers"},"pdf":{"value":"/pdf/03e5d4326116c387dbd69c38e194211db0736cda.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"lu|routerarena_an_open_platform_for_comprehensive_comparison_of_llm_routers"},"authorids":{"value":["~Yifan_Lu8","~Rixin_Liu1","~Jiayi_Yuan1","~Xingqi_Cui1","~Shenrun_Zhang1","~Hongyi_Liu5","~Jiarong_Xing1"]},"authors":{"value":["Yifan Lu","Rixin Liu","Jiayi Yuan","Xingqi Cui","Shenrun Zhang","Hongyi Liu","Jiarong Xing"]}},"version":2},{"content":{"comment":{"value":"We would like to thank the reviewer for their thoughtful comments and feedback. Please find new experiments and proposed edits in the new version of the PDF, highlighted in blue.\n\n> Some parts of the writing are unclear, e.g., in Table 3, we don't know the name of each model.\n\nWe extend the table caption to include more model descriptions:\n\n*“Performance when directly employing the trained switcher model for other model settings: PFA from another model family (qwen 2.5 7B), PA only (of Tulu-v3), PF and two versions of A (of Tulu-v3, one DPO and one RLVR), and three specialized LM experts (seperately fine-tuned versions of qwen 2.5 7b). The trained switcher consistently generalizes to these settings to varying extents.”*\n\nNow updated in the revised pdf.\n\n> The size of the model selector seems to be the same as the pretrained and aligned models, which raises efficiency concerns. It would be nice to see either the result of a smaller selector model or an efficiency analysis. You mentioned that the switcher is a \"small\" model in section 2, but it's actually the same size of the pretrained and aligned models (8B), is there a reason for this?\n\nIn the model development lifecycle for any lab/company, the pretrained, finetuned, aligned LMs share the same size and architecture, because they are further-trained versions of each other. This is the setting we target in this work: model collaboration done in-house, with different model versions in the development lifecycle, without introducing external models. Thus it makes most sense to use one of the versions to initialize the switcher.\n\nHowever the methodology does not prevent using other LMs as the switcher. We try two extra:\n\n|                      | PopQA |  BBH  | CocoNot |\n|:--------------------:|:-----:|:-----:|:-------:|\n|      pretrained      | 15.30 | 38.10 |  11.90  |\n|       finetuned      | 26.10 | 26.70 |  64.00  |\n|        aligned       | 31.20 | 35.20 |  53.10  |\n|     qwen 2.5 1.5b    | 42.10 | 50.90 |  68.20  |\n|      gemma 2 2b      | 40.70 | 51.20 |  68.90  |\n| tulu-v3 8b (default) | 44.20 | 56.20 |  72.80  |\n\nResults show that 1) larger switcher LMs are better & 2) even for 1-2B switcher LMs, switch generation is still better than using any model individually.\n\nNow added as Table 20 in the updated pdf.\n\n> In the main experiment, has any of the baseline methods finetuned on these tasks (i.e., is the comparison with switch-g fair)? Has any of the baseline methods finetuned specifically for each of these tasks (i.e., is the comparison with switch-t fair)?\n\nYes: the routellm, text collab, logit merge, proxy tuning, greedy soup, and dare ties baselines are trained/optimized/hyperparameter-tuned/… (because they need to) on each specific task, so it is fair comparison with switch-t (stronger than switch-g)."},"title":{"value":"Author Response"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763423231109,"tcdate":1763421695624,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission14125/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission14125/Authors"],"forum":"1TTeOEufHz","number":7,"license":"CC BY 4.0","cdate":1763421695624,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission14125/-/Official_Comment"],"mdate":1763423231109,"domain":"ICLR.cc/2026/Conference","replyto":"9PuxNts3Sm","id":"tJbi1rchD0","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["model collaboration","collaborative inference"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Alignment training has tradeoffs: it helps language models (LMs) gain in reasoning and instruction following but might lose out on skills such as creativity and calibration, where unaligned base models are better at. We aim to make the best of both worlds through model collaboration, where different models in the training pipeline collaborate and complement each other. Since LM responses feature interleaving skills that favor different models, we propose Switch Generation, where pretrained and aligned model versions take turns to ``speak'' in a response sequence. Specifically, we train a switcher LM by learning from outcomes of choosing different models to generate the next segment across diverse queries and contexts. At inference time, the switcher LM guides different model checkpoints to dynamically generate the next segment where their strengths are most needed. Extensive experiments with 8 model collaboration baselines and 18 datasets show that 1) model collaboration consistently outperforms individual models on 16 out of 18 tasks, and 2) Switch Generation further outperforms baselines by 12.9% on average. Further analysis reveals that Switch Generation discovers compositional skills to solve problems where individual models struggle and generalizes to unseen models and tasks, reusing and repurposing by-products in expensive model training pipelines that are otherwise discarded."},"_bibtex":{"value":"@inproceedings{\nfeng2026dont,\ntitle={Don't Throw Away Your Pretrained Model},\nauthor={Shangbin Feng and Wenhao Yu and Yike Wang and Hongming Zhang and Yulia Tsvetkov and Dong Yu},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=1TTeOEufHz}\n}"},"title":{"value":"Don't Throw Away Your Pretrained Model"},"pdf":{"value":"/pdf/64f21733950004a23ae3d444dab7a7a241d57fdc.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"feng|dont_throw_away_your_pretrained_model"},"authorids":{"value":["~Shangbin_Feng1","~Wenhao_Yu2","~Yike_Wang1","~Hongming_Zhang2","~Yulia_Tsvetkov1","~Dong_Yu2"]},"authors":{"value":["Shangbin Feng","Wenhao Yu","Yike Wang","Hongming Zhang","Yulia Tsvetkov","Dong Yu"]}},"version":2},{"content":{"comment":{"value":"> The inference cost of Mistral7b is minimal, as the primary expense arises from GPT-4, and the cost of a single forward pass of Mistral is significantly lower. However, we will add to Table 1 and Figure 2 for further clarification.\n\nThank you!\n\n> The equation presented in Section 3.3 includes a parameter (\\alpha), which adjusts the influence of inference cost when computing the performance-to-cost ratio. This parameter represents the budget (B) in our optimization framework and operates independently of the query order.\n\nMakes sense. Thank you!\n\n> Performance Predictor: We propose the first LLM-based performance predictor capable of estimating the performance of individual LLMs without executing them.\n\n[3] (HybridLLM) uses DeBERTa for score prediction, [4] (RouteLLM) uses text-embedding-3-small for its matrix factorization router and the similarity-weighted ranking model and also explores BERT and Llama 3 8B based classifiers. During test time, all of these models predict performance of LLMs without executing the query on them.\n\n> We utilized the \"text-embedding-ada-002\" model from OpenAI [1] to embed the question-answer pairs from the training data defined in Section 4.2. Each question-answer pair was converted into a string using the format: <question> #question </question><answer> #answer </answer>. For efficient inference implementation, we utilized the Faiss library [2,3].\nDuring testing, the question field was converted into a string using the format: <question> #question </question>.\nWe then retrieved the top-5 related LLMs by applying cosine similarity function, then applied the cost-aware selection method defined in Section 3.3. Specifically, the estimated scores are the cosine similarity scores between the test question and the related question-answer pairs from the training data. Note: Johnson et al. [3] demonstrated that Faiss retrieval time is negligible during inference (2–3 ms per query).\n\nWhy would an embedding similarity between a question text to question+answer text be predictive of the correctness of the answer? The embedding model is not trained to do so nor does it have the reasoning capability. A better approach would be: for the query $q_{test}$ during test time, find the most similar query $q_{train}$ from training time, and use the LLM that can correctly answer $q_{train}$ with the lowest cost. To smoothen the noise, you can expand to topN retrieved $q_{train}$ queries and topK lowest budget LLMs that answer each $q_{train}$ query and pick the most frequently occurring LLM from that list.\n\nI am continuing to maintain my rating."}},"tmdate":1732456951532,"tcdate":1732456951532,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission11514/Reviewer_o5Db"],"signatures":["ICLR.cc/2025/Conference/Submission11514/Reviewer_o5Db"],"forum":"RQ9fQLEajC","number":5,"license":"CC BY 4.0","cdate":1732456951532,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission11514/-/Official_Comment"],"mdate":1732456951532,"domain":"ICLR.cc/2025/Conference","replyto":"zD1sZB3WO3","id":"0Ed07Wf6qH","forumContent":{"TLDR":{"value":"An LLM-based Router trained to route to the most effective LLM optimizing performance, cost, and efficiency, creating a new state-of-the-art model by integrating knowledge of underlying models."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["LLM","efficiency","knowledge integration","generation"]},"supplementary_material":{"value":"/attachment/2e84b1f1c381960311d11bdd23f4d081c4e4d14a.zip"},"primary_area":{"value":"generative models"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"LLMs with superior response quality—particularly larger or closed-source models—often come with higher inference costs, making their deployment inefficient and costly. Meanwhile, developing foundational LLMs from scratch is becoming increasingly resource-intensive and impractical for many applications. To address the challenge of balancing quality and cost, we introduce Routoo, an architecture designed to optimize the selection of LLMs for specific prompts based on performance, cost, and efficiency. Routoo provides controllability over the trade-off between inference cost and quality, enabling significant reductions in inference costs for a given quality requirement.\nRoutoo comprises two key components: a performance predictor and cost-aware selector. The performance predictor is a lightweight LLM that estimates the expected performance of various underlying LLMs on a given prompt without executing them. The cost-aware selector module then selects the most suitable model based on these predictions and constraints such as cost and latency, significantly reducing inference costs for the same quality. \nWe evaluated Routoo using the MMLU benchmark across 57 domains employing open-source models. Our results show that Routoo matches the performance of the Mixtral 8x7b model while reducing inference costs by one-third. Additionally, by allowing increased costs, Routoo surpasses Mixtral's accuracy by over 5\\% at equivalent costs, achieving an accuracy of 75.9\\%. When integrating GPT4 into our model pool, Routoo nearly matches GPT4's performance at half the cost and exceeds it with a 25\\% cost reduction. \nThese outcomes highlight Routoo's potential to significantly reduce inference costs without compromising quality, and even to establish new state-of-the-art results by leveraging the collective capabilities of multiple LLMs."},"_bibtex":{"value":"@misc{\nmohammadshahi2025routoo,\ntitle={Routoo: Learning to Route to Large Language Models Effectively},\nauthor={Alireza Mohammadshahi and Arshad Rafiq Shaikh and Majid Yazdani},\nyear={2025},\nurl={https://openreview.net/forum?id=RQ9fQLEajC}\n}"},"title":{"value":"Routoo: Learning to Route to Large Language Models Effectively"},"pdf":{"value":"/pdf/454cdce2a12d959ded0bc50a6c5b05c494a2d4dd.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"mohammadshahi|routoo_learning_to_route_to_large_language_models_effectively"},"authorids":{"value":["~Alireza_Mohammadshahi1","~Arshad_Rafiq_Shaikh1","~Majid_Yazdani1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Alireza Mohammadshahi","Arshad Rafiq Shaikh","Majid Yazdani"]}},"version":2},{"content":{"comment":{"value":"Thank you for your thoughtful feedback. To address your concerns more comprehensively, we provide detailed responses with evidence below:\n\n> **Q1: Relevance of model routing as models evolve.**\n\nA1: While we acknowledge that foundation models are evolving and becoming more versatile, we want to emphasize that model routing remains highly relevant, supported by both current empirical evidence and enduring constraints that will persist regardless of advancements in foundation models.\n\n1. **Current State:** While general-purpose foundation models have made significant progress, no single model achieves universal expertise across all domains. Task-specific optimization continues to produce specialized models (e.g., Qwen3-coder, Qwen2.5-Math) that outperform generalist alternatives in their respective areas, underscoring **the continued importance of model routing**. The increase in routing research at premier conferences (e.g., [1][2][3][4][5]) further highlights the ongoing relevance of this area. Additionally, platforms like Not Diamond and LangChain leverage routing to integrate specialized models effectively.\n    \n2. **Future Outlook:** Even as foundation models grow more capable, **structural limitations ensure the continued relevance of intelligent routing mechanisms**. Industries handling proprietary or sensitive data (e.g., financial systems, medical databases) face legal, ethical, and competitive barriers preventing direct integration into public foundation models. These constraints necessitate domain-specific solutions that rely on robust model-routing architectures to integrate specialized knowledge seamlessly within broader AI ecosystems.\n    \n\n[1] BEST-Route: Adaptive LLM Routing with Test-Time Optimal Compute. ICML 2025.\n\n[2] Capability Instruction Tuning: A New Paradigm for Dynamic LLM Routing. AAAI 2025.\n\n[3] RouteLLM: Learning to Route LLMs from Preference Data. ICLR 2025.\n\n[4] RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Models. NeurIIPS 2024.\n\n[5] AutoMix: Automatically Mixing Language Models. NeurIPS 2024.\n\n> **Q2: Regarding the novelty of latent response representation prediction.**\n\nA2: To clarify, the response representation prediction in Lookahead is **not merely an auxiliary task but a core innovation that fundamentally transforms the routing** **paradigm**.\n\n1. **Beyond an auxiliary task:** Traditional auxiliary tasks serve as regularizers to enhance representation robustness. In contrast, Lookahead's response modeling is integral to routing decisions. Our preliminary study in Section 1 clearly demonstrates that routing performance critically depends on access to response semantics, which our latent representations directly encode.\n    \n2. **Empirical evidence:** As shown in Section 5.4 and Appendix D (Fig. 5a, 6a), Lookahead's **performance gains persist with increasing** **training data**. This indicates that response modeling captures intrinsic task-specific signals rather than acting as a simple regularizer.\n\nIn essence, Lookahead's response-aware framework shifts routing from a query-prediction paradigm to a response-anticipation paradigm, where **latent representations serve as the causal bridge between input and optimal model selection**. This is fundamentally different from auxiliary multi-task learning.\n\nWe hope these clarifications address your concerns. Please feel free to reach out if you have any follow-up questions."}},"parentInvitations":"NeurIPS.cc/2025/Conference/-/Official_Comment","tmdate":1761724828768,"tcdate":1754407290668,"writers":["NeurIPS.cc/2025/Conference","NeurIPS.cc/2025/Conference/Submission25372/Authors"],"signatures":["NeurIPS.cc/2025/Conference/Submission25372/Authors"],"forum":"DRIRD9ELMb","number":3,"license":"CC BY 4.0","cdate":1754407290668,"readers":["everyone"],"invitations":["NeurIPS.cc/2025/Conference/Submission25372/-/Official_Comment","NeurIPS.cc/2025/Conference/-/Edit"],"mdate":1761724828768,"domain":"NeurIPS.cc/2025/Conference","replyto":"kmwKTU6WZ8","id":"UneSs66DID","forumContent":{"venue":{"value":"NeurIPS 2025 poster"},"keywords":{"value":["LLM Routing","Large Language Models"]},"supplementary_material":{"value":"/attachment/0aa39ac7e46cd3dacb2b72dfb53be1ef46888584.zip"},"primary_area":{"value":"deep_learning"},"abstract":{"value":"Large language model (LLM) routers improve the efficiency of multi-model systems by directing each query to the most appropriate model while leveraging the diverse strengths of heterogeneous LLMs. Most existing approaches frame routing as a classification problem based solely on the input query. While this reduces overhead by avoiding inference across all models, it overlooks valuable information that could be gleaned from potential outputs and fails to capture implicit intent or contextual nuances that often emerge only during response generation. These limitations can result in suboptimal routing decisions, particularly for complex or ambiguous queries that require deeper semantic understanding. To address this challenge, we propose Lookahead, a routing framework that \"foresees\" potential model outputs by predicting their latent representations and uses these predictions to guide model selection, thus enabling more informed routing without full inference. Within this framework, we implement two approaches based on causal and masked language models. Empirical evaluations across seven public benchmarks—spanning instruction following, mathematical reasoning, and code generation—show that Lookahead consistently outperforms existing routing baselines, achieving an average performance gain of 7.7\\% over the state-of-the-art. Our code is available at https://github.com/huangcb01/lookahead-routing."},"_bibtex":{"value":"@inproceedings{\nhuang2025lookahead,\ntitle={Lookahead Routing for Large Language Models},\nauthor={Canbin Huang and Tianyuan Shi and Yuhua Zhu and Ruijun Chen and Xiaojun Quan},\nbooktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},\nyear={2025},\nurl={https://openreview.net/forum?id=DRIRD9ELMb}\n}"},"title":{"value":"Lookahead Routing for Large Language Models"},"pdf":{"value":"/pdf/552fece3eff509f39e0f54fafd8aafc36248c8a8.pdf"},"venueid":{"value":"NeurIPS.cc/2025/Conference"},"paperhash":{"value":"huang|lookahead_routing_for_large_language_models"},"authorids":{"value":["~Canbin_Huang1","~Tianyuan_Shi1","~Yuhua_Zhu3","~Ruijun_Chen4","~Xiaojun_Quan1"]},"authors":{"value":["Canbin Huang","Tianyuan Shi","Yuhua Zhu","Ruijun Chen","Xiaojun Quan"]}},"version":2},{"content":{"comment":{"value":"[Table 1] ECE and APGR of model cascading on more uncalibrated settings. (ImageNet-C, Severity=5)\n| APGR(ECE)             | gaussian_noise | motion_blur |   fog   |\n| :--------------: | :-------------: | :----------: | :-----: |\n| before_calibrated |     0.6364(0.0584)      |    0.5999(0.1602)    | 0.5409(0.3140)  |\n| After_calibrated  |     0.6364(0.0537)      |    0.6122(0.0589)    | 0.6888(0.0285)  |\n\n\n[Table 2] APGR comparison of model cascading methods on language tasks.\n| Method                               |   MMLU   |  GSM8K  |\n| :-----------------------------------: | :------: | :-----: |\n| Random Baseline                      |  0.5000  | 0.5000  |\n| Matrix Factorization (Pretrained)     |  0.5418  | 0.5672  |\n| Matrix Factorization (Trained)        |  0.5267  | 0.6234  |\n| Matrix Factorization (Finetuned)      |  0.5497  | 0.5718  |\n| Calibrated confidence cascading (Histogram Binning[5], Training-Free, 15 bins)      |  0.7006  | 0.7489  |\n| Calibrated confidence cascading (Temperature Scaling, 15 bins)    |  0.7013  | 0.7593  |\n| Calibrated confidence cascading (Platt Scaling, 15 bins)          |  0.7013  | 0.7496  |\n\n[Table 3] Performance comparison of model cascading methods on language tasks.\n| Method                               | Active Param | Top-1 ACC |\n| :-----------------------------------: | :----------: | :--------: |\n| Single model (DeiT-B)      |   86.57      |   81.98    |\n| Single model (TinyViT)      |   11.00               |   81.53          | \n| MoEE (DeiT-B)       |   58.55      |   80.12    |\n| Calibrated confidence cascading (DeiT-B + TinyViT) (Temperature Scaling, 15 bins K=10) |   39.85   |   90.46   |\n| Calibrated confidence cascading (DeiT-B + TinyViT) (Histogram Binning, Training-Free, 15 bins K=10) |   39.85   |   90.46   |\n\n**Q3: “... frameworks are verified with limited comparisons… routing methods which should also be included for comparison.”**\n\n**A3**: As suggested, we have added the following comparisons.  \nFor **language tasks**, we compare with a widely cited routing-based method [1].  \nFor **vision tasks**, we compare with an MoE-based approach [3].  \nTo ensure faithfulness and practicality under rebuttal-time constraints, we adopt the same baselines and follow the original experimental protocols used in the corresponding papers.\n\nFor the routing baseline in language models, we make our best effort to faithfully reproduce its performance under three settings:\n\n- **Using the provided pretrained checkpoint**, as the original work claims:\n  *“our routers have learned common characteristics of queries that allow them to distinguish between strong and weak models, generalizing to new models at inference time without additional training”* [1].\n\n- **Fine-tuning the pretrained checkpoint** on the target task dataset.\n\n- **Training the router from scratch** on the target task dataset.\n\nAs shown in *Table 2*, across all settings, the results consistently show that the proposed method **significantly outperforms these baselines while maintaining its simplicity**, suggesting that the potential of simple approaches has not yet been fully explored.\n\nFor the vision-model baseline, we follow the original paper and use the same **DeiT model** as the large model. In addition, we introduce a much smaller **TinyViT** model as the lightweight model for cascading.  \nNotably, we observe that combining two models with individual accuracies in the **81–82%** range yields a cascading accuracy of **90%**, while simultaneously reducing the total number of activated parameters during inference by a substantial margin, as shown in *Table 3*.\n\n**Q4: Minor: typo around Line 389: \"as show\".**\n**A4**: Thank you for pointing this out. We have corrected the typo.\n\n[1] RouteLLM: Learning to Route LLMs with Preference Data. ICLR 2025\n\n[2] Confident Learning: Estimating Uncertainty in Dataset Labels. JAIR\n\n[3] Efficient Data Driven Mixture-of-Expert Extraction from Trained Networks. CVPR 2025\n\n[4] Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers. ICML 2001"},"title":{"value":"Author Response to Reviewer xhYV (2/2)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764384312447,"tcdate":1764384242940,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission8659/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission8659/Authors"],"forum":"imzDS4N8CW","number":14,"license":"CC BY 4.0","cdate":1764384242940,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission8659/-/Official_Comment"],"mdate":1764384312447,"domain":"ICLR.cc/2026/Conference","replyto":"wJwYFOLbpo","id":"hHrPXcTcqS","forumContent":{"TLDR":{"value":"a general framework that enables a model to know when it does not know and to make use of this property"},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Model Calibration","Model Cascades","Data Cleaning"]},"primary_area":{"value":"alignment, fairness, safety, privacy, and societal considerations"},"abstract":{"value":"When a model knows when it does not know, many possibilities emerge. The first question is how to enable a model to recognize that it does not know. A promising approach is to use confidence, computed from the model’s internal signals, to reflect its ignorance. Prior work in specific domains has shown that calibration can provide reliable confidence estimates. In this work, we propose a simple, effective, and universal training-free method that applies to both vision and language models, performing model calibration, cascading, and data cleaning to better exploit a model's ability to recognize when it does not know. We first highlight two key empirical observations: higher confidence corresponds to higher accuracy within a single model, and models calibrated on the validation set remain calibrated on a held-out test set. These findings empirically establish the reliability and comparability of calibrated confidence. Building on this, we introduce two applications: 1. Model cascading with calibrated advantage routing and 2. Data cleaning based on mixture of experts. Using the routing signal derived from the comparability of calibrated confidences, we cascade large and small models to improve efficiency with almost no compromise in accuracy, and we further cascade two large state-of-the-art models to achieve performance beyond either model alone. Leveraging multiple experts and their calibrated confidences, we design a simple yet effective data-cleaning method that balances precision and detection rate to identify mislabeled samples in ImageNet and Massive Multitask Language Understanding (MMLU) datasets. Our results demonstrate that enabling models to recognize when they do not know is a practical step toward more efficient, reliable, and trustworthy AI."},"_bibtex":{"value":"@misc{\nhao2026when,\ntitle={When Models Know When They Do Not Know: Calibration, Cascading and Cleaning},\nauthor={Chenjie Hao and Weyl Lu and Yuko Ishiwaka and ZENGYI LI and Weier Wan and Yubei Chen},\nyear={2026},\nurl={https://openreview.net/forum?id=imzDS4N8CW}\n}"},"title":{"value":"When Models Know When They Do Not Know: Calibration, Cascading and Cleaning"},"pdf":{"value":"/pdf/15f153d32eedccd6d5def2e3c98e8ad6d40eaffb.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"hao|when_models_know_when_they_do_not_know_calibration_cascading_and_cleaning"},"authorids":{"value":["~Chenjie_Hao1","~Weyl_Lu1","~Yuko_Ishiwaka1","~ZENGYI_LI1","~Weier_Wan1","~Yubei_Chen1"]},"authors":{"value":["Chenjie Hao","Weyl Lu","Yuko Ishiwaka","ZENGYI LI","Weier Wan","Yubei Chen"]}},"version":2},{"content":{"comment":{"value":"## **W2: Another issue is the computational cost of the method. The method does seem more costly than certain router baselines, but performs slightly better. Have the authors considered potentially skipping an LLM when say the first model's confidence is very low, or outputting more nuanced reject answers, such as \"I don't now.\", \"I really don't know.\", and depending on such output, skip an LLM or two in the chain?**\n\nWe appreciate this practical perspective, which was echoed by other reviewers. To address concerns regarding latency and overhead, we have added a detailed statistical analysis of actual inference and routing costs in **Appendix A.1** of the revised paper.\n\n### **Latency Analysis**\n\n1. **Negligible Routing Overhead:** Our empirical data (**Table 6**) shows that although a query may be rejected by multiple agents before being solved, the **cumulative routing cost** (time spent generating rejections) remains negligible. Even in the worst-case scenario (routing to the final agent), this overhead constitutes less than **5%** of the actual inference cost of generating the final answer (**Table 7**). \n2. **Competitive Latency:** We also benchmarked **System Latency** (Time to First Token, **Table R4 and Table 8 in paper**). Results indicate that while DiSRouter incurs slightly higher latency than predictive routers (e.g., RouteLLM) which use lightweight classifiers, it is significantly faster than generative baselines that rely on full model outputs or verification steps (e.g., FrugalGPT, AutoMix).\n\n*Table R4: Time to First Token (s) Results Comparison for Different Routing Systems.*\n\n|  | RouterLLM | FrugalGPT | Automix | FORC | GraphRouter | DiSRouter |\n| --- | --- | --- | --- | --- | --- | --- |\n| Average | 0.048 | 2.620 | 3.475 | 0.070 | 0.024 | 0.081 |\n| Worst Case | 0.063 | 9.565 | 11.34 | 0.076 | 0.028 | 0.241 |\n\n### **Design of Cross-Level Routing**\n\nWe appreciate this forward-looking suggestion. We have actually initiated preliminary research into it, where agents are conditioned on system-level information to potentially route difficult queries directly to a much larger model, skipping intermediate steps. However, our initial experimental results showed no significant improvement in **system Utility**. We attribute this to two primary factors:\n\n1. **Complexity of Routing Precision:** Accurately determining which specific future agent (e.g., skipping from 1.5B to 14B vs. 7B) is required is a significantly harder problem than the binary decision of determining one's own incompetence. Our current algorithm for this fine-grained skipping is not yet precise enough to outweigh the risks of over-routing (using too large a model) or under-routing.\n2. **Marginal Cost Benefits:** As discussed in our response regarding routing overhead (and in **Appendix A.1**), the cost of traversing the cascade is already very low (<5% of inference cost). Therefore, the latency savings gained by \"skipping a level\" are minimal. The primary theoretical value of skipping would be to bypass intermediate agents that might confidently answer incorrectly. However, since our Self-Awareness Training already ensures agents reject when uncertain, this \"error skipping\" value is also diminished.\n\nWe consider cross-level routing a promising direction for optimizing extremely deep cascades or heterogeneous agent pools and plan to explore it further in future work."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764157291610,"tcdate":1764157291610,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7635/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission7635/Authors"],"forum":"KDcwXKr0NU","number":13,"license":"CC BY 4.0","cdate":1764157291610,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7635/-/Official_Comment"],"mdate":1764157291610,"domain":"ICLR.cc/2026/Conference","replyto":"2E7rUSRc1n","id":"nkycQnT7To","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"TLDR":{"value":"We introduce DiSRouter, a distributed system where LLMs leverage \"self-awareness\" to route queries among themselves, outperforming conventional centralized routers."},"keywords":{"value":["query routing","model selection","distributed system","self-awareness of LLM"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"The proliferation of Large Language Models (LLMs) has created a diverse ecosystem of models with highly varying performance and costs, necessitating effective query routing to balance performance and expense. Current routing systems often rely on a centralized external router trained on a fixed set of LLMs, making them inflexible and prone to poor performance since the small router can not fully understand the knowledge boundaries of different LLMs. We introduce DiSRouter (Distributed Self-Router), a novel paradigm that shifts from centralized control to distributed routing. In DiSRouter, a query traverses a network of LLM agents, each independently deciding whether to answer or route to other agents based on its own self-awareness—its ability to judge its competence. This distributed design offers superior flexibility, scalability, and generalizability. To enable this, we propose a two-stage Self-Awareness Training pipeline that enhances each LLM's self-awareness. Extensive experiments demonstrate that DiSRouter significantly outperforms existing routing methods in utility across various scenarios, effectively distinguishes between easy and hard queries, and shows strong generalization to out-of-domain tasks. Our work validates that leveraging an LLM's intrinsic self-awareness is more effective than external assessment, paving the way for more modular and efficient multi-agent systems."},"_bibtex":{"value":"@inproceedings{\nzheng2026disrouter,\ntitle={Di{SR}outer: Distributed Self-Routing for {LLM} Selections},\nauthor={Hang Zheng and Hongshen Xu and Yongkai.lin and Shuai Fan and Lu Chen and Kai Yu},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=KDcwXKr0NU}\n}"},"title":{"value":"DiSRouter: Distributed Self-Routing for LLM Selections"},"pdf":{"value":"/pdf/0190f98d94890b700d03febdee678e22a2a5e380.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"zheng|disrouter_distributed_selfrouting_for_llm_selections"},"authorids":{"value":["~Hang_Zheng3","~Hongshen_Xu1","~Yongkai.lin1","~Shuai_Fan1","~Lu_Chen3","~Kai_Yu3"]},"authors":{"value":["Hang Zheng","Hongshen Xu","Yongkai.lin","Shuai Fan","Lu Chen","Kai Yu"]}},"version":2},{"content":{"comment":{"value":"Dear AC and Reviewers,\n\nWe sincerely thank you for your insightful reviews and constructive feedback. We are encouraged by the consensus that:\n\n1.  **Novelty:** Our market-based perspective, treating uncertainty as a tradable asset to resolve information asymmetry, is novel and conceptually powerful (Reviewers $oqqC$, $VAy3$).\n2.  **Theoretical Foundation:** The formalization of \"Agnostic Coordination\" and the Inefficiency Theorem provides a strong theoretical grounding for the problem (Reviewer $d1Fk$, $VAy3$).\n3.  **Effectiveness:** Agora demonstrates substantial gains in accuracy (e.g., +8.5% on MMMU) and cost-efficiency across multiple benchmarks (Reviewers $oqqC$, $iVUH$).\n\nWe have updated our manuscript to incorporate your suggestions. Changes are highlighted in the revised PDF. Below, we address the three most common concerns regarding **Theoretical Optimality**, **Computational Complexity**, and **Generalization**.\n\n>**1. Local vs. Global Optimality (Response to $iVUH$, $d1Fk$)**\nA primary theoretical concern was whether Agora converges to a global optimum. We clarified in Section 3.3 that finding a global optimum in a decentralized, heterogeneous agent market is NP-hard. Therefore, Agora is designed to guarantee convergence to a **Locally Optimal Equilibrium** via deterministic greedy descent on the global cost function.\n\nTo quantify the quality of this equilibrium, we added an **Optimality Gap Analysis** (Appendix D). We constructed a Brute-Force Global Oracle on a subset of MMBench tasks. The results (Table R1 below) show that Agora's cost is only **0.80%** higher than the theoretical global minimum, while significantly outperforming greedy and random baselines. This empirically confirms that our Market-Aware Broker (MAB) initialization effectively guides the system into a high-quality basin of attraction.\n\n**Table R1. Optimality Gap Analysis (from Appendix D)**\n\n| Method                    | Avg. Cost (Normalized) | Gap to Global Opt. | Accuracy (%) |\n| :------------------------ | :--------------------- | :----------------- | :----------- |\n| Global Optimal (Oracle)   | 1.000                  | 0.00%              | 89.6         |\n| **Agora (Decentralized)** | **1.008**              | **+0.80%**         | **89.5**     |\n| Greedy (Local Only)       | 1.152                  | +15.2%             | 84.3         |\n\n---\n\n\n---\n\n\n>**2. Computational Complexity & Real-Time Viability (Response to $oqqC$, $VAy3$)**\nConcerns were raised regarding the overhead of the Broker and trading protocol. We have clarified in Section 3.3 and 4.1 that:\n\n* **Complexity:** The Broker selection is $O(N)$ and the trading protocol is $O(N^2)$. For typical pool sizes ($N \\le 15$), these operations consume milliseconds.\n* **End-to-End Latency:** The \"Average Inference Time\" reported in our experiments (Figure 4 & Table 9) is **strictly end-to-end**, encompassing all coordination overheads. Agora achieves state-of-the-art accuracy with inference times (2.5s - 3.5s) comparable to standard routing baselines.\n* **Tractability:** We added Appendix M.3 to explain that the Strategic Uncertainty Index ($U_{strategic}$) uses a closed-form marginal approximation, avoiding expensive model re-execution.\n\n>**3. Robustness and Cross-Dataset Generalization (Response to $d1Fk$, $oqqC$)**\nTo address concerns about fairness and generalization:\n\n* **Fairness:** We implemented a **Unified Multimodal Wrapper** for text-centric baselines (FrugalGPT, RouteLLM) to ensure fair multimodal comparison (Section 4.1).\n* **Generalization:** We extended the head-to-head comparison against strong baselines (MoA, KABB) to three additional datasets (MathVision, InfoVQA, CC-OCR) in Appendix E.3. Agora consistently outperforms baselines across reasoning-heavy and perception-heavy tasks."},"title":{"value":"General Response (1)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763927080779,"tcdate":1763785052370,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission582/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission582/Authors"],"forum":"zeqCjGQB4U","number":9,"license":"CC BY 4.0","cdate":1763785052370,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission582/-/Official_Comment"],"mdate":1763927080779,"domain":"ICLR.cc/2026/Conference","replyto":"zeqCjGQB4U","id":"0ODm8GM5Ws","forumContent":{"TLDR":{"value":"We presents a framework that reframes coordination as a decentralized market for uncertainty."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["agent; Vision Language Model; Uncernity"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Vision-Language Models (VLMs) enable powerful multi-agent systems, but scaling them is economically unsustainable: coordinating heterogeneous agents under information asymmetry often spirals costs. Existing paradigms, such as Mixture-of-Agents and knowledge-based routers, rely on heuristic proxies that ignore costs and collapse uncertainty structure, leading to provably suboptimal coordination.\nWe introduce Agora, a framework that reframes coordination as a decentralized market for uncertainty. Agora formalizes epistemic uncertainty into a structured, tradable asset (perceptual, semantic, inferential), and enforces profitability-driven trading among agents based on rational economic rules. A market-aware broker, extending Thompson Sampling, initiates collaboration and guides the system toward cost-efficient equilibria. Experiments on five multimodal benchmarks (MMMU, MMBench, MathVision, InfoVQA, CC-OCR) show that Agora outperforms strong VLMs and heuristic multi-agent strategies, e.g., achieving +8.5% accuracy over the best baseline on MMMU while reducing cost by over 3×. These results establish market-based coordination as a principled and scalable paradigm for building economically viable multi-agent visual intelligence systems."},"_bibtex":{"value":"@inproceedings{\nzhang2026why,\ntitle={Why Keep Your Doubts to Yourself? Trading Visual Uncertainties among Vision-Language Models},\nauthor={Jusheng Zhang and Yijia Fan and Kaitong Cai and Jing Yang and Jiawei Yao and Jian Wang and Guanlong Qu and Ziliang Chen and Keze Wang},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=zeqCjGQB4U}\n}"},"title":{"value":"Why Keep Your Doubts to Yourself? Trading Visual Uncertainties among Vision-Language Models"},"pdf":{"value":"/pdf/03f4bead771e441ade14013c814de209cd0630ea.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"zhang|why_keep_your_doubts_to_yourself_trading_visual_uncertainties_among_visionlanguage_models"},"authorids":{"value":["~Jusheng_Zhang3","~Yijia_Fan1","~Kaitong_Cai1","~Jing_Yang39","~Jiawei_Yao3","~Jian_Wang10","~Guanlong_Qu2","~Ziliang_Chen1","~Keze_Wang1"]},"authors":{"value":["Jusheng Zhang","Yijia Fan","Kaitong Cai","Jing Yang","Jiawei Yao","Jian Wang","Guanlong Qu","Ziliang Chen","Keze Wang"]}},"version":2},{"content":{"comment":{"value":"Dear reviewer iqRc,\n\nSincerely thanks for your constructive feedback and positive assessment of our work. We address each concern and question in detail below.\n\n---\n\n**Q1**: Are the router training samples (prompts and responses) taken from the benchmarks? If so, are they taken from all the benchmarks/some? What is a typical size of the router training set?\n\n**R1**: Yes, the router training samples (i.e., prompts and responses) are taken from the benchmark datasets used in our experiments. For each main experiment, we train the router on the training split of the corresponding benchmark (e.g., CSQA, ARC-Challenge, MMLU-PP, GSM8k, or MATH) and evaluate it on the respective test set. In each case, the training set comprises all available samples from the benchmark's training split (see Table 1 in the paper for exact dataset sizes). Specifically, the CommonsenseQA training set contains 9,741 examples; ARC-Challenge has 1,119 examples; MMLU-PP includes 612 samples; GSM8k and MATH have 7,473 and 7,500 samples, respectively.\n\n---\n\n**Q2**: How generalizable is the trained router to new tasks, benchmarks, or domains?\n\n**R2**: We appreciate the reviewer for raising this point. In our paper, we primarily focus on in-domain experiments, where CITER is trained and evaluated on the training and test sets of the same benchmark dataset. However, we note that the router’s input features (i.e., the hidden states of the SLM) are generic representations, which indicates a potential for generalization across different domains. To explore this, we conducted additional experiments where we trained our router on a mixture of all the datasets used in our study, including CSQA, ARC-Challenge, MMLU-PP, GSM8k, and MATH, and then evaluated its performance on the OpenBookQA dataset. The results are presented in the table below.\n\nDue to the limitations of the rebuttal system, we are unable to include figures or plots in our response. Therefore, we present our results in table format, where each row corresponds to a different method, each column represents a specific value of the x-axis variable (e.g., data transfer amount), and each cell shows the accuracy achieved under that setting. To facilitate a fair and direct comparison between methods, we performed linear interpolation on the original data to obtain accuracy values for all methods at the same set of x-axis points. This approach ensures that our comparisons are consistent and transparent across all methods and conditions.\n\n**Table R2**: Performance comparison between CITER and our baselines on the OpenBookQA dataset. Higher values indicate better performance. The best result is **bolded**.\n| Accuracy (%) \\ Data Transfer Amount (GB) | 7.1  | 13.4     | 50.0     | 100.0    | 150.0    | 179.3    | 191.1 |\n| ---------------------------------------- | ---- | -------- | -------- | -------- | -------- | -------- | ----- |\n| SLM                                      | 56.6 | /        | /        | /        | /        | /        | /     |\n| LLM                                      | /    | /        | /        | /        | /        | /        | 91.1  |\n| Speculative Decoding                     | /    | /        | /        | /        | /        | **91.1** | /     |\n| RouteLLM                                 | /    | 58.0     | 64.7     | 73.8     | 83.6     | 88.8     | /     |\n| Co-LLM                                   | /    | 59.6     | 66.9     | 76.7     | 86.2     | 89.4     | /     |\n| CITER (Ours)                             | /    | **60.0** | **67.3** | **79.1** | **89.8** | 90.6     | /     |\n\nThese results demonstrate that CITER continues to outperform all baselines on the new dataset, highlighting the generalizability of our approach."},"title":{"value":"Response to Reviewer iqRc (1/3)"}},"parentInvitations":"colmweb.org/COLM/2025/Conference/-/Official_Comment","tmdate":1756226122501,"tcdate":1748920378004,"writers":["colmweb.org/COLM/2025/Conference","colmweb.org/COLM/2025/Conference/Submission1200/Authors"],"signatures":["colmweb.org/COLM/2025/Conference/Submission1200/Authors"],"forum":"nqX9UYW9Af","number":8,"license":"CC BY 4.0","cdate":1748920378004,"readers":["everyone"],"invitations":["colmweb.org/COLM/2025/Conference/Submission1200/-/Official_Comment","colmweb.org/COLM/2025/Conference/-/Edit"],"mdate":1756226122501,"domain":"colmweb.org/COLM/2025/Conference","replyto":"vLkAfoAzPa","id":"cDY6aJ6RIo","forumContent":{"TLDR":{"value":"A novel Collaborative Inference with Token-lEvel Routing (CITER) framework that introduces a token-level routing mechanism, enabling efficient collaboration between small and large language models (SLMs & LLMs)."},"venue":{"value":"COLM 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the COLM Code of Ethics on https://colmweb.org/CoE.html"},"keywords":{"value":["collaborative inference","efficient inference","token-level routing","large language model"]},"abstract":{"value":"Large language models have achieved remarkable success in various tasks but suffer from high computational costs during inference, limiting their deployment in resource-constrained applications. To address this issue, we propose a novel Collaborative Inference with Token-lEvel Routing (CITER) framework that enables efficient collaboration between small and large language models (SLMs & LLMs) through a token-level routing strategy. Specifically, CITER routes non-critical tokens to an SLM for efficiency and routes critical tokens to an LLM for generalization quality. We formulate router training as a policy optimization, where the router receives rewards based on both the quality of predictions and the inference costs of generation. This allows the router to learn to predict token-level routing scores and make routing decisions based on both the current token and the future impact of its decisions. To further accelerate the reward evaluation process, we introduce a shortcut which significantly reduces the costs of the reward estimation and improving the practicality of our approach. Extensive experiments on five benchmark datasets demonstrate that CITER reduces the inference costs while preserving high-quality generation, offering a promising solution for real-time and resource-constrained applications."},"_bibtex":{"value":"@inproceedings{\nzheng2025citer,\ntitle={{CITER}: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routing},\nauthor={Wenhao Zheng and Yixiao Chen and Weitong Zhang and Souvik Kundu and Yun Li and Zhengzhong Liu and Eric P. Xing and Hongyi Wang and Huaxiu Yao},\nbooktitle={Second Conference on Language Modeling},\nyear={2025},\nurl={https://openreview.net/forum?id=nqX9UYW9Af}\n}"},"title":{"value":"CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routing"},"pdf":{"value":"/pdf/44adc116c02c1f30777bdd97bc65accd40cb593a.pdf"},"venueid":{"value":"colmweb.org/COLM/2025/Conference"},"paperhash":{"value":"zheng|citer_collaborative_inference_for_efficient_large_language_model_decoding_with_tokenlevel_routing"},"authorids":{"value":["~Wenhao_Zheng4","~Yixiao_Chen2","~Weitong_Zhang2","~Souvik_Kundu2","~Yun_Li7","~Zhengzhong_Liu1","~Eric_Xing1","~Hongyi_Wang1","~Huaxiu_Yao1"]},"author_guide":{"value":"I certify that this submission complies with the submission instructions as described on https://colmweb.org/AuthorGuide.html"},"authors":{"value":["Wenhao Zheng","Yixiao Chen","Weitong Zhang","Souvik Kundu","Yun Li","Zhengzhong Liu","Eric P. Xing","Hongyi Wang","Huaxiu Yao"]}},"version":2},{"content":{"comment":{"value":"Thank you for your insightful review and feedback. We appreciate you raising these important points, and we will incorporate your suggestions into the revised manuscript to strengthen our work. Here are our responses to your specific comments:\n\n1. Regarding the presentation of experimental results, we agree that including tabular results alongside the figures would enhance clarity and interpretability. In the revised manuscript, we will incorporate tables presenting relevant metrics for the different benchmarks and LLM configurations, complementing the existing figures.\n2. We acknowledge your concern about the potential complexity introduced by integrating the multi-armed bandit framework with dynamic routing based on user preferences. To address this, we will provide an open-source implementation of our code, allowing others to examine and potentially extend our work. Additionally, we will include a discussion on the modular design and extensibility of our framework, highlighting how it can be efficiently implemented and scaled.\n3. Concerning the exploration of the routing policy components, you make a fair point. We did not extensively tune hyperparameters for certain components, such as model quizzing and incorporating pairwise comparison data. The exact hyperparameters we used are provided in appendix. In the revised manuscript, we will acknowledge this limitation and emphasize that further improvements are possible through more comprehensive hyperparameter tuning.\n4. Regarding the predictor baseline's performance, you raise an excellent point. In theory, if the performance prediction is accurate, the predictor policy should be optimal. Our routing policy aims to improve upon the predictor when those predictions are inaccurate or uncertain. For settings where our routing policy does not significantly outperform the predictor, it is likely due to the inherent difficulty in accurately predicting certain types of scores, such as discontinuous metrics like accuracy. We will include this analysis in the revised manuscript, as it provides valuable insights into the strengths and limitations of our approach.\n5. Concerning the efficiency analysis, we appreciate you highlighting this aspect. Both the IRT model and routing policy are relatively lightweight, consisting of only a few linear layers. Therefore, the computational overhead introduced by our framework, compared to running LLM inference, is relatively low. We will quantify this overhead in the revised manuscript.\n6. Regarding the MT-Bench comparison, thank you for bringing this up. We do have results on this benchmark, but we left them out of the initial submission as it is a relatively small dataset. Our routing policy obtains similar performance to RouteLLM. We agree that including these results would provide a more comprehensive evaluation. In the revised manuscript, we will add results on this dataset.\n\nWe sincerely appreciate your thorough review and constructive feedback. Addressing these points will significantly strengthen our work and contribute to a better understanding of the proposed routing framework's capabilities, limitations, and potential areas for further improvement.\n\nPlease let us know if you have any additional comments or suggestions."}},"tmdate":1731736843771,"tcdate":1731736843771,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission10949/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission10949/Authors"],"forum":"rEqETC88RY","number":1,"license":"CC BY 4.0","cdate":1731736843771,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission10949/-/Official_Comment"],"mdate":1731736843771,"domain":"ICLR.cc/2025/Conference","replyto":"EhCS0fnsh4","id":"zbWcH6Jb9e","forumContent":{"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["LLM Routing","Reinforcement Learning","Multi-objective Optimization"]},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The rapid advancement in large language models (LLMs) has brought forth a diverse range of models with varying capabilities that excel in different tasks and domains. However, selecting the optimal LLM for user queries often involves a challenging trade-off between accuracy and cost, a problem exacerbated by the diverse demands of individual queries. In this work, we present a novel framework that formulates the LLM selection process as a multi-armed bandit problem, enabling dynamic and intelligent routing of queries to the most appropriate model. Our approach incorporates a preference-conditioned dynamic routing mechanism, allowing users to specify their preferences at inference time, thereby offering a customizable balance between performance and cost. Additionally, our selection policy is designed to generalize to unseen LLMs, ensuring adaptability to new models as they emerge. Experimental results demonstrate that our method achieves significant improvements in both accuracy and cost-effectiveness across various LLM platforms, showcasing the potential of our framework to adaptively optimize LLM selection in real-world scenarios."},"_bibtex":{"value":"@misc{\nli2025llm,\ntitle={{LLM} Bandit: Cost-Efficient {LLM} Generation via Preference-Conditioned Dynamic Routing},\nauthor={Yang Li},\nyear={2025},\nurl={https://openreview.net/forum?id=rEqETC88RY}\n}"},"title":{"value":"LLM Bandit: Cost-Efficient LLM Generation via Preference-Conditioned Dynamic Routing"},"pdf":{"value":"/pdf/3965a78198d6f2123d2e62592cf27641212d9c5a.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"li|llm_bandit_costefficient_llm_generation_via_preferenceconditioned_dynamic_routing"},"authorids":{"value":["~Yang_Li19"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Yang Li"]}},"version":2},{"content":{"comment":{"value":"Dear ACs and Reviewers,\n\nWe sincerely appreciate your time and effort in reviewing our paper and providing constructive feedback. Besides the response to each reviewer, here we would like to further 1) thank reviewers for their recognition of our work and 2) highlight the major modifications in our revision:\n\n**1. We are glad that all reviewers appreciate and recognize our novelty and contribution.**\n* \"**Well-Motivated** Problem:...  clearly articulates the inefficiency of one-size-fits-all reasoning strategies.\"; \"**Clear Motivation & Principled Design**: ...\"[Reviewers opSX, Y9xi]\n* \"... provides **a novel mechanism** ...\"; \"This **innovative approach** improves interpretability and realism of model reasoning.\"; \"**a novel classification perspective** and a **new method** for understanding problem difficulty\" [Reviewers EgK3, Bhkw]\n* \"**Solid Experimental Results**: Achieves competitive accuracy ... significant **efficiency gains** ... emonstrates some **generalization capability**.\"; \"... achieving **notable gains in accuracy, efficiency**, and **reduced latency** compared to standard fixed or scaling-based strategies.\"; \"... **SOTA accuracy** on both ID and OOD tasks. The **efficiency gains are significant**, ...\" [Reviewers opSX, EgK3, Y9xi]\n\n**2. We summarize the main modifications in our revised paper (highlighted in blue).**\n* **Stronger and fairer baseline.** We add RouteLLM as a directly comparable routing baseline and report results on both ID and OOD datasets, and we update **Table 1** accordingly to include this stronger baseline. [Reviewers opSX]\n* **Broader evaluation.** We evaluate CogER on additional benchmarks, including MBPP (code generation), QuALITY (long-context QA), and Natural Questions (retrieval-augmented QA), to demonstrate generalization beyond the training distribution, and we report these results in the newly added **Table 6**. [Reviewers opSX]\n* **Efficiency and scalability analyses.** We include a controlled latency breakdown showing that the CogER-Agent adds only ~0.01s overhead per query, a data-scale ablation on the number of training samples, and a router-size ablation showing modest gains beyond 7B. The corresponding results are reported in **Tables 10, 12, and 14**.[Reviewers opSX, Y9xi]\n* **Clearer definition of the minimal sufficient level.** We provide a detailed description of how $L_{min}$ is estimated via deterministic runs of the four fixed levels, explain how stochasticity is avoided, and clarify that $L_min$ is only used as a shaping signal in the hierarchical reward, not as a test-time oracle. These details are provided in **Appendix D.3**.  [Reviewers opSX, EgK3, Bhkw]\n* **Comprehensive routing diagnostics.** We add (i) an oracle upper bound with perfect routing, (ii) per-level precision/recall/F1, and (iii) per-level accuracy for queries routed to each level, together with a quantitative breakdown of failures into routing vs. execution errors. The corresponding analyses are reported in **Tables 10, 11, 13, and 15**. [Reviewers opSX, Y9xi]\n* **Robustness, design clarifications, and limitations.** We clarify the design of CoTool and the L4 mode, describe how we handle tool errors and prevent format-reward gaming, and explain the factorized MDP action space. We further study robustness under model upgrades and add qualitative failure cases plus a more explicit discussion of current limitations and future work. These updates are reflected in **Section 4.2, Appendix D.1, Appendix E, and Appendix G.1**. [Reviewers opSX, EgK3, Bhkw]\n\n\nBest regards,\n\nThe Authors"},"title":{"value":"General Response"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764212364070,"tcdate":1764212364070,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission705/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission705/Authors"],"forum":"z9HUMon2Br","number":21,"license":"CC BY 4.0","cdate":1764212364070,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission705/-/Official_Comment"],"mdate":1764212364070,"domain":"ICLR.cc/2026/Conference","replyto":"z9HUMon2Br","id":"lsSyNCtz6K","forumContent":{"TLDR":{"value":"We propose the  Cognitive-Inspired Elastic Reasoning framework for efficient scaling of language model reasoning, which dynamically selects the most appropriate processing mode for each query."},"venue":{"value":"ICLR 2026 Conference Desk Rejected Submission"},"keywords":{"value":["Large language models; Reinforcement Learning; Markov decision process"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Large language models (LLMs) have demonstrated impressive performance across various language tasks. However, existing LLM reasoning strategies mainly rely on the LLM itself with fast or slow mode (like o1 thinking) and thus struggle to balance reasoning efficiency and accuracy across queries of varying difficulties. In this paper, we propose Cognitive-Inspired Elastic Reasoning (CogER), a framework inspired by human hierarchical reasoning that dynamically selects the most suitable reasoning strategy for each query. Specifically, CogER first assesses the complexity of incoming queries and assigns them to one of several predefined levels, each corresponding to a tailored processing strategy, thereby addressing the challenge of unobservable query difficulty. To achieve automatic strategy selection, we model the process as a Markov Decision Process and train a CogER-Agent using reinforcement learning. The agent is guided by a reward function that balances solution quality and computational cost, ensuring resource-efficient reasoning. Moreover, for queries requiring external tools, we introduce Cognitive Tool-Assisted Reasoning, which enables the LLM to autonomously invoke external tools within its chain-of-thought. Extensive experiments demonstrate that CogER outperforms state‑of‑the‑art Test‑Time scaling methods, achieving at least a 13% relative improvement in average exact match on In‑Domain tasks and an 8% relative gain on Out‑of‑Domain tasks."},"_bibtex":{"value":"@misc{\nanonymous2026beyond,\ntitle={Beyond Fast and Slow: Cognitive-Inspired Elastic Reasoning for Large Language Models},\nauthor={Anonymous},\nyear={2026},\nurl={https://openreview.net/forum?id=z9HUMon2Br}\n}"},"title":{"value":"Beyond Fast and Slow: Cognitive-Inspired Elastic Reasoning for Large Language Models"},"pdf":{"value":"/pdf/1252c0b740227fa4d6c59edd0735bd20fd07cbe8.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Desk_Rejected_Submission"},"paperhash":{"value":"hu|beyond_fast_and_slow_cognitiveinspired_elastic_reasoning_for_large_language_models"},"authorids":{"value":["~Jinwu_Hu1","~Dongjin_Yang1","~LangYu_Bian1","~Zhiquan_Wen1","~Yufeng_Wang5","~Yaofo_Chen1","~Bin_Xiao4","~Yuanqing_Li2","~Mingkui_Tan2"]},"authors":{"value":["Jinwu Hu","Dongjin Yang","LangYu Bian","Zhiquan Wen","Yufeng Wang","Yaofo Chen","Bin Xiao","Yuanqing Li","Mingkui Tan"]}},"version":2},{"content":{"comment":{"value":"Dear Reviewer, Thank you so much for your review. We tried to rewrite the paper based on your reviews for correcting the Table formatting and Figures to enhance the readability. We addressed the following concerns as follows – \n\n1. As we know DynaMoE basically uses similarity score to find a specific token’s complexity. It derives token difficulty from similarity-based heuristics without having any access to supervised or semantically grounded labels. As a result token difficulty is indirectly approximated and fixed at deployment time. On the contrary, our proposed router shifts routing decisions based on a threshold value demonstrating a complexity-aware behavior which is not present in prior systems.\n\nNow coming to the Flextron, it uses a surrogate model to approximate downstream loss, but it does not perform token-level complexity-aware routing inside the MLP pathway. Our proposed router, computes per-token complexity scores dynamically thereby choosing lightweight or strong experts depending on the score which allows the routing decision to be adaptive across different tasks and input. Our results also show that, a specific threshold does not outperform for all the benchmarks, but it shifts depending on the task of the benchmark which adds another novelty to the framework. \n\n2. The heuristic definition of token-complexity was shown using the token surprisal as shown in Table. 2 with perplexity score. Hence, we can say the heuristic token-complexity labels were validated through both quantitative and task-level analyses. First, we measured the alignment between our complexity predictions and statistical difficulty measures such as token surprisal and decide how many layers we should fine-tune with the router. Table 2 shows that integrating the complexity-aware router reduces perplexity for both Mistral and LLaMA-2, indicating that the complexity classifier captures stable difficulty patterns when 6-12 transformer layers were fine-tuned. Also, we have Complex-Simple token-distribution patterns in Tables 3 and 6 demonstrating allocation across thresholds and models, supporting the validity of our complexity definition. However, we will work for creating a specific definition for the complexity where we can show a correlation between the perplexity and the results we got after evaluation to validate the complexity definition.\n\n3. The thresholding of the routers was interesting. We wanted to try a wide range of thresholding. However, below and above this threshold range either one of the experts (either light expert or strong expert) was receiving zero tokens for processing. The number of tokens being processed we reported in the paper are basically originated when we were analyzing errors in our results with a random threshold. Only optimal range where both lightweight and strong experts were receiving a number of tokens to process was 0.478-0.5 depending on our architecture. \n\n4. As we mentioned in the Related Works, RouteLLM is not a framework of Mixture of Expert. They simple use a threshold on the surface level to find an optimal cost efficient LLM to route the question. Hence, there is no option or point to compare their results with ours. Same goes for RouterBench. We are inspired to develop a MoE-based architecture from these frameworks but comparing their surface-level MoE architecture with a transformer-level architecture might not make sense. \n\n5. We were expecting a higher inference cost while integrating the router. However, we have somewhat reasonable cost for both inference and memory while using it. For breaking down the cost, we can refer to the Table. 4 and 5, where the base model and proposed model shows the cost difference which indicates the additional cost which comes with the router integration."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1770917692245,"tcdate":1764096096926,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission13468/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission13468/Authors"],"forum":"miyzT0uhhF","number":1,"license":"CC BY 4.0","cdate":1764096096926,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission13468/-/Official_Comment","ICLR.cc/2026/Conference/-/Edit"],"mdate":1770917692245,"domain":"ICLR.cc/2026/Conference","replyto":"WQ9cBds31i","id":"l3Fvbq9XJM","forumContent":{"TLDR":{"value":"Mixture of Expert Architecture for Large Language Models to enhance scaling and performance"},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Mixture of Experts","Large Language Model","Router","Token Complexity Threshold."]},"primary_area":{"value":"generative models"},"abstract":{"value":"Mixture-of-Experts (MoE) architectures have emerged as a powerful technique for improving and scaling Large Language Models by conditionally activating Feed Forward subnetworks and distributing tokens through a routing system,  within the Transformer layers. However, existing MoE methods often rely on static top-k routing strategies that do not involve token-level variability in complexity, leading to suboptimal expert utilization. In this research, we propose a novel token-complexity-based routing framework that dynamically allocates tokens to either lightweight or strong feedforward networks (FFNs) based on their estimated token complexity. Our router is trained using a few-shot classification objective to distinguish between easy and complex tokens and surrogate neural network layer. The efficacy of the framework is evaluated while integrating the router with Mistral-7B and Llama-2-7B model. We evaluate our approach on several benchmarks from various fields, and our proposed MoE framework improves accuracy up to 12% compared to the state-of-the-art results using different MoE architecture, with reasonable computational cost."},"_bibtex":{"value":"@misc{\nrahman2026tokencomplexity,\ntitle={Token-Complexity based Routing Technique within Mixture of Experts Architecture for Large Language Model},\nauthor={Tasnia Rahman and Sathish A.P. Kumar},\nyear={2026},\nurl={https://openreview.net/forum?id=miyzT0uhhF}\n}"},"title":{"value":"Token-Complexity based Routing Technique within Mixture of Experts Architecture for Large Language Model"},"pdf":{"value":"/pdf/8daaf66a8305dcfe6a2ae91cf82adf0862008c1a.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"rahman|tokencomplexity_based_routing_technique_within_mixture_of_experts_architecture_for_large_language_model"},"authorids":{"value":["~Tasnia_Rahman1","~Sathish_A.P._Kumar1"]},"authors":{"value":["Tasnia Rahman","Sathish A.P. Kumar"]}},"version":2},{"content":{"comment":{"value":"We sincerely thank the reviewer for the detailed and thoughtful feedback.  \nBelow we address each concern, clarify the contributions of our work, and provide additional evidence and explanation drawn from the paper.\n\n---\n\n# 1. **“Limited methodological novelty; the paper mainly consolidates existing techniques.”**\n\nWe respectfully disagree and clarify that the novelty of our work does not lie in inventing *new isolated fusion algorithms*, but in **introducing the first unified, serving-compatible, multi-level fusion framework grounded in real multi-LLM log data**. This is a new research direction not studied in prior work.\n\n### **Key points of novelty (clarified):**\n\n### **(A) A new *multi-stage fusion perspective* grounded in serving constraints**  \nPrior work studies routing *alone*, thought retrieval *alone*, or distillation *alone*.  \nOur contribution is the **first systematic formulation** of LLM capability fusion across:\n\n1. **Query-level (early fusion)** – routing using *reward-based* evaluation combining performance, cost, and LLM-judge scores (Eq.1).  \n2. **Thought-level (mid fusion)** – *retrieval of distilled thought templates*, which is not standard few-shot prompting.  \n3. **Model-level (late fusion)** – multi-response distillation with hybrid selection.\n\nThese three fusion stages are **explicitly positioned as a unified design space for fusion**, enabling rigorous cross-level comparisons (Table 6). This perspective does not exist in prior work like RouteLLM, GraphRouter, or Thought Propagation.\n\n### **(B) New insight: fusion behaviors *strongly differ* across levels**  \nOur benchmark reveals previously unknown findings:\n\n- **Thought-level fusion** provides the strongest gains (+12.7% for small models, +4.8% for medium, +8.3% for large; Fig.6).  \n- **Model-level fusion** underperforms despite access to “best” responses (Table 5).  \n- **World Knowledge and Math** tasks exhibit **negative fusion transfer**, contradicting the assumption that “fusion is always better.”\n\nThese patterns were **not discovered or compared in any prior work**, because prior works evaluated only isolated components.\n\n### **(C) A new reward formulation enabling *serving-compatible fusion***  \nUnlike prior routing studies that rely on logits or internal model states, FusionFactory focuses on **API-compatible** signals—performance, cost, and LLM-judge scores—allowing fusion across 20 diverse open-source and closed-source models.\n\nThis design explicitly targets *real-world serving scenarios*, which existing fusion papers do not support.\n\n---\n\n# 2. **“How is LLMFusionBench better than previous datasets used for distillation or fusion?”**\n\nWe appreciate this question and clarify the **distinct strengths of LLMFusionBench**.\n\n### **(A) Multi-LLM log data with both *direct* and *reasoning* responses**  \nLLMFusionBench contains for each query:\n\n- direct answers  \n- reasoning (CoT) answers  \n- performance metrics  \n- cost  \n- LLM-judge scores  \n- distilled thought templates\n\nThis combination **does not exist** in prior datasets like FLAN or DistilGPT datasets, which include only task labels and single responses.\n\n### **(B) First dataset explicitly designed for *three-stage fusion***  \nPrevious datasets enable only:\n\n- single-model distillation (e.g., Alpaca, Self-Instruct)  \n- routing-focused logs without thought templates (RouterBench)  \n- CoT datasets without multi-model variation (CoT reasoning corpora)\n\nLLMFusionBench is the first to support *all three fusion stages*—query, thought, model—allowing direct comparative research.\n\n### **(C) Scale and diversity unmatched by prior multi-LLM datasets**  \n- **20 LLMs** (8B–671B)  \n- **14 tasks** across **6 domains**  \n- **103M+ tokens**  \n- **Both open-source and closed-source logs**  \n\nThis scale allows studying **cross-model complementarity**, which smaller distillation datasets cannot support.\n\n### **(D) Reusable thought templates**  \nOur dataset includes **summaries distilled from the top-k responses**, an artifact explicitly designed for thought-level fusion, not available in any prior work."},"title":{"value":"Response to Reviewer 918D (1/2)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764784377746,"tcdate":1764784377746,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission20262/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission20262/Authors"],"forum":"B7otvE3ExU","number":1,"license":"CC BY 4.0","cdate":1764784377746,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission20262/-/Official_Comment"],"mdate":1764784377746,"domain":"ICLR.cc/2026/Conference","replyto":"0hXsjmE4iP","id":"ZU6hCT62PY","forumContent":{"TLDR":{"value":"We introduce LLMFusionBench, a large-scale routing benchmark, and FusionFactory, a multi-level LLM fusion framework that leverages routing data and thought templates to outperform the best individual LLMs across diverse tasks."},"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["LLM fusion","Query-level fusion","Thought-level fusion","Model-level fusion"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"The rapid advancement of large language models (LLMs) has created a diverse\nlandscape of models, each excelling at different tasks. This diversity drives re-\nsearchers to employ multiple LLMs in practice, leaving behind valuable multi-\nLLM log data. This naturally leads to the question of whether such logs can be\nfully leveraged to fuse LLMs’ complementary capabilities. Although prior work\nhas explored various strategies for integrating multiple LLMs, we argue that prac-\ntical fusion must meet two essential requirements: (1) compatibility with real-\nworld serving scenarios (e.g., local and API-based serving), and (2) flexibility to\noperate at different stages of the LLM pipeline to meet varied user needs (e.g.,\nfine-tuning and inference stages). To this end, we introduce LLMFusionBench,\na large-scale benchmark for LLM fusion that spans 14 tasks across five domains,\nwith responses from 20 open-source LLMs (8B–671B) totaling 103M tokens.\nBuilding on LLMFusionBench, we propose FusionFactory, a systematic\nframework with three elaborated levels: (1) query-level fusion via tailored LLM\nrouters, (2) thought-level fusion leveraging retrieved abstract reasoning tem-\nplates, and (3) model-level fusion via distillation from top-ranked responses. Ex-\nperiments show that FusionFactory consistently outperforms the best individ-\nual LLM across all 14 benchmarks, with the optimal fusion configuration varying\nacross benchmarks, highlighting the promise of multi-LLM log data as a practical\nfoundation for fusing diverse LLM capabilities."},"_bibtex":{"value":"@misc{\nfeng2026fusionfactory,\ntitle={FusionFactory: Fusing {LLM} Capabilities with Multi-{LLM} Log Data},\nauthor={Tao Feng and Haozhen Zhang and Zijie Lei and Pengrui Han and Mostofa Patwary and Mohammad Shoeybi and Bryan Catanzaro and Jiaxuan You},\nyear={2026},\nurl={https://openreview.net/forum?id=B7otvE3ExU}\n}"},"title":{"value":"FusionFactory: Fusing LLM Capabilities with Multi-LLM Log Data"},"pdf":{"value":"/pdf/cad5c9cf3d97feb62ebb1659f0a3c9438dfab0da.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"feng|fusionfactory_fusing_llm_capabilities_with_multillm_log_data"},"authorids":{"value":["~Tao_Feng5","~Haozhen_Zhang1","~Zijie_Lei1","~Pengrui_Han1","~Mostofa_Patwary1","~Mohammad_Shoeybi1","~Bryan_Catanzaro1","~Jiaxuan_You2"]},"authors":{"value":["Tao Feng","Haozhen Zhang","Zijie Lei","Pengrui Han","Mostofa Patwary","Mohammad Shoeybi","Bryan Catanzaro","Jiaxuan You"]}},"version":2},{"content":{"comment":{"value":"## **W1: The core utility metric (Utility = Performance - α * Cost) is flawed. It completely ignores the significant cumulative latency incurred by the cascade structure. A query rejected multiple times will have a real-world response time far exceeding that of a single-shot centralized router, a critical cost factor this paper overlooks.**\n\nWe appreciate the reviewer’s rigorous scrutiny regarding the formulation of our Utility metric. We wish to clarify that the omission of routing overhead in the original formula was a **deliberate simplification for mathematical conciseness**, predicated on our analysis that this term is **statistically negligible** compared to inference costs, as we’ve declared in **Section 2.1** and discussed in **Appendix A.1**.\n\nWhile the omission was a statistically justified simplification, we agree with the reviewer that explicitly formalizing this term enhances theoretical rigor. We will **revise the formal definition** of the cost function in the later version of manuscript to explicitly include the routing overhead term, ensuring the methodology is mathematically flawless.\n\nTo validate this design choice and further address your concern, we provide a detailed breakdown proving that the \"cumulative latency\" does not materially affect the cost-benefit landscape.\n\n### **1. Empirical Analysis of Routing Overhead**\n\nAs detailed in **Appendix A.1**, we conducted a comprehensive statistical analysis of inference and routing costs.\n\n- **Marginal Cost Ratio:** Our empirical data confirms that the \"routing cost\" (time spent on rejections) is minimal compared to the \"inference cost\" (time spent generating the solution) (**Table 7**). Even in the worst-case path (traversing the entire chain to the final agent), the cumulative time spent on rejections constitutes **less than 5%** of the time required for the final answer generation. This is structurally inherent: a rejection is a short sequence (e.g., \"I don't know!\"), whereas a valid answer is typically a long CoT sequence.\n- **Latency Analysis:** We further benchmarked System Latency (Time to First Token) in **Table R1**. While DiSRouter is slightly slower than lightweight predictive routers (e.g., RouteLLM), it is **significantly faster** than generative verification baselines (e.g., FrugalGPT, AutoMix), confirming its practical efficiency.\n    \n    *Table R1: Comparison of Time to First Token (s) between routing systems.*\n    \n    |  | RouterLLM | FrugalGPT | Automix | FORC | GraphRouter | DiSRouter |\n    | --- | --- | --- | --- | --- | --- | --- |\n    | Average | 0.048 | 2.620 | 3.475 | 0.070 | 0.024 | 0.081 |\n    | Worst Case | 0.063 | 9.565 | 11.34 | 0.076 | 0.028 | 0.241 |\n\n### **2. Sensitivity Analysis: Impact on System Utility**\n\nTo directly address the reviewer’s concern, we re-evaluated our system using an Adjusted Utility Metric that explicitly subtracts the routing overhead cost:\n\n$$Utility_{adjusted} = Performance - \\alpha \\cdot (Cost_{inference} + Cost_{routing})$$\n\nThe comparison between the original and adjusted metrics is presented in **Table R2**.\n\nTable R2: Sensitivity Analysis of System Utility: A comparison between original metrics (inference cost only) and adjusted metrics (incorporating cumulative routing overhead, **marked with a \\***).\n\n| Scenario | Alpha | Accuracy | Cost | Cost* | Utility | Utility* |\n| --- | --- | --- | --- | --- | --- | --- |\n| Performance First | 0.2 | 0.81 | 0.396 | 0.413 | 0.731 | 0.727 |\n| Balance | 0.5 | 0.77 | 0.321 | 0.333 | 0.610 | 0.604 |\n| Cost First | 0.8 | 0.75 | 0.289 | 0.299 | 0.519 | 0.511 |\n- **Minimal Deviation:** The results demonstrate that including the routing cost has a minimal impact on the overall System Utility. Even in the sensitive \"Cost First\" scenario, the overall utility drops by only approximately **1.5%**.\n- **Conclusion:** This confirms that the cumulative latency, while physically present, does not significantly alter the cost-benefit landscape or invalidate the comparative advantages of DiSRouter."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764224779917,"tcdate":1764155954249,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7635/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission7635/Authors"],"forum":"KDcwXKr0NU","number":8,"license":"CC BY 4.0","cdate":1764155954249,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7635/-/Official_Comment"],"mdate":1764224779917,"domain":"ICLR.cc/2026/Conference","replyto":"6yw99h60N0","id":"lcrSfTaBex","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"TLDR":{"value":"We introduce DiSRouter, a distributed system where LLMs leverage \"self-awareness\" to route queries among themselves, outperforming conventional centralized routers."},"keywords":{"value":["query routing","model selection","distributed system","self-awareness of LLM"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"The proliferation of Large Language Models (LLMs) has created a diverse ecosystem of models with highly varying performance and costs, necessitating effective query routing to balance performance and expense. Current routing systems often rely on a centralized external router trained on a fixed set of LLMs, making them inflexible and prone to poor performance since the small router can not fully understand the knowledge boundaries of different LLMs. We introduce DiSRouter (Distributed Self-Router), a novel paradigm that shifts from centralized control to distributed routing. In DiSRouter, a query traverses a network of LLM agents, each independently deciding whether to answer or route to other agents based on its own self-awareness—its ability to judge its competence. This distributed design offers superior flexibility, scalability, and generalizability. To enable this, we propose a two-stage Self-Awareness Training pipeline that enhances each LLM's self-awareness. Extensive experiments demonstrate that DiSRouter significantly outperforms existing routing methods in utility across various scenarios, effectively distinguishes between easy and hard queries, and shows strong generalization to out-of-domain tasks. Our work validates that leveraging an LLM's intrinsic self-awareness is more effective than external assessment, paving the way for more modular and efficient multi-agent systems."},"_bibtex":{"value":"@inproceedings{\nzheng2026disrouter,\ntitle={Di{SR}outer: Distributed Self-Routing for {LLM} Selections},\nauthor={Hang Zheng and Hongshen Xu and Yongkai.lin and Shuai Fan and Lu Chen and Kai Yu},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=KDcwXKr0NU}\n}"},"title":{"value":"DiSRouter: Distributed Self-Routing for LLM Selections"},"pdf":{"value":"/pdf/0190f98d94890b700d03febdee678e22a2a5e380.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"zheng|disrouter_distributed_selfrouting_for_llm_selections"},"authorids":{"value":["~Hang_Zheng3","~Hongshen_Xu1","~Yongkai.lin1","~Shuai_Fan1","~Lu_Chen3","~Kai_Yu3"]},"authors":{"value":["Hang Zheng","Hongshen Xu","Yongkai.lin","Shuai Fan","Lu Chen","Kai Yu"]}},"version":2},{"content":{"comment":{"value":"## **Q2: Can DiSRouter's self-awareness training be applied to LLMs with different architectures (e.g., encoder-decoder models)? Please provide results with at least one non-Qwen model family to validate broad applicability.**\n\nAs detailed in our Response to `W1`, our method has proven effective across distinct model families (Gemma3, Phi4). We prioritized decoder-only architectures in our experiments simply because they currently dominate the high-performance LLM landscape, leaving few competitive encoder-decoder options for complex reasoning.\n\nCrucially, our Self-Awareness Training is a **data-driven alignment strategy** (SFT + RL) rather than an architecture-specific modification. Consequently, the methodology is architecture-agnostic and should be applicable to any model architecture—including encoder-decoder frameworks—provided the model possesses sufficient capacity to learn self-reflection from data.\n\n## **Q3: How does DiSRouter perform with large-scale agent pools (e.g., >10 agents)?**\n\nIn our current experimental setup, we utilized a pool of **general-purpose LLMs** (`Qwen2.5-Instruct` series). Since these models are homogeneous and possess broad capabilities across various domains, a concise cascade of 5 agents proved sufficient to effectively cover the difficulty spectrum of the test tasks without requiring a large-scale pool (>10 agents).\n\nHowever, we are actively researching **heterogeneous routing networks** for future work. In such scenarios, the pool would contain diverse, domain-specific expert models (e.g., **specialized math solvers or coding experts**). This setting would naturally benefit from a larger scale DiSRouter network, likely organized in more complex topologies such as **tree structures** rather than a simple linear cascade. As this paper serves as the foundational work validating the efficacy of the distributed self-routing paradigm, we have focused on the cascade structure to establish the core concept. We intend to explore large-scale, heterogeneous agent pools in subsequent research.\n\n## **Q4: What are the practical limitations (e.g., latency, fault tolerance) in deploying DiSRouter? Case studies or simulations with real-world constraints would be valuable.**\n\nWe appreciate this practical perspective, which was echoed by other reviewers. To address concerns regarding latency and overhead, we have added a detailed statistical analysis of actual inference and routing costs in **Appendix A.1** of the revised paper.\n\n### **Latency Analysis**\n\n1. **Negligible Routing Overhead:** Our empirical data (**Table 6**) shows that although a query may be rejected by multiple agents before being solved, the cumulative **routing cost** (time spent generating rejections) remains negligible. Even in the worst-case scenario (routing to the final agent), this overhead constitutes less than **5%** of the actual inference cost of generating the final answer (**Table 7**). \n2. **Competitive Latency:** We also benchmarked **System Latency** (Time to First Token, **Table R6 and Table 8 in paper**). Results indicate that while DiSRouter incurs slightly higher latency than predictive routers (e.g., RouteLLM) which use lightweight classifiers, it is significantly faster than generative baselines that rely on full model outputs or verification steps (e.g., FrugalGPT, AutoMix).\n\n    *Table R6: Time to First Token (s) Results Comparison for Different Routing Systems.*\n\n    |  | RouterLLM | FrugalGPT | Automix | FORC | GraphRouter | DiSRouter |\n    | --- | --- | --- | --- | --- | --- | --- |\n    | Average | 0.048 | 2.620 | 3.475 | 0.070 | 0.024 | 0.081 |\n    | Worst Case | 0.063 | 9.565 | 11.34 | 0.076 | 0.028 | 0.241 |\n\n3. **Deployment Constraints:** We acknowledge that our current evaluation assumes efficient inter-agent communication. In **physically distributed deployments**, **network latency** becomes a non-negligible factor. Real-world implementation would require incorporating network transmission overhead into the global cost function to optimize for total system latency."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764154327090,"tcdate":1764154327090,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7635/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission7635/Authors"],"forum":"KDcwXKr0NU","number":4,"license":"CC BY 4.0","cdate":1764154327090,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7635/-/Official_Comment"],"mdate":1764154327090,"domain":"ICLR.cc/2026/Conference","replyto":"hD3voPCEXO","id":"VfNBXqGk56","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"TLDR":{"value":"We introduce DiSRouter, a distributed system where LLMs leverage \"self-awareness\" to route queries among themselves, outperforming conventional centralized routers."},"keywords":{"value":["query routing","model selection","distributed system","self-awareness of LLM"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"The proliferation of Large Language Models (LLMs) has created a diverse ecosystem of models with highly varying performance and costs, necessitating effective query routing to balance performance and expense. Current routing systems often rely on a centralized external router trained on a fixed set of LLMs, making them inflexible and prone to poor performance since the small router can not fully understand the knowledge boundaries of different LLMs. We introduce DiSRouter (Distributed Self-Router), a novel paradigm that shifts from centralized control to distributed routing. In DiSRouter, a query traverses a network of LLM agents, each independently deciding whether to answer or route to other agents based on its own self-awareness—its ability to judge its competence. This distributed design offers superior flexibility, scalability, and generalizability. To enable this, we propose a two-stage Self-Awareness Training pipeline that enhances each LLM's self-awareness. Extensive experiments demonstrate that DiSRouter significantly outperforms existing routing methods in utility across various scenarios, effectively distinguishes between easy and hard queries, and shows strong generalization to out-of-domain tasks. Our work validates that leveraging an LLM's intrinsic self-awareness is more effective than external assessment, paving the way for more modular and efficient multi-agent systems."},"_bibtex":{"value":"@inproceedings{\nzheng2026disrouter,\ntitle={Di{SR}outer: Distributed Self-Routing for {LLM} Selections},\nauthor={Hang Zheng and Hongshen Xu and Yongkai.lin and Shuai Fan and Lu Chen and Kai Yu},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=KDcwXKr0NU}\n}"},"title":{"value":"DiSRouter: Distributed Self-Routing for LLM Selections"},"pdf":{"value":"/pdf/0190f98d94890b700d03febdee678e22a2a5e380.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"zheng|disrouter_distributed_selfrouting_for_llm_selections"},"authorids":{"value":["~Hang_Zheng3","~Hongshen_Xu1","~Yongkai.lin1","~Shuai_Fan1","~Lu_Chen3","~Kai_Yu3"]},"authors":{"value":["Hang Zheng","Hongshen Xu","Yongkai.lin","Shuai Fan","Lu Chen","Kai Yu"]}},"version":2},{"content":{"comment":{"value":"## **Response to Question 2: Model Performance vs. $\\lambda$**\n\n**Question:** \"How is the model's selection with different values of lambda? I think the author can try to plot the model performance under various values of lambda.\"\n\nThank you for this suggestion. We need to clarify how different routing formulations handle $\\lambda$.\n\n**Two Routing Formulations.** Our paper evaluates two types of routing methods:\n\n- **Utility prediction methods** (our main results, Table 2): These methods predict both performance $\\hat{s}(x,m)$ and cost $\\hat{c}(x,m)$ for each model. At inference time, we can compute utility $\\hat{u}(x,m) = \\hat{s}(x,m) - \\lambda \\cdot \\hat{c}(x,m)$ for \\emph{any} value of $\\lambda$ and select the model with highest predicted utility. This allows us to vary $\\lambda$ continuously and trace the full cost-performance Pareto frontier, which is exactly what you suggest.\n\n- **Selection-based methods** (Appendix D): These methods train a classifier to directly predict which model to select. Crucially, the training process fixes a specific $\\lambda$ value---the ground truth label for each query is the model that maximizes $s(x,m) - \\lambda \\cdot c(x,m)$ for that fixed $\\lambda$. Once trained, the classifier cannot be evaluated at different $\\lambda$ values because it has learned to optimize for a specific cost-performance preference.\n\n**Our Evaluation Approach.** For utility prediction methods, we do exactly what you suggest: we vary $\\lambda$ continuously and plot the resulting cost-performance curves. The AUC metric (Table 2) measures the area under these curves, capturing overall routing quality across all $\\lambda$ values.\n\nFor selection-based methods, we cannot vary $\\lambda$ post-training. Instead, we train separate classifiers for three different $\\lambda$ values representing different preferences: high-performance ($\\lambda = 0.1/c_{\\max}$), balanced ($\\lambda = 0.5/c_{\\max}$), and low-cost ($\\lambda = 1.0/c_{\\max}$). This follows the evaluation protocol from prior work (GraphRouter, TensorOpera).\n\n**Why We Emphasize Utility Prediction.** This limitation of selection-based methods is one reason we advocate for utility prediction as the primary evaluation approach. Utility prediction methods provide flexibility to adjust $\\lambda$ at deployment time based on changing requirements, while selection-based methods are locked into their training preference.\n\n\n## **Response to Related Work Summary**\n\n**Reviewer's Concern:** \"It is better to have a paragraph summarizing the existing work on training state-of-the-art routers such as linear regression, MLP, attention-based NN, etc.\"\n\nThank you for this suggestion. We will expand Section 2 to include a more comprehensive summary of routing architectures beyond the methods listed in Table 1. We will organize this by approach:\n\n- Linear methods: RouteLLM [Ong et al., 2024] uses matrix factorization\n- Neural methods: Various works use MLPs, attention mechanisms, and graph neural networks\n- Non-parametric methods: Several works have explored kNN-based routing\n- Hybrid approaches: Combinations of learned and retrieval-based methods\n\nWe will also add the recent works you mentioned (CARROT, arxiv 2509.09782) to provide a more complete picture of the field.\n\n-----\n\nWe appreciate your thoughtful feedback and hope these clarifications address your concerns. The surprising effectiveness of simple methods is not due to data limitations, but rather reflects the fundamental structure of the routing problem---low intrinsic dimensionality and strong locality favor local methods over complex global models. Thank you for helping us articulate this more clearly."},"title":{"value":"Continue"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763874803795,"tcdate":1763874803795,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission21199/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission21199/Authors"],"forum":"Chn50flK4X","number":11,"license":"CC BY 4.0","cdate":1763874803795,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission21199/-/Official_Comment"],"mdate":1763874803795,"domain":"ICLR.cc/2026/Conference","replyto":"HlJMX4RWDH","id":"JOxsTy6glp","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["LLM Routing"]},"supplementary_material":{"value":"/attachment/adf43ee825c55910aeb5b0c4e987a2dcba526805.zip"},"primary_area":{"value":"applications to computer vision, audio, language, and other modalities"},"abstract":{"value":"As large language models (LLMs) grow in scale and specialization, routing—selecting the best model for a given input—has become essential for efficient and effective deployment. While recent methods rely on increasingly complex learned routing strategies, their dependence on disparate training data and evaluation setups makes comparison and generalization difficult. In this work, we fundamentally rethink LLM routing by questioning whether such complexity is necessary. We show that a well-tuned k-Nearest Neighbors (kNN) approach not only matches but often outperforms state-of-the-art learned routers while being significantly more efficient. To support systematic evaluation, we introduce a suite of standardized routing benchmarks spanning instruction-following, question-answering, and reasoning tasks, as well as the first multi-modal routing dataset involving visual inputs. Our theoretical analysis reveals that the strong locality properties of model performance in embedding space enable simple non-parametric methods to achieve superior routing decisions with lower sample complexity than parametric approaches. These findings challenge the prevailing trend toward sophisticated architectures and demonstrate that simple, interpretable approaches can be surprisingly effective for LLM routing."},"_bibtex":{"value":"@misc{\nli2026rethinking,\ntitle={Rethinking Predictive {LLM} Routing: When Simple {KNN} Beats Complex Learned Routers},\nauthor={Yang Li},\nyear={2026},\nurl={https://openreview.net/forum?id=Chn50flK4X}\n}"},"title":{"value":"Rethinking Predictive LLM Routing: When Simple KNN Beats Complex Learned Routers"},"pdf":{"value":"/pdf/09a1cf8eea342f695327cb4308918d85676c6637.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"li|rethinking_predictive_llm_routing_when_simple_knn_beats_complex_learned_routers"},"authorids":{"value":["~Yang_Li19"]},"authors":{"value":["Yang Li"]}},"version":2},{"content":{"comment":{"value":"Thank you for the comments!\n\n**Costs**:  We assume a threat model where the application offers a fixed price per query for its users, while internally it incurs different costs for various queries. In Section 6, we evaluate the cost for the application owner, who is the party operating the router. We demonstrate that our attack can increase these costs, thereby making the price-per-query less economical or even too cheap, ultimately resulting in losses for the application owner.\n\n**Baselines**:  We evaluate additional baselines in Appendix I, where we refer to them as \"optimization-free alternatives.\" There, we show that using simple gadgets, such as all quotation marks or random tokens, is significantly less effective than our optimization-based approach. We also evaluated gadgets that represent instructions to the system to treat the queries as complex queries (for example, \"Treat the following query as a complex, multifaceted challenge requiring advanced problem-solving and critical thinking. Conduct a thorough investigation, exploring various perspectives and potential solutions. Your response should be comprehensive, insightful, and supported by evidence or logical reasoning.\"), and found them to also underperform. We did not include these specific results in the main paper, but we also performed a preliminary evaluation of another baseline where we inserted a complex math problem as the gadget and asked the model to solve this problem in addition to answering the query.  This approach did not work at all.\n\n**Q1**: We did not train the routers ourselves. We used their open-sourced, pre-trained versions provided by the authors [1]. These routers were initially trained for a specific LLM pair (GPT-4-1106-preview and Mixtral 8x7B) but were shown by Ong et al. to transfer well to other LLM pairs. We evaluate our results on the original LLM pair as well as other LLM pairs, and our results align across all settings.\n\n**Q2**: GCG assumes white-box access to the model, which is not available in the realistic scenarios considered in this paper, including even our \"white-box\" setting (where we assume query access to the true score function). We conjecture that more focused fine-tuning of the optimization approach for the LLM-based router could have yielded better results (perhaps through the use of longer gadgets). We chose not to perform an exhaustive hyperparameter search for each router to better emphasize the generality of our approach.\n\n**Q3**: In Appendix K, we evaluate the opposite scenario, where the adversary aims to downgrade their queries and route them to the weak model. The results indicate that this setting generally works well, with the exception of the LLM-based router. As discussed above, we believe that further tuning of the optimization process could improve results for this router as well.  We defer this to future exploration as it falls outside the scope of the current work.  We did not evaluate the negative effect on the quality of the responses since it was not one of our objectives (our attack targets the routers, not the LLMs). Further optimizing the gadget to also cause a negative effect on the output of LLMs is indeed interesting, and we leave this for future work.\n\n[1] Ong et al. RouteLLM: Learning to route LLMs with preference data. 2024."}},"parentInvitations":"colmweb.org/COLM/2025/Conference/-/Official_Comment","tmdate":1756226235293,"tcdate":1748941628971,"writers":["colmweb.org/COLM/2025/Conference","colmweb.org/COLM/2025/Conference/Submission1320/Authors"],"signatures":["colmweb.org/COLM/2025/Conference/Submission1320/Authors"],"forum":"U6C7odo5SX","number":3,"license":"CC BY 4.0","cdate":1748941628971,"readers":["everyone"],"invitations":["colmweb.org/COLM/2025/Conference/Submission1320/-/Official_Comment","colmweb.org/COLM/2025/Conference/-/Edit"],"mdate":1756226235293,"domain":"colmweb.org/COLM/2025/Conference","replyto":"jaLqgl4z3T","id":"fPJKDPVeDx","forumContent":{"TLDR":{"value":"Proposing a novel class of vulnerabilities, where adversaries can manipulate LLM routing decisions to their advantage."},"venue":{"value":"COLM 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the COLM Code of Ethics on https://colmweb.org/CoE.html"},"keywords":{"value":["LLMs","Routers","Adversarial Machine Learning","ML Security"]},"abstract":{"value":"LLM routers balance quality and cost of responding to queries by routing them to a cheaper or more expensive LLM depending on the query's estimated complexity.  Routers are a type of what we call ``LLM control planes,'' i.e., systems that orchestrate multiple LLMs.\n\nIn this paper, we investigate adversarial robustness of LLM control planes using routers as a concrete example.  We formulate LLM control-plane integrity as a distinct problem in AI safety, where the adversary's goal is to control the order or selection of LLMs employed to process users' queries.  We then demonstrate that it is possible to generate query-independent ``gadget'' strings that, when added to any query, cause routers to send this query to a strong LLM.  In contrast to conventional adversarial inputs, gadgets change the control flow but preserve or even improve the quality of outputs generated in response to adversarially modified queries.\n\nWe show that this attack is successful both in white-box and black-box settings against several open-source and commercial routers.  We also show that perplexity-based defenses fail, and investigate alternatives."},"_bibtex":{"value":"@inproceedings{\nshafran2025rerouting,\ntitle={Rerouting {LLM} Routers},\nauthor={Avital Shafran and Roei Schuster and Tom Ristenpart and Vitaly Shmatikov},\nbooktitle={Second Conference on Language Modeling},\nyear={2025},\nurl={https://openreview.net/forum?id=U6C7odo5SX}\n}"},"title":{"value":"Rerouting LLM Routers"},"pdf":{"value":"/pdf/90196a560af282e8b63901f276e00ea0dd18c970.pdf"},"venueid":{"value":"colmweb.org/COLM/2025/Conference"},"paperhash":{"value":"shafran|rerouting_llm_routers"},"authorids":{"value":["~Avital_Shafran1","~Roei_Schuster1","~Tom_Ristenpart1","~Vitaly_Shmatikov1"]},"author_guide":{"value":"I certify that this submission complies with the submission instructions as described on https://colmweb.org/AuthorGuide.html"},"authors":{"value":["Avital Shafran","Roei Schuster","Tom Ristenpart","Vitaly Shmatikov"]}},"version":2},{"content":{"comment":{"value":"**Q5. Interest in this paper will be limited because it mainly uses older models and the fusion techniques it uses are not very novel.**\n\n**Response:**\n\n**(a) Model recency.** Our 20-model pool predates the most recent frontier models (GPT-5, Claude 4, Gemini 2) but includes the strongest open-source models available at the time of data collection (DeepSeek-R1 671B, Llama-3.3 Nemotron Super 49B, Llama-3.1 Nemotron 51B, Qwen2.5 7B). The framework is model-agnostic by construction: extending the LLM pool requires only re-running response collection on the existing query set, with no architectural change to the fusion pipeline. The revised Conclusion (§8) adds a sentence to its closing paragraph stating that *\"FusionFactory is model-agnostic by construction: extending the LLM pool to newer models (e.g., GPT-5, Claude 4, Gemini 2) requires only re-running response collection on the existing query set, with no architectural change to the fusion pipeline.\"* The model pool affects the absolute numbers reported in Tables 4–6 but not the protocol-level findings, which concern the relative behavior of fusion stages.\n\n**(b) Novelty of contribution.** The paper does not propose a new fusion algorithm and does not claim algorithmic novelty. The contributions are at the framework and benchmark level:\n\n1. **A unified framework spanning all three fusion stages with API-serving compatibility and six-domain coverage.** Per Table 1, prior work (RouteLLM, GraphRouter, RouterBench, LLM-Blender, FuseLLM) supports only one fusion stage; FuseLLM additionally requires model logits and is therefore incompatible with API-served models. FusionFactory is the only framework in Table 1 that covers all three fusion stages, supports API-served models, and evaluates across the broadest set of task domains (6 vs 3–5 for prior work).\n2. **LLMFusionBench, a released and reusable benchmark** — 14 tasks across 6 domains, 20 LLMs, approximately 103M tokens of response data with both direct and reasoning-augmented modes, per-response performance and cost annotations (Table 9 schema), and LLM-judge scores (definition and prompt in Appendix I / Table 16; comparison against task-metric scoring in Table 17). To our knowledge, no comparable multi-LLM log dataset is publicly available; the cost of reconstructing such a dataset is a primary barrier to fusion research at present.\n\nThese contributions enable the cross-stage comparison in §7, which the reviewer identifies as a useful part of the paper.\n\n---\n\n**Q6. It looks like thoughts were collected from all of the models studied using a CoT prompt. How many of the models were using a trained think token approach - e.g., where they were fine-tuned to produce thoughts between think tokens like <think> ... </think> rather than just being prompted to do so?**\n\n**Response:**\n\nPer §2.1, all `-think` outputs were collected by applying the same chain-of-thought elicitation procedure across all 20 models. This was a deliberate design choice to enable apples-to-apples comparison across the pool: the 20-model pool contains both LLMs with native trained think-token behavior (e.g., DeepSeek-R1 and the Nemotron *-think* variants) and LLMs without it, and uniform prompting avoids confounding fusion-stage comparisons with differences in reasoning-elicitation behavior. For models with trained think-token behavior, the model's native reasoning style is preserved when responding to our prompt; we do not override or constrain it. For models without trained think-token behavior, the `-think` responses are produced via prompted CoT.\n\nWe agree with the reviewer that the distinction between trained-think-token and prompted-CoT origins is methodologically informative. The revised §2.1 adds a sentence to the paragraph that introduces the `-think` response collection, explicitly stating that the 20-model pool contains both LLMs with native trained think-token behavior and LLMs without it, that `-think` responses are produced via prompted CoT for the latter while preserving native reasoning style for the former, and that uniform prompting was chosen to avoid confounding fusion-stage comparisons with reasoning-elicitation differences."},"title":{"value":"Response (5)"}},"parentInvitations":"TMLR/-/Official_Comment","tmdate":1779261132914,"tcdate":1779261132914,"writers":["TMLR","TMLR/Paper7564/Authors"],"signatures":["TMLR/Paper7564/Authors"],"forum":"N951scS3yE","number":13,"license":"CC BY 4.0","cdate":1779261132914,"readers":["everyone"],"invitations":["TMLR/Paper7564/-/Official_Comment"],"mdate":1779261132914,"domain":"TMLR","replyto":"iFST72wffo","id":"X7xOoaLbRr","forumContent":{"submission_length":{"value":"Regular submission (no more than 12 pages of main content)"},"venue":{"value":"Accepted by TMLR"},"code":{"value":"https://github.com/ulab-uiuc/FusionFactory"},"abstract":{"value":"The rapid advancement of large language models (LLMs) has created a diverse landscape of models, each excelling at different tasks. This diversity drives researchers to employ multiple LLMs in practice, leaving behind valuable multi-LLM log data. This naturally leads to the question of whether such logs can be fully leveraged to fuse LLMs' complementary capabilities. Although prior work has explored various strategies for integrating multiple LLMs, we argue that practical fusion must meet two essential requirements: (1) compatibility with real-world serving scenarios (e.g., local and API-based serving), and (2) flexibility to operate at different stages of the LLM pipeline to meet varied user needs (e.g., fine-tuning and inference stages). To this end, we introduce LLMFusionBench, a large-scale benchmark for LLM fusion that spans 14 tasks across six domains, with responses from 20 open-source LLMs (8B-671B) totaling 103M tokens. Building on LLMFusionBench, we propose FusionFactory, a systematic framework with three elaborated levels: (1) query-level fusion via tailored LLM routers, (2) thought-level fusion leveraging retrieved abstract reasoning templates, and (3) model-level fusion via distillation from top-ranked responses. Experiments show that FusionFactory consistently outperforms the best individual LLM across all 14 benchmarks, with the optimal fusion configuration varying across benchmarks, highlighting the promise of multi-LLM log data as a practical foundation for fusing diverse LLM capabilities."},"_bibtex":{"value":"@article{\nfeng2026fusionfactory,\ntitle={FusionFactory: Fusing {LLM} Capabilities with Multi-{LLM} Log Data},\nauthor={Tao Feng and Haozhen Zhang and Zijie Lei and Pengrui Han and Mostofa Patwary and Mohammad Shoeybi and Bryan Catanzaro and Jiaxuan You},\njournal={Transactions on Machine Learning Research},\nissn={2835-8856},\nyear={2026},\nurl={https://openreview.net/forum?id=N951scS3yE},\nnote={}\n}"},"title":{"value":"FusionFactory: Fusing LLM Capabilities with Multi-LLM Log Data"},"pdf":{"value":"/pdf/96397604f23ee7e8f7f9ff9fb9b73ae6622ffdad.pdf"},"venueid":{"value":"TMLR"},"paperhash":{"value":"feng|fusionfactory_fusing_llm_capabilities_with_multillm_log_data"},"authorids":{"value":["~Tao_Feng5","~Haozhen_Zhang1","~Zijie_Lei1","~Pengrui_Han1","~Mostofa_Patwary1","~Mohammad_Shoeybi1","~Bryan_Catanzaro1","~Jiaxuan_You2"]},"assigned_action_editor":{"value":"~Jaeho_Lee3"},"authors":{"value":["Tao Feng","Haozhen Zhang","Zijie Lei","Pengrui Han","Mostofa Patwary","Mohammad Shoeybi","Bryan Catanzaro","Jiaxuan You"]}},"version":2},{"content":{"comment":{"value":"> The router is evaluated in a batch setting, so that it assumes all tasks may be known upfront. In real-world settings, the router makes per-task decisions without knowing the full distribution. Also, the binary \"easy/hard\" classification with constrained decoding is simplistic; a regression-based difficulty score might perform better, but no alternatives are explored or discussed.\n\n* We thank the reviewer for the suggestion.\n* **On regression-based difficulty scores:** As stated in Section 7.2, the router already outputs a continuous numerical difficulty score between 0 (easiest) and 1 (hardest) for each task \\- this is precisely the regression-based scoring the reviewer suggests.   \n* **On batch vs. per-task routing:** We agree that real-world deployment requires per-task routing decisions. We would like to clarify that our setup already supports this: since scores are numerical, an appropriate threshold $\\\\tau$ can be chosen such that any task scoring above $\\\\tau$ is routed to the large model.   \n  * Crucially, this per-task threshold setting and our batch setting are equivalent \\- they route the same tasks and achieve the same RCRPS: the \"fraction of tasks routed\" reported in Table 2 simply corresponds to exploring different percentile-based thresholds (20%, 40%, 60%, 80% quantiles of each router's score distribution), making the results directly transferable to the per-task setting.\n\n* We explore this further and find that score ranges vary meaningfully across router sizes: larger routers spread scores across their range (e.g., Qwen2.5-32B: \\[0.03, 0.39\\], Qwen2.5-72B: \\[0.05, 0.95\\]), while smaller routers collapse tasks into a narrow band (e.g., Qwen2.5-0.5B: \\[0.76, 0.84\\]). Once a router is chosen, calibrating $\\\\tau$ would require only a small held-out set of tasks to estimate the relevant percentile of that router's score distribution \\- a lightweight, one-time step identical to standard practice in confidence-based routing \\[1, 2\\]. \n \n\n* We have updated the paper to reflect this: Section 7.2 now clarifies that our setup supports both batch and per-task routing, and a new Appendix F.2 provides a dedicated discussion with score range analysis across router sizes (Figure F.2), showing that both settings yield equivalent results.\n\n\\[1\\] Chen, Lingjiao, Matei Zaharia, and James Zou. \"FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance.\" *Transactions on Machine Learning Research*.                  \n   \n\\[2\\] Ong, Isaac, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M. Waleed Kadous, and Ion Stoica. \"RouteLLM: Learning to Route LLMs from Preference Data.\" In *The Thirteenth International Conference on Learning Representations*.\n\n> Table 1 shows that CorDP performance varies depending on the base forecaster, but the paper provides limited discussion on how to select the right base forecaster.\n\n* We thank the reviewer for the question. As we discuss in the \"SampleWise vs Median variants\" paragraph of Sec 5.3, the performance of CorDP methods does differ depending on the base forecaster used; however, all winning methods except one use Lag-Llama as the base forecaster. This is because Lag-Llama is the best performing quantitative forecaster on this data, as shown in the base forecaster row of Table 1\\.  \n* This points to a clear and practical selection criterion: to choose the best available quantitative forecaster as the base model for CorDP. Crucially, this selection does not require any labeled context-aided forecasting data \\- one can simply evaluate candidate base forecasters on standard held-out no-context data (a routine step in any forecasting pipeline) and use the best performer. This is also practically convenient: in real deployments, practitioners typically already have a quantitative forecasting model in place, and CorDP is designed to augment it with contextual reasoning rather than replace it. We show that LLMs with CorDP are flexible enough to use forecasts from any base model and incorporate context on top.   \n* We have made this selection guideline explicit in Section 5.3 in the revised version of the paper."}},"parentInvitations":"TMLR/-/Official_Comment","tmdate":1777843067561,"tcdate":1777840529675,"writers":["TMLR","TMLR/Paper7655/Authors"],"signatures":["TMLR/Paper7655/Authors"],"forum":"dkjHHFJkVI","number":5,"license":"CC BY 4.0","cdate":1777840529675,"readers":["everyone"],"invitations":["TMLR/Paper7655/-/Official_Comment"],"mdate":1777843067561,"domain":"TMLR","replyto":"5MOhYgGkWU","id":"aDr0dAYNMx","forumContent":{"submission_length":{"value":"Regular submission (no more than 12 pages of main content)"},"venue":{"value":"Accepted by TMLR"},"code":{"value":"https://github.com/ashok-arjun/beyond-naive-prompting"},"abstract":{"value":"Real-world forecasting requires models to integrate not only historical data but also relevant contextual information provided in textual form. While large language models (LLMs) show promise for context-aided forecasting, critical challenges remain: we lack diagnostic tools to understand failure modes, performance remains far below their potential, and high computational costs limit practical deployment. We introduce a unified framework of four strategies that address these limitations along three orthogonal dimensions: model diagnostics, accuracy, and efficiency. Through extensive evaluation across model families from small open-source models to frontier models including Gemini, GPT, and Claude, we uncover both fundamental insights and practical solutions. Our findings span three key dimensions: diagnostic strategies reveal the “Execution Gap” where models correctly explain how context affects forecasts but fail to apply this reasoning; accuracy-focused strategies achieve substantial performance improvements of 25-50%; and efficiency-oriented approaches show that adaptive routing between small and large models can approach large model accuracy on average while significantly reducing inference costs. These orthogonal strategies can be flexibly integrated based on deployment constraints, providing practitioners with a comprehensive toolkit for practical LLM-based context-aided forecasting. Code is made available at \\url{https://github.com/ashok-arjun/beyond-naive-prompting}."},"_bibtex":{"value":"@article{\nashok2026beyond,\ntitle={Beyond Na{\\\"\\i}ve Prompting: Strategies for Improved Context-aided Forecasting with {LLM}s},\nauthor={Arjun Ashok and Andrew Robert Williams and Vincent Zhihao Zheng and Irina Rish and Nicolas Chapados and {\\'E}tienne Marcotte and Valentina Zantedeschi and Alexandre Drouin},\njournal={Transactions on Machine Learning Research},\nissn={2835-8856},\nyear={2026},\nurl={https://openreview.net/forum?id=dkjHHFJkVI},\nnote={}\n}"},"title":{"value":"Beyond Naïve Prompting: Strategies for Improved Context-aided Forecasting with LLMs"},"changes_since_last_submission":{"value":"# Changes post-rebuttal (May 2026):                       \n                                              \nAll post-rebuttal changes appear in blue text in the PDF. \n\n  - CorDP (Sec 5.3). Promoted the task-type finding into an explicit practitioner selection criterion (SampleWise-CorDP for partial-RoI tasks; Median-CorDP for full-shape/constrained tasks).                  \n  - CorDP (Sec 5.2). Clarified that Median-CorDP and SampleWise-CorDP are mechanistically distinct and complementary, not combinable.                                                                            \n  - RouteDP (Table 2). Added a note that percentage labels are nominal and actual task counts are floored.  \n  - RouteDP (Sec 7.2 + App F.2). Clarified that the setup supports per-task routing via a score threshold, equivalent to the batch setting; added score-range analysis across router sizes.                          \n  - FxDP (Sec 4.1).  Added a note that without FxDP, both steps are performed implicitly in a single forward pass, and that FxDP externalizes this process to make intermediate failures diagnosable.    \n  - FxDP (Sec 4.3). Replaced Fig 3 with an improved version: the Execution Gap segment is now highlighted in orange (previously gray) to draw attention to the central finding, and bracket annotations below the x-axis explicitly group models and label the two key results. \n  - FxDP (Sec 4.3). Added the evaluation reliability finding (no incorrect explanation leads to an improved forecast) in the results section.    \n  - Discussion and Future Work (Sec 9). Expanded on the connections between CorDP and tool-use agents, emphasized on the need for more unconstrained benchmarks, and elaborated on future directions related to finetuning.\n\n# Initial submission (Feb 2026):                       \n\nIn this submission we address all major concerns from the previous submission through new human evaluation studies that validate our ground truth and confirm agreement between our LLM judges and human annotators, a unified presentation of our strategies, and updated results with recent frontier models.\n\n## Major changes\n\n* **FxDP: Ground truth forecast effects.** We ran a human evaluation on the Prolific platform to verify the ground truth forecast effects. Each task was evaluated by five independent annotators; we aggregated responses by majority vote and used attention checks to ensure annotation quality. We obtained at least 4 out of 5 annotator agreement on all tasks and 80–100% agreement per item, thereby validating the gold standard. Further details are given in Appendix C.2.\n\n* **FxDP: LLM judges.** We compare models’ forecast effects to the ground truth using a panel of LLM judges (GPT-5.2, Claude-Sonnet-4.5, Gemini-2.5-Pro). All pairs of judges show high agreement, which indicates that the evaluation protocol provides clear, consistent, and objective criteria.\n\n* **FxDP: Human–LLM alignment.** To verify that LLM judge results align with human judgment, we designed a human evaluation on a subset of items. Five independent annotators per item performed the same comparison task as the judges, with majority-vote aggregation and attention checks, following standard annotation practice. Agreement between human evaluators and the LLM judges is 91.7–93.3% (Cohen’s κ = 0.81–0.84), with high inter-annotator and inter-judge agreement; see Appendix C.3. We open-source all evaluation artifacts, and our original findings remain unchanged.\n\n* **FxDP: Threshold analysis.** We use a 50% relative improvement threshold in RoI as the primary criterion and report sensitivity analyses at 30%–80% in Appendix C.4. We show that our findings hold across this range of thresholds.\n\n* **FxDP, CorDP, IC-DP.** We updated results for GPT-5.2, Gemini-2.5-Pro, and Claude-Sonnet-4.5. We also report costs and inference times for all of these models.\n\n* **Unification.** We present a unified view of our proposed strategies in a new Figure 1 and report results with multiple strategies combined.\n\n* **Supplementary material.** We provide code for all proposed methods.\n\n## Minor changes\n\n* **Renaming ReDP to FxDP:** We renamed ReDP (Direct Prompting with Reasoning over Context) to FxDP (Direct Prompt with Forecast Effect Explanation) to reflect our focus on the model’s perceived effect of the context on the forecast rather than full step-by-step reasoning traces. We now call the outputs “forecast effect explanations.” The methodology and main results are unchanged.\n\n* **FxDP:** We add Figure 2, which summarizes the major results, and add the full results table in the appendix."},"pdf":{"value":"/pdf/963cd6cbcfd30ca6fbab8f051d1c3bbd6dd3c144.pdf"},"venueid":{"value":"TMLR"},"paperhash":{"value":"ashok|beyond_naïve_prompting_strategies_for_improved_contextaided_forecasting_with_llms"},"authorids":{"value":["~Arjun_Ashok1","~Andrew_Robert_Williams1","~Vincent_Zhihao_Zheng1","~Irina_Rish1","~Nicolas_Chapados1","~Étienne_Marcotte1","~Valentina_Zantedeschi2","~Alexandre_Drouin2"]},"previous_TMLR_submission_url":{"value":"https://openreview.net/forum?id=CLuE0GL5xE"},"assigned_action_editor":{"value":"~Zachary_B._Charles1"},"authors":{"value":["Arjun Ashok","Andrew Robert Williams","Vincent Zhihao Zheng","Irina Rish","Nicolas Chapados","Étienne Marcotte","Valentina Zantedeschi","Alexandre Drouin"]}},"version":2},{"content":{"comment":{"value":"## **W3/Q1. LLM-as-judge bias ------ Empirical difficulty evaluation and human validation provided**\n>W3. The use of DeepSeek-V3.1 for automated difficulty annotation may introduce systematic bias; no quantitative bias analysis or annotation-consistency study is provided. \n>Q1. Regarding the validation of Bloom's taxonomy classification—was any human sampling verification conducted, and how consistent were the results with the LLM's judgments? \n\nThank you for the thoughtful comment. We have revised Section 3 to clarify the role of Bloom's Taxonomy and the LLM-as-judge labels in our benchmark. \n\nBloom's Taxonomy measures the cognitive skill required by a question (from remembering to evaluating), and in RouterArena we **use it only as a coarse before ensuring coverage of diverse cognitive skills**. The **actual difficulty used in all analyses is defined empirically**: for each query, we compute the average accuracy over 42 LLMs (from a range of small to large models, details in Appendix C.4) and sort queries by this empirical difficulty.\n\nConcretely, we define **three difficulty bands based on the number of models that answer a query correctly**: hard (≤4/42 LLMs), medium (5~19/42), and easy (>19/42), resulting in 23.4%, 29.1%, and 47.5% of the dataset, respectively (Fig. 3 in our paper). As shown in our revised Fig. 4, the overall difficulty curve (sorted by empirical accuracy) is smooth from hardest to easiest queries, and each Dewey domain exhibits a similar smooth trend without large gaps. This indicates that the final 8.4k-query subset we used spans a broad and well-populated difficulty range for LLM routers.\n\nWe also conducted a **human validation study** by sampling roughly 5% of the data stratified over category and Bloom level (450 questions). Eighteen volunteers (undergraduate and graduate students) labeled Bloom levels under the same protocol as the judging LLM. We observe 54.9% exact agreement and 76.7% agreement within ±1 Bloom level with the human majority vote, reflecting that disagreements are mostly between adjacent levels in an inherently subjective taxonomy. **Importantly**, we do not use Bloom labels as ground-truth difficulty in any metric; they only guide dataset construction. For router evaluation, we need queries that distinguish models in the zoo, and our empirically derived difficulty levels are designed to do exactly that.\n\n---\n\n## **W4/Q2. Robustness implementation mismatch ------ We aligned the definition with the new experiment**\n>W4. The robustness test is limited, and the actual evaluation method appears inconsistent with the definition of robustness given in Section 4.\n\n>Q2. The robustness definition in Section 4 differs from the implementation described in Section 6.2 (“adding irrelevant keywords”). If only keyword-insertion was used, does this cause inconsistency between the definition and the actual test? \n\n\n\nThank you for the careful reading. **We have now aligned the evaluation procedure with our robustness definition.** Specifically, we assign equal weight to the four noisy variants described in the definition. For each category and its subcategories, we randomly sampled 5% of the entire dataset, which is 420 examples, and generated perturbed queries using GPT-4o. The prompt we used is provided in Appendix G. After regenerating all noisy queries, we reran the robustness evaluation and found that the open-source routers that rely on BERT for latent representations (e.g., vLLM-SR, MIRT-BERT, and NIRT-BERT) exhibit noticeably weaker robustness than the others. This aligns with prior findings that BERT is sensitive to surface-level perturbations.[1]\n\n---\n\n### Table 1: Robustness score on rotuers. \n| Router      | Robustness |\n| ----------- | ---------- |\n| vLLM-SR | 35.00      |\n| NIRT-BERT        | 49.29      |\n| Auzre-Router       | 54.07      |\n| NotDiamond  | 55.91      |\n| MIRT-BERT        | 61.19      |\n| MLP         | 80.00      |\n| KNN         | 83.33      |\n| Kmeans      | 84.52      |\n| RouterDC    | 85.24      |\n| CARROT      | 89.05      |\n| GraphRouter | 94.29      |\n| RouteLLM    | 100.00     |\n\n[1] Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment. arXiv:1907.11932"},"title":{"value":"Thanks! (2/3)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763966020808,"tcdate":1763964704254,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission13871/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission13871/Authors"],"forum":"9HsaIi4ngF","number":5,"license":"CC BY 4.0","cdate":1763964704254,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission13871/-/Official_Comment"],"mdate":1763966020808,"domain":"ICLR.cc/2026/Conference","replyto":"LHAfoFB9LF","id":"Fgth9qWeiS","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"TLDR":{"value":"We proposed a comprehensive LLM router evaluation framework, with a new dataset, metrics, and ranking system."},"keywords":{"value":["LLM Router","Evaluation"]},"primary_area":{"value":"datasets and benchmarks"},"abstract":{"value":"Today's LLM ecosystem comprises a wide spectrum of models that differ in size, capability, and cost. No single model is optimal for all scenarios; hence, LLM routers have become essential for selecting the most appropriate model under varying circumstances.\nHowever, the rapid emergence of various routers has led to fragmented evaluation practices and inconsistent metrics, making it difficult to systematically assess progress in this space. To address this problem, we need a comprehensive router comparison and a standardized leaderboard, similar to those available for models. In this work, we introduce RouterArena, the first open platform enabling comprehensive comparison of LLM routers. RouterArena has (1) a principally constructed dataset with broad knowledge domain coverage, (2) distinguishable difficulty levels for each domain, (3) an extensive list of evaluation metrics, and (4) an automated framework for evaluation and leaderboard updates. Leveraging this framework, we have produced the initial leaderboard with detailed metrics comparison.\nFigure1 provides a preview of the leaderboard. The complete framework and the latest router leaderboard are publicly available at https://routeworks.github.io/"},"_bibtex":{"value":"@inproceedings{\nlu2026routerarena,\ntitle={RouterArena: An Open Platform for Comprehensive Comparison of {LLM} Routers},\nauthor={Yifan Lu and Rixin Liu and Jiayi Yuan and Xingqi Cui and Shenrun Zhang and Hongyi Liu and Jiarong Xing},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=9HsaIi4ngF}\n}"},"title":{"value":"RouterArena: An Open Platform for Comprehensive Comparison of LLM Routers"},"pdf":{"value":"/pdf/03e5d4326116c387dbd69c38e194211db0736cda.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"lu|routerarena_an_open_platform_for_comprehensive_comparison_of_llm_routers"},"authorids":{"value":["~Yifan_Lu8","~Rixin_Liu1","~Jiayi_Yuan1","~Xingqi_Cui1","~Shenrun_Zhang1","~Hongyi_Liu5","~Jiarong_Xing1"]},"authors":{"value":["Yifan Lu","Rixin Liu","Jiayi Yuan","Xingqi Cui","Shenrun Zhang","Hongyi Liu","Jiarong Xing"]}},"version":2},{"content":{"comment":{"value":"**Table R1**: Performance comparison between CITER and our baselines on the CommonSense QA dataset. Higher values indicate better performance. The best result is **bolded**.\n| Accuracy (%) \\ Data Transfer Amount (GB) | 9.1  | 29.7     | 100.0    | 200.0    | 300.0    | 400.0    | 487.9    | 547.7 |\n| ---------------------------------------- | ---- | -------- | -------- | -------- | -------- | -------- | -------- | ----- |\n| SLM                                      | 34.4 | /        | /        | /        | /        | /        | /        | /     |\n| LLM                                      | /    | /        | /        | /        | /        | /        | /        | 87.0  |\n| Speculative Decoding                     | /    | /        | /        | /        | /        | /        | **87.0** | /     |\n| RouteLLM                                 | /    | 37.8     | 49.1     | 59.5     | 66.4     | 71.5     | 81.4     | /     |\n| Co-LLM                                   | /    | 41.0     | 62.0     | 67.6     | 76.3     | 79.9     | 83.9     | /     |\n| LMC                                      | /    | 36.3     | 42.9     | 52.1     | 62.7     | 72.5     | 81.2     | /     |\n| FCSD                                     | /    | 40.5     | 49.2     | 62.0     | /        | /        | /        | /     |\n| CITER (Ours)                             | /    | **57.6** | **63.6** | **71.3** | **78.0** | **81.5** | 84.6     | /     |\n\n---\n\n**Q2**: KV caching is an important component of modern LLM inference. The paper needs to expand the relevant discussion (Remark 2.1) to make it more comprehensive. For instance, let's say $KV_{small}$ and $KV_{large}$ denote the KV caches of SLM and LLM, respectively. If the router returns to LLM, say after t tokens (with last t tokens decoded by SLM), then LLM would have to compute Key-values for the last t tokens in the context (which were generated by the LLM). Thus, it appears that the proposed method would require running a forward pass on all the tokens by both SLM and LLM. Could the authors clarify this point? Also, did the authors take this into account while studying the quality vs. cost tradeoff for the proposed method?\n\n**R2**: We thank the reviewer for highlighting the critical issue of KV cache reuse and the true cost of switching between SLM and LLM.\n\nWe acknowledge that if the router decides to route to the large model (LLM) near the end of the generation process, then both the SLM and LLM will have processed almost the entire sequence of tokens, except for the tokens generated by the SLM after the final LLM invocation. In our experimental setup, we have already included this recomputation and the associated data movement overhead in our calculation of inference cost (data transfer amount), as reported in the main results.\n\nAlthough in the worst-case scenario the LLM may need to process nearly all tokens, in practice, the majority of tokens during generation are produced by the SLM. The number of LLM forward passes is thus greatly reduced compared to always using the LLM. At the decoding stage, although KV cache mechanisms significantly reduce the per-step computational requirements of LLM inference, each forward pass still incurs the cost of transferring the model’s KV cache from off-chip memory (GPU HBM) to on-chip cache. By reducing the number of LLM forward passes, our method substantially reduces the frequency of KV cache transfers, thereby achieving significant savings in data transfer and associated inference cost.\n\nTo be more concrete, suppose the input has m tokens and the output has n tokens. In standard LLM decoding, the total data movement (in terms of KV cache transfer) is proportional to $m + (m+1) + \\cdots + (m+n-1) = n(m + m + n - 1)/2 ≈ mn + n^2/2$, with the proportionality constant determined by the LLM’s per-token KV cache size. While, in our method, let $\\alpha$ denote the proportion of tokens routed to the LLM (assuming routing is uniformly distributed). Ignoring the comparatively negligible KV cache size of the SLM, our approach incurs total data movement proportional to $\\alpha n(m + n/2) = \\alpha(mn + n^2/2)$. For clarity, this analytic estimate neglects the SLM’s data movement, but in our experiments we do account for the SLM’s actual data transfer contribution. Our results show that even with a small $\\alpha$ (e.g., <5%), CITER delivers a significant accuracy boost compared to SLM alone; with $\\alpha$ between 1/3 and 1/2, CITER achieves output quality comparable to the full LLM, but at much lower total inference cost. As we discuss in Section 3.1, since LLM generation is memory-bound, reducing the total data transfer and KV cache movement directly alleviates the I/O bottleneck in practical deployments."},"title":{"value":"Response to Reviewer mhTX (2/4)"}},"parentInvitations":"colmweb.org/COLM/2025/Conference/-/Official_Comment","tmdate":1756226122908,"tcdate":1748919619885,"writers":["colmweb.org/COLM/2025/Conference","colmweb.org/COLM/2025/Conference/Submission1200/Authors"],"signatures":["colmweb.org/COLM/2025/Conference/Submission1200/Authors"],"forum":"nqX9UYW9Af","number":3,"license":"CC BY 4.0","cdate":1748919619885,"readers":["everyone"],"invitations":["colmweb.org/COLM/2025/Conference/Submission1200/-/Official_Comment","colmweb.org/COLM/2025/Conference/-/Edit"],"mdate":1756226122908,"domain":"colmweb.org/COLM/2025/Conference","replyto":"VCh7Jyz8K9","id":"x7YJ67Xzo6","forumContent":{"TLDR":{"value":"A novel Collaborative Inference with Token-lEvel Routing (CITER) framework that introduces a token-level routing mechanism, enabling efficient collaboration between small and large language models (SLMs & LLMs)."},"venue":{"value":"COLM 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the COLM Code of Ethics on https://colmweb.org/CoE.html"},"keywords":{"value":["collaborative inference","efficient inference","token-level routing","large language model"]},"abstract":{"value":"Large language models have achieved remarkable success in various tasks but suffer from high computational costs during inference, limiting their deployment in resource-constrained applications. To address this issue, we propose a novel Collaborative Inference with Token-lEvel Routing (CITER) framework that enables efficient collaboration between small and large language models (SLMs & LLMs) through a token-level routing strategy. Specifically, CITER routes non-critical tokens to an SLM for efficiency and routes critical tokens to an LLM for generalization quality. We formulate router training as a policy optimization, where the router receives rewards based on both the quality of predictions and the inference costs of generation. This allows the router to learn to predict token-level routing scores and make routing decisions based on both the current token and the future impact of its decisions. To further accelerate the reward evaluation process, we introduce a shortcut which significantly reduces the costs of the reward estimation and improving the practicality of our approach. Extensive experiments on five benchmark datasets demonstrate that CITER reduces the inference costs while preserving high-quality generation, offering a promising solution for real-time and resource-constrained applications."},"_bibtex":{"value":"@inproceedings{\nzheng2025citer,\ntitle={{CITER}: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routing},\nauthor={Wenhao Zheng and Yixiao Chen and Weitong Zhang and Souvik Kundu and Yun Li and Zhengzhong Liu and Eric P. Xing and Hongyi Wang and Huaxiu Yao},\nbooktitle={Second Conference on Language Modeling},\nyear={2025},\nurl={https://openreview.net/forum?id=nqX9UYW9Af}\n}"},"title":{"value":"CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routing"},"pdf":{"value":"/pdf/44adc116c02c1f30777bdd97bc65accd40cb593a.pdf"},"venueid":{"value":"colmweb.org/COLM/2025/Conference"},"paperhash":{"value":"zheng|citer_collaborative_inference_for_efficient_large_language_model_decoding_with_tokenlevel_routing"},"authorids":{"value":["~Wenhao_Zheng4","~Yixiao_Chen2","~Weitong_Zhang2","~Souvik_Kundu2","~Yun_Li7","~Zhengzhong_Liu1","~Eric_Xing1","~Hongyi_Wang1","~Huaxiu_Yao1"]},"author_guide":{"value":"I certify that this submission complies with the submission instructions as described on https://colmweb.org/AuthorGuide.html"},"authors":{"value":["Wenhao Zheng","Yixiao Chen","Weitong Zhang","Souvik Kundu","Yun Li","Zhengzhong Liu","Eric P. Xing","Hongyi Wang","Huaxiu Yao"]}},"version":2},{"content":{"comment":{"value":"We thank the reviewer for the follow-up questions. \n\n**KQ1.** We identified this phenomenon during our algorithm development, and it directly motivated the hint size threshold $\\\\eta_{\\text{hint}}$ in our policy design (Section 3.4). The gap at large hint sizes is structural: even with a near-complete LLM response as a prefix, the SLM generates a response that may still produce errors – wasting the LLM cost while failing to match LLM accuracy.\nOur framework addresses this with a three-way decision: (1) no-hint → SLM only, (2) predicted hint size ≤ $\\\\eta_{\\text{hint}}$ → shepherding, (3) predicted hint > $\\\\eta_{\\text{hint}}$ → route directly to LLM, bypassing the SLM entirely. The threshold $\\\\eta_{\\text{hint}}​$ is tuned per dataset (Table 5, Appendix F): GSM8K = 58, CNK12 = 60, HumanEval = 110, MBPP = 130 tokens. This ensures shepherding operates only in the regime where partial hints are most cost-effective, while queries requiring near-complete LLM reasoning are handled by the LLM at full accuracy. The reviewer's observation about the turning point is precisely what $\\\\eta_{\\text{hint}}$ captures.\n\n**KQ2.** Thank you for this important clarification – we now better understand the reviewer's concern about possible alternative SLM-LLM collaboration architectures that could achieve similar cost-accuracy trade-offs. We address each suggestion.\n\n*On DEER / early answer inducing.* We appreciate this suggestion and the analogy between increasing accuracy with hint size in our paper and increasing accuracy with exit positions in Figure 2 of DEER (https://openreview.net/pdf?id=NpU7ZXafRi). However, we highlight two important differences that make direct comparison challenging.\n\nFirst, DEER requires fine-grained control over the LLM's generation process: it must interrupt generation mid-stream at specific token positions, inject an answer-inducing prompt, evaluate confidence via logprobs, and conditionally resume or roll back. Standard LLM APIs – including Groq – treat generation as a single atomic call with no mechanism to pause, inspect, inject, and conditionally continue. While some APIs expose logprobs (e.g., OpenAI's logprobs parameter), the mid-stream interruption and injection mechanism DEER relies on is not supported by standard LLM APIs. Shepherding, by contrast, requires only the standard max_new_tokens parameter available in every LLM API, making it compatible with any provider.\n\nSecond, DEER increases LLM compute overhead rather than reducing it. At every candidate exit point, DEER performs additional forward passes to generate trial answers and evaluate confidence (DEER-Pro uses N=4 passes per exit point). The savings come from stopping the main CoT early, not from fewer model calls. Shepherding makes at most one LLM API call per query, making the cost structures fundamentally different.\n\nThe reviewer’s question about where the turning point lies – when it is cheaper to let the LLM finish rather than hand off to the SLM – is precisely what our threshold $\\\\eta_{\\text{hint}}$ addresses. \n\n*On SLM CoT → LLM final answer.* This is a legitimate alternative architecture that exploits the input/output price differential. However, prior work by [Yue et al.,2024] shows that passing weaker-model outputs as context to an LLM can degrade performance, particularly when the SLM reasoning contains errors that mislead the LLM. Our approach avoids this failure mode entirely: the hint flows from the LLM to the SLM, ensuring the reasoning prefix is always high-quality.\n\nWe did not intend to suggest that no other SLM-LLM combinations exist. Nevertheless, within the routing/cascading paradigm, our baselines (RouteLLM, GraphRouter, FrugalGPT, ABC) are representative of the state of the art. We totally agree that the design space of SLM-LLM collaboration is broader. Yet, different paradigms like DEER and Speculative Decoding often operate with different structures or optimize for disparate objectives – making a head-to-head benchmark inherently challenging. \n\n*[Yue et al.,2024] Yue, Murong, Jie Zhao, Min Zhang, Liang Du, and Ziyu Yao. \"Large Language Model Cascades with Mixture of Thought Representations for Cost-Efficient Reasoning.\" in Proc. ICLR, 2024.\nhttps://arxiv.org/abs/2310.03094*\n\nWe will add a discussion of DEER, the ICLR 2024 paper, and similar approaches to the related work section of the revised paper."}},"parentInvitations":"ICML.cc/2026/Conference/-/Reply_Rebuttal_Comment","tmdate":1782369956852,"tcdate":1775235878795,"writers":["ICML.cc/2026/Conference","ICML.cc/2026/Conference/Submission30175/Authors"],"signatures":["ICML.cc/2026/Conference/Submission30175/Authors"],"forum":"H8rywbFXuM","number":1,"license":"CC BY 4.0","cdate":1775235878795,"readers":["everyone"],"invitations":["ICML.cc/2026/Conference/Submission30175/Official_Review2/Rebuttal1/Rebuttal_Acknowledgement1/-/Reply_Rebuttal_Comment","ICML.cc/2026/Conference/-/Edit"],"mdate":1782369956852,"domain":"ICML.cc/2026/Conference","replyto":"klJnnoXAWa","id":"y7HnYAG7LC","forumContent":{"venue":{"value":"Submitted to ICML 2026"},"keywords":{"value":["LLM routing; LLM cascading; SLM-LLM collaboration; Cost-efficient inference"]},"_bibtex":{"value":"@misc{\nanonymous2026pay,\ntitle={Pay for Hints, Not Answers: {LLM} Shepherding for Cost-Efficient Inference},\nauthor={Anonymous},\nyear={2026},\nurl={https://openreview.net/forum?id=H8rywbFXuM}\n}"},"title":{"value":"Pay for Hints, Not Answers: LLM Shepherding for Cost-Efficient Inference"},"paperhash":{"value":"dong|pay_for_hints_not_answers_llm_shepherding_for_costefficient_inference"},"primary_area":{"value":"deep_learning->large_language_models"},"abstract":{"value":"Large Language Models (LLMs) deliver state-of-the-art performance on complex reasoning tasks, but their inference costs limit deployment at scale. Small Language Models (SLMs) offer dramatic cost savings yet lag substantially in accuracy. Existing approaches—routing and cascading—treat the LLM as an all-or-nothing resource: either the query bypasses the LLM entirely, or the LLM generates a complete response at full cost. We introduce **LLM Shepherding**, a framework that requests only a short prefix (a hint) from the LLM and provides it to SLM. This simple mechanism is surprisingly effective for math and coding tasks: even hints comprising 10--30\\% of the full LLM response improve SLM accuracy significantly. Shepherding generalizes both routing and cascading, and it achieves lower cost under oracle decision-making. We develop a two-stage predictor that jointly determines whether a hint is needed and how many tokens to request. On the widely-used mathematical reasoning (GSM8K, CNK12) and code generation (HumanEval, MBPP) benchmarks, Shepherding reduces costs by 42--94\\% relative to LLM-only inference. Compared to state-of-the-art routing and cascading baselines, shepherding delivers up to 2.8$\\times$ cost reduction while matching accuracy. To our knowledge, this is the first work to exploit token-level budget control for SLM-LLM collaboration. Code is available at https://anonymous.4open.science/r/LLM_Sheperding-77C3/."},"pdf":{"value":"/pdf/b5f1805f8847279424d1a89d1e99c69e9d93c924.pdf"},"venueid":{"value":"ICML.cc/2026/Conference/Rejected_Submission"},"authorids":{"value":["~Ziming_Dong1","~Hardik_Sharma3","~Evan_O'Toole1","~Jaya_Prakash_Champati1","~Kui_Wu1"]},"authors":{"value":["Ziming Dong","Hardik Sharma","Evan O'Toole","Jaya Prakash Champati","Kui Wu"]}},"version":2},{"content":{"comment":{"value":"**Q3. Many existing routing and distillation methods already support API-only settings. The unique value of this work over prior art must be strengthened. Is it necessary to study this matrix in this paper?**\n\n**Response:**  Thank you for the important comment. We agree that API-only compatibility has been considered in several existing routing and distillation methods. We would like to clarify that our paper does not position API-only compatibility itself as the sole novelty. Rather, our motivation is to study how the same source of multi-LLM log data can be systematically used for capability fusion at different stages of the LLM pipeline. This motivation has already been discussed in the original paper. In **Section 1**, we argue that practical fusion should satisfy two requirements: compatibility with real-world serving scenarios, including local and API-based serving, and flexibility to operate at different stages of the LLM pipeline. This motivation is further illustrated in **Table 1**, where we compare prior work along query-level, thought-level, and model-level fusion stages, API-serving compatibility, and task-domain coverage.\n\nThe key distinction from prior work is that existing routing and distillation methods usually study one fusion stage in isolation. Routing methods mainly use multi-LLM logs for query-level model selection, while distillation methods mainly use model outputs for model-level capability transfer. These works are valuable, but they do not answer the broader practical question studied in our paper: given the same multi-LLM log data, should we use it for routing, thought-level prompting, or distillation? Therefore, the matrix studied in this paper is necessary because it turns isolated API-compatible techniques into a systematic comparison of different ways to exploit multi-LLM logs.\n\nTo make this contribution clearer, we have revised **Section 1** to strengthen the positioning of our work. Specifically, we now emphasize that our goal is not simply to support API-only settings, but to provide a unified benchmark and framework for studying multi-LLM log data as a general supervision source across different fusion stages. We also revised the discussion around **Table 1** to clarify that prior routing methods, such as RouteLLM, GraphRouter, RouterBench, and LLM-Blender, mainly focus on query-level fusion, while prior distillation or model-fusion methods mainly focus on model-level fusion. In contrast, FusionFactory jointly studies query-level, thought-level, and model-level fusion under the same benchmark and evaluation protocol.\n\nWe have also revised **Section 3: FusionFactory** to connect this matrix to a stage-aware taxonomy, as illustrated in **Figure 4**. Query-level fusion is an early-stage intervention that selects a model before generation. Thought-level fusion is a mid-stage intervention that retrieves and reuses abstract reasoning templates during generation. Model-level fusion is a late-stage intervention that consolidates high-quality logged responses into a single model through supervised fine-tuning. This revised explanation makes clear that the matrix is not an arbitrary engineering checklist, but a structured organization of fusion methods according to where they intervene in the LLM pipeline.\n\nIn addition, we have revised **Section 7** to better explain why studying the matrix leads to insights that cannot be obtained by studying only routing or only distillation. As shown in the cross-level comparison, different fusion stages exhibit different strengths and failure modes: thought-level fusion achieves the strongest overall performance, query-level fusion offers a favorable accuracy-cost trade-off, and model-level fusion may suffer from generalization limits. These findings demonstrate the value of comparing fusion levels under a shared benchmark, shared log source, and shared evaluation setup.\n\nFinally, we have added a discussion in **Limitations and Future Work** to further clarify the role of this matrix. The fusion-stage matrix provides a practical decision framework for users with different deployment constraints. Query-level fusion is suitable for lightweight and cost-sensitive serving, thought-level fusion is useful when users want to reuse reasoning patterns without fine-tuning, and model-level fusion is useful when users want to deploy a single fused model after offline training. Future work can further extend this matrix by studying hybrid or adaptive strategies that dynamically switch among fusion levels based on query difficulty, budget constraints, domain type, or uncertainty estimates."},"title":{"value":"Response (3)"}},"parentInvitations":"TMLR/-/Official_Comment","tmdate":1779260578545,"tcdate":1779260578545,"writers":["TMLR","TMLR/Paper7564/Authors"],"signatures":["TMLR/Paper7564/Authors"],"forum":"N951scS3yE","number":6,"license":"CC BY 4.0","cdate":1779260578545,"readers":["everyone"],"invitations":["TMLR/Paper7564/-/Official_Comment"],"mdate":1779260578545,"domain":"TMLR","replyto":"7XBJahTacJ","id":"VKdSDenrbX","forumContent":{"submission_length":{"value":"Regular submission (no more than 12 pages of main content)"},"venue":{"value":"Accepted by TMLR"},"code":{"value":"https://github.com/ulab-uiuc/FusionFactory"},"abstract":{"value":"The rapid advancement of large language models (LLMs) has created a diverse landscape of models, each excelling at different tasks. This diversity drives researchers to employ multiple LLMs in practice, leaving behind valuable multi-LLM log data. This naturally leads to the question of whether such logs can be fully leveraged to fuse LLMs' complementary capabilities. Although prior work has explored various strategies for integrating multiple LLMs, we argue that practical fusion must meet two essential requirements: (1) compatibility with real-world serving scenarios (e.g., local and API-based serving), and (2) flexibility to operate at different stages of the LLM pipeline to meet varied user needs (e.g., fine-tuning and inference stages). To this end, we introduce LLMFusionBench, a large-scale benchmark for LLM fusion that spans 14 tasks across six domains, with responses from 20 open-source LLMs (8B-671B) totaling 103M tokens. Building on LLMFusionBench, we propose FusionFactory, a systematic framework with three elaborated levels: (1) query-level fusion via tailored LLM routers, (2) thought-level fusion leveraging retrieved abstract reasoning templates, and (3) model-level fusion via distillation from top-ranked responses. Experiments show that FusionFactory consistently outperforms the best individual LLM across all 14 benchmarks, with the optimal fusion configuration varying across benchmarks, highlighting the promise of multi-LLM log data as a practical foundation for fusing diverse LLM capabilities."},"_bibtex":{"value":"@article{\nfeng2026fusionfactory,\ntitle={FusionFactory: Fusing {LLM} Capabilities with Multi-{LLM} Log Data},\nauthor={Tao Feng and Haozhen Zhang and Zijie Lei and Pengrui Han and Mostofa Patwary and Mohammad Shoeybi and Bryan Catanzaro and Jiaxuan You},\njournal={Transactions on Machine Learning Research},\nissn={2835-8856},\nyear={2026},\nurl={https://openreview.net/forum?id=N951scS3yE},\nnote={}\n}"},"title":{"value":"FusionFactory: Fusing LLM Capabilities with Multi-LLM Log Data"},"pdf":{"value":"/pdf/96397604f23ee7e8f7f9ff9fb9b73ae6622ffdad.pdf"},"venueid":{"value":"TMLR"},"paperhash":{"value":"feng|fusionfactory_fusing_llm_capabilities_with_multillm_log_data"},"authorids":{"value":["~Tao_Feng5","~Haozhen_Zhang1","~Zijie_Lei1","~Pengrui_Han1","~Mostofa_Patwary1","~Mohammad_Shoeybi1","~Bryan_Catanzaro1","~Jiaxuan_You2"]},"assigned_action_editor":{"value":"~Jaeho_Lee3"},"authors":{"value":["Tao Feng","Haozhen Zhang","Zijie Lei","Pengrui Han","Mostofa Patwary","Mohammad Shoeybi","Bryan Catanzaro","Jiaxuan You"]}},"version":2},{"content":{"comment":{"value":"> (strengthen) Can the two CorDP approaches be combined, or can we predict beforehand which one is more likely to succeed?\n\n* Thank you for the questions. The two variants cannot be directly combined, as they are fundamentally different in how they handle forecast distributions (Sec 5.2). Median-CorDP discards the original distribution entirely \\- it uses only the base forecast median as a starting point and draws new samples from the LLM to form a corrected distribution. SampleWise-CorDP, by contrast,  preserves the original probabilistic structure by correcting each individual sample from the base forecast. The two variants are instead complementary by design, each suited to different task types, as discussed in Sec 5.3.  \n* On predicting which variant is more likely to succeed: the choice depends weakly on the LLM \\- among the 13 models where CorDP outperforms DP,  SampleWise-CorDP is best for 7 models and Median-CorDP for the remaining 6, with no clear LLM-level predictor. However,  task type is a stronger and more actionable signal (Sec 5.3 \\- paragraph titled “Performance varies by task type”) - SampleWise-CorDP excels on partial-RoI tasks, while Median-CorDP dominates full-shape and constrained tasks. This task-type specialization provides practitioners with a principled guide for variant selection.  \n* We have made this distinction and the task-type guidance explicit in Sections 5.2 and 5.3 of the revised version of the paper.\n\n> (strengthen) p10 If you have 71 tasks, how do you round/truncate to pick 20% increments?\n\n\n* Thank you for the question. The reported percentages are nominal labels for readability; the actual number of tasks routed is obtained by flooring (e.g., 20% of 71 tasks gives 14 tasks, 40% gives 28, and so on). We note that Appendix F.2 already provides the full routing curve without any rounding, so the coarse increments in the main text table are purely a presentational choice for conciseness.  \n* We have made this rounding convention clearer in the caption of Table 2\\.\n\n> (strengthen) For routing, if you didn't have multiple tasks to establish relative rankings, how would you decide which model to pick for a single task?\n    \n* We thank the reviewer for the question. As stated in Section 7.2, the router outputs a continuous numerical difficulty score between 0 (easiest) and 1 (hardest) for each task, so per-task routing is already supported: if we do not have multiple tasks to establish relative ranking, a threshold $\\\\tau$ can be chosen such that any task scoring above $\\\\tau$ is routed to the large model. This per-task setting and our batch setting are equivalent \\- they route the same tasks and achieve the same RCRPS: the \"fraction of tasks routed\" in Table 2 simply corresponds to exploring different percentile-based thresholds (20%, 40%, 60%, 80% quantiles of the router's score distribution).   \n\n\n* We explore this further and find that score ranges vary meaningfully across router sizes: larger routers (e.g., Qwen2.5-32B: \\[0.03, 0.39\\], Qwen2.5-72B: \\[0.05, 0.95\\]) spread scores across their range, while smaller routers collapse tasks into a narrow band (e.g., Qwen2.5-0.5B: \\[0.76, 0.84\\]). Once a router is chosen, calibrating $\\\\tau$ would require only a small held-out set of 5–10 tasks to estimate the relevant percentile of that router's score distribution \\- a lightweight, one-time step identical to standard practice in confidence-based routing \\[1, 2\\]. \n\n* We have updated the paper to reflect this: Section 7.2 now clarifies that our setup supports both batch and per-task routing, and a new Appendix F.2 provides a dedicated discussion with score range analysis across router sizes (Figure 48), showing that both settings yield equivalent results.\n\n\\[1\\] Chen, Lingjiao, Matei Zaharia, and James Zou. \"FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance.\" *Transactions on Machine Learning Research*.                  \n   \n\\[2\\] Ong, Isaac, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M. Waleed Kadous, and Ion Stoica. \"RouteLLM: Learning to Route LLMs from Preference Data.\" In *The Thirteenth International Conference on Learning Representations*.\n\n\n---\nWe hope these revisions fully address the raised concerns and that the updated paper provides a clearer and more complete picture of our contributions; we welcome any further questions."}},"parentInvitations":"TMLR/-/Official_Comment","tmdate":1777843344052,"tcdate":1777841612884,"writers":["TMLR","TMLR/Paper7655/Authors"],"signatures":["TMLR/Paper7655/Authors"],"forum":"dkjHHFJkVI","number":10,"license":"CC BY 4.0","cdate":1777841612884,"readers":["everyone"],"invitations":["TMLR/Paper7655/-/Official_Comment"],"mdate":1777843344052,"domain":"TMLR","replyto":"GQSfVq1Evn","id":"p4ONbLWvzm","forumContent":{"submission_length":{"value":"Regular submission (no more than 12 pages of main content)"},"venue":{"value":"Accepted by TMLR"},"code":{"value":"https://github.com/ashok-arjun/beyond-naive-prompting"},"abstract":{"value":"Real-world forecasting requires models to integrate not only historical data but also relevant contextual information provided in textual form. While large language models (LLMs) show promise for context-aided forecasting, critical challenges remain: we lack diagnostic tools to understand failure modes, performance remains far below their potential, and high computational costs limit practical deployment. We introduce a unified framework of four strategies that address these limitations along three orthogonal dimensions: model diagnostics, accuracy, and efficiency. Through extensive evaluation across model families from small open-source models to frontier models including Gemini, GPT, and Claude, we uncover both fundamental insights and practical solutions. Our findings span three key dimensions: diagnostic strategies reveal the “Execution Gap” where models correctly explain how context affects forecasts but fail to apply this reasoning; accuracy-focused strategies achieve substantial performance improvements of 25-50%; and efficiency-oriented approaches show that adaptive routing between small and large models can approach large model accuracy on average while significantly reducing inference costs. These orthogonal strategies can be flexibly integrated based on deployment constraints, providing practitioners with a comprehensive toolkit for practical LLM-based context-aided forecasting. Code is made available at \\url{https://github.com/ashok-arjun/beyond-naive-prompting}."},"_bibtex":{"value":"@article{\nashok2026beyond,\ntitle={Beyond Na{\\\"\\i}ve Prompting: Strategies for Improved Context-aided Forecasting with {LLM}s},\nauthor={Arjun Ashok and Andrew Robert Williams and Vincent Zhihao Zheng and Irina Rish and Nicolas Chapados and {\\'E}tienne Marcotte and Valentina Zantedeschi and Alexandre Drouin},\njournal={Transactions on Machine Learning Research},\nissn={2835-8856},\nyear={2026},\nurl={https://openreview.net/forum?id=dkjHHFJkVI},\nnote={}\n}"},"title":{"value":"Beyond Naïve Prompting: Strategies for Improved Context-aided Forecasting with LLMs"},"changes_since_last_submission":{"value":"# Changes post-rebuttal (May 2026):                       \n                                              \nAll post-rebuttal changes appear in blue text in the PDF. \n\n  - CorDP (Sec 5.3). Promoted the task-type finding into an explicit practitioner selection criterion (SampleWise-CorDP for partial-RoI tasks; Median-CorDP for full-shape/constrained tasks).                  \n  - CorDP (Sec 5.2). Clarified that Median-CorDP and SampleWise-CorDP are mechanistically distinct and complementary, not combinable.                                                                            \n  - RouteDP (Table 2). Added a note that percentage labels are nominal and actual task counts are floored.  \n  - RouteDP (Sec 7.2 + App F.2). Clarified that the setup supports per-task routing via a score threshold, equivalent to the batch setting; added score-range analysis across router sizes.                          \n  - FxDP (Sec 4.1).  Added a note that without FxDP, both steps are performed implicitly in a single forward pass, and that FxDP externalizes this process to make intermediate failures diagnosable.    \n  - FxDP (Sec 4.3). Replaced Fig 3 with an improved version: the Execution Gap segment is now highlighted in orange (previously gray) to draw attention to the central finding, and bracket annotations below the x-axis explicitly group models and label the two key results. \n  - FxDP (Sec 4.3). Added the evaluation reliability finding (no incorrect explanation leads to an improved forecast) in the results section.    \n  - Discussion and Future Work (Sec 9). Expanded on the connections between CorDP and tool-use agents, emphasized on the need for more unconstrained benchmarks, and elaborated on future directions related to finetuning.\n\n# Initial submission (Feb 2026):                       \n\nIn this submission we address all major concerns from the previous submission through new human evaluation studies that validate our ground truth and confirm agreement between our LLM judges and human annotators, a unified presentation of our strategies, and updated results with recent frontier models.\n\n## Major changes\n\n* **FxDP: Ground truth forecast effects.** We ran a human evaluation on the Prolific platform to verify the ground truth forecast effects. Each task was evaluated by five independent annotators; we aggregated responses by majority vote and used attention checks to ensure annotation quality. We obtained at least 4 out of 5 annotator agreement on all tasks and 80–100% agreement per item, thereby validating the gold standard. Further details are given in Appendix C.2.\n\n* **FxDP: LLM judges.** We compare models’ forecast effects to the ground truth using a panel of LLM judges (GPT-5.2, Claude-Sonnet-4.5, Gemini-2.5-Pro). All pairs of judges show high agreement, which indicates that the evaluation protocol provides clear, consistent, and objective criteria.\n\n* **FxDP: Human–LLM alignment.** To verify that LLM judge results align with human judgment, we designed a human evaluation on a subset of items. Five independent annotators per item performed the same comparison task as the judges, with majority-vote aggregation and attention checks, following standard annotation practice. Agreement between human evaluators and the LLM judges is 91.7–93.3% (Cohen’s κ = 0.81–0.84), with high inter-annotator and inter-judge agreement; see Appendix C.3. We open-source all evaluation artifacts, and our original findings remain unchanged.\n\n* **FxDP: Threshold analysis.** We use a 50% relative improvement threshold in RoI as the primary criterion and report sensitivity analyses at 30%–80% in Appendix C.4. We show that our findings hold across this range of thresholds.\n\n* **FxDP, CorDP, IC-DP.** We updated results for GPT-5.2, Gemini-2.5-Pro, and Claude-Sonnet-4.5. We also report costs and inference times for all of these models.\n\n* **Unification.** We present a unified view of our proposed strategies in a new Figure 1 and report results with multiple strategies combined.\n\n* **Supplementary material.** We provide code for all proposed methods.\n\n## Minor changes\n\n* **Renaming ReDP to FxDP:** We renamed ReDP (Direct Prompting with Reasoning over Context) to FxDP (Direct Prompt with Forecast Effect Explanation) to reflect our focus on the model’s perceived effect of the context on the forecast rather than full step-by-step reasoning traces. We now call the outputs “forecast effect explanations.” The methodology and main results are unchanged.\n\n* **FxDP:** We add Figure 2, which summarizes the major results, and add the full results table in the appendix."},"pdf":{"value":"/pdf/963cd6cbcfd30ca6fbab8f051d1c3bbd6dd3c144.pdf"},"venueid":{"value":"TMLR"},"paperhash":{"value":"ashok|beyond_naïve_prompting_strategies_for_improved_contextaided_forecasting_with_llms"},"authorids":{"value":["~Arjun_Ashok1","~Andrew_Robert_Williams1","~Vincent_Zhihao_Zheng1","~Irina_Rish1","~Nicolas_Chapados1","~Étienne_Marcotte1","~Valentina_Zantedeschi2","~Alexandre_Drouin2"]},"previous_TMLR_submission_url":{"value":"https://openreview.net/forum?id=CLuE0GL5xE"},"assigned_action_editor":{"value":"~Zachary_B._Charles1"},"authors":{"value":["Arjun Ashok","Andrew Robert Williams","Vincent Zhihao Zheng","Irina Rish","Nicolas Chapados","Étienne Marcotte","Valentina Zantedeschi","Alexandre Drouin"]}},"version":2},{"content":{"comment":{"value":"Thank you for your detailed review and constructive feedback. We appreciate your time in evaluating our work. We are pleased that you find our paper has \"clear writing,\" is \"thoroughly constructed and well-designed,\" and presents a method that is \"simple and easy to understand.\" We address your concerns point-by-point below and believe our responses demonstrate significant research contributions.\n\n## 1. Summary of Contributions\n\n### **W1**: \"The contribution is limited. This work looks like a technical report instead of a research paper.\"\n\nWe respectfully disagree. This submission is to the **Datasets and Benchmarks area**, where benchmark and measurement contributions are central. TrafficBench enables a new class of scientific questions about local-cloud routing: when and how LLM inference workloads can be shifted from frontier cloud models to local LMs, and what this implies for accuracy, latency, energy, compute, and cost. The benchmark supports three research questions (Q1-Q3 in “Introduction”) on (i) local LM coverage, (ii) routing architectures, and (iii) efficiency tradeoffs. Our contributions are:\n\n- **A benchmark for intelligence efficiency in local-cloud routing (Datasets & Benchmarks contribution):** TrafficBench is the first benchmark to measure \"intelligence efficiency\" (accuracy per watt) across models and accelerators. Prior routing benchmarks (RouterBench, RouteLLM, RouterEval) do not report energy efficiency metrics. In contrast, TrafficBench: (1) covers 1M real-world queries and 20+ models across 9 hardware platforms, (2) provides accuracy, latency, cost, energy, compute, and memory for each model-hardware pair, and (3) includes a hardware-agnostic profiling harness (1.2M GPU-hours, nanosecond-resolution energy measurements) that the community can reuse as new models and accelerators emerge.\n- **A novel routing formulation and systematic comparison of routing paradigms:** Using TrafficBench, we propose a decoder-based binary router that decomposes routing into per-model capability decisions. This yields: (1) Macro F1 = 0.875 (chat) and 0.825 (reasoning), improving over prior approaches by up to +0.214 macro F1, and (2) a scaling study revealing clear design guidelines: embedding-based routers are preferable in data/parameter-constrained regimes (<10K samples, <1B parameters), while decoder-based routers dominate in data/parameter-rich regimes (>100K samples, >1B parameters).\n- **New empirical findings on local model capabilities and efficiency:** TrafficBench quantifies, for the first time at this scale: (1) **Local LM coverage**: Joint local coverage reaches approximately 80% of real queries (>90% in creative/social domains, approximately 68% in technical domains). (2) **Temporal trends**: From 2023 to 2025, traffic handled by models with at most 20B parameters increased by roughly 3.1 times (23.2% to approximately 71% win rate). (3) **Efficiency gains**: Our router yields 77.1% lower energy, 67.1% less compute, and 60.2% lower cost than cloud-only deployment, achieving 78.1% of perfect-oracle efficiency with <5% quality drop. (4) **Intelligence efficiency**: Accuracy per watt improves by 5.3 times (up to 9.5 times) between 2023 and 2025. These results directly answer our central questions, constituting a substantive benchmark-plus-research contribution.\n\n### **W2**: \"The proposed benchmark mainly sources from existing works.\"\n\nWe agree TrafficBench builds on established datasets (WildChat and NaturalReasoning), but many influential benchmarks do the same: **MTEB** aggregates 58 existing datasets, **SuperGPQA** extends GPQA, and **MMLU Pro** refines MMLU. The value lies in how we transform and extend sources to enable new research questions. TrafficBench's novelty includes:\n\n- **Curation and domain-specific labeling:** We curate and relabel 1M queries into a unified dataset for local-cloud routing, with new annotations mapping queries to 22 occupational domains, enabling domain-specific analyses not possible with original datasets.\n- **Comprehensive evaluation across 10+ SOTA models and 4 diverse hardware platforms:** We evaluate 15 models including August 2025 releases (Qwen3, GPT-OSS, Gemma-3, Granite) on 4 diverse platforms (NVIDIA A100/H200, AMD MI300X, Apple M4 Max), with full efficiency profiling, enabling realistic local-vs-cloud analysis impossible with prior benchmarks. \n- **Comprehensive Efficiency Metrics:** We pair each (query, model, hardware), evaluation with detailed hardware telemetry (latency, energy, memory, FLOPs) across 10 models and 4 hardware platforms. Existing routing benchmarks lack this cross-hardware, per-query efficiency data.\n- **Open-source, hardware-agnostic profiling harness:** We release a hardware-agnostic harness for reproducing and extending efficiency measurements as new models and hardware emerge, making TrafficBench an extensible benchmark."},"title":{"value":"Response to Reviewer t7Xt"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1770918068808,"tcdate":1763488436828,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission5453/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission5453/Authors"],"forum":"6Pv1dvbliS","number":4,"license":"CC BY 4.0","cdate":1763488436828,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission5453/-/Official_Comment","ICLR.cc/2026/Conference/-/Edit"],"mdate":1770918068808,"domain":"ICLR.cc/2026/Conference","replyto":"fek9cdALXU","id":"hkcyj86LgT","forumContent":{"TLDR":{"value":"TrafficBench benchmark (1M queries) shows 80.7% of LLM workloads can run on small local models (<20B), with routers achieving 77.1% energy, 67.1% compute, and 60.2% cost savings vs cloud-only deployment."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["model routing","local-cloud compute","inference workloads","inference-time compute"]},"primary_area":{"value":"datasets and benchmarks"},"abstract":{"value":"The vast majority of large language model (LLM) queries today are processed by frontier models in centralized cloud infrastructure. However, recent advances have produced small language models (≤20B parameters) that match or exceed larger models on many tasks while offering superior energy and cost efficiency. To better understand what fraction of inference workloads can be shifted away from cloud to local compute, we present TrafficBench, a comprehensive benchmark for evaluating query routing between local and cloud-deployed LLMs. TrafficBench is comprised of 1M real-world queries derived from ChatGPT user conversations and naturalistic reasoning queries, with evaluations across 10 state-of-the-art (SOTA) models, 4 hardware accelerators, and 8 performance metrics. Using TrafficBench, we address three critical questions: (1) what fraction of current inference queries can be handled by small LMs on local accelerators, (2) how effectively can modern routing architectures identify these queries, and (3) what are the downstream efficiency implications of local routing? Our analysis reveals that 80.7% of TrafficBench queries can be successfully handled by small local models, with coverage varying by domain—exceeding 90% for creative tasks but dropping just below 68% for technical fields. We start by evaluating existing SOTA embedding- and decoder-based routing approaches, finding that they do not push the Pareto frontier beyond individual local models. To enable better routing, we introduce a novel binary variation of decoder-based routing that achieves superior performance (F1 = 0.851) when we have access to large training datasets (>100K); we also show that embedding models excel in data-constrained settings (<10K). When deployed over real-world traffic distributions, our decoder-based router reduces energy by 77.1%, compute by 67.1%, and cost by 60.2% versus cloud-only deployment, while maintaining comparable task accuracy. Our longitudinal analysis from 2023-2025 shows a 9.5× improvement in intelligence efficiency (accuracy per watt), with the fraction of locally-serviceable queries increasing from 23.2% to 80.7%, suggesting significant efficiency gains from better routing systems. We release TrafficBench along with a hardware-agnostic profiling harness for measuring model efficiency metrics (e.g., energy utilization), enabling reproducible benchmarking and supporting new research as models and accelerators emerge."},"_bibtex":{"value":"@misc{\nsaad-falcon2026support,\ntitle={Support Your Local {LM}s: Redistributing {LM} Traffic from Cloud to Edge with TrafficBench},\nauthor={Jon Saad-Falcon and Avanika Narayan and Hakki Orhun Akengin and J. Wes Griffin and Herumb Shandilya and Adrian Gamarra Lafuente and Rebecca Joseph and Etash Kumar Guha and Shlok Natarajan and Shang Zhu and Ben Athiwaratkun and Azalia Mirhoseini and Christopher Re},\nyear={2026},\nurl={https://openreview.net/forum?id=6Pv1dvbliS}\n}"},"title":{"value":"Support Your Local LMs: Redistributing LM Traffic from Cloud to Edge with TrafficBench"},"pdf":{"value":"/pdf/3e95bb3c06269103ac4463d18e98f625a3913022.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"saadfalcon|support_your_local_lms_redistributing_lm_traffic_from_cloud_to_edge_with_trafficbench"},"authorids":{"value":["~Jon_Saad-Falcon1","~Avanika_Narayan1","~Hakki_Orhun_Akengin1","~J._Wes_Griffin1","~Herumb_Shandilya1","~Adrian_Gamarra_Lafuente1","~Rebecca_Joseph1","~Etash_Kumar_Guha1","~Shlok_Natarajan1","~Shang_Zhu1","~Ben_Athiwaratkun1","~Azalia_Mirhoseini3","~Christopher_Re1"]},"authors":{"value":["Jon Saad-Falcon","Avanika Narayan","Hakki Orhun Akengin","J. Wes Griffin","Herumb Shandilya","Adrian Gamarra Lafuente","Rebecca Joseph","Etash Kumar Guha","Shlok Natarajan","Shang Zhu","Ben Athiwaratkun","Azalia Mirhoseini","Christopher Re"]}},"version":2},{"content":{"comment":{"value":"**Q3:** The contributions highlight logical inference and general problem solving, but experiments focus on math-only datasets. The conclusion acknowledges this limitation; however, the earlier claims of breadth may mislead.\n\n**A3:** We acknowledge that the current benchmarks are math-centric and have tempered the language in the abstract and conclusion to reflect this scope. \n\nFirst, we have added a new experiment on CommonsenseQA a widely used non-mathematical benchmark covering everyday commonsense inference. We evaluated the same trained model without any additional fine-tuning. As reported in Section 5.2, the model achieves 74.9% accuracy compared to 73.1% for DeGRPO while reducing average token usage by approximately 2× (152 vs 312 tokens). This experiment demonstrates that MAGO's multi-objective adaptive optimization transfers effectively beyond mathematical reasoning.\n\n- We have updated the paper's Abstract (at lines 034 - 036) and added Subsection 5.2 Cross-domain Generalization (at lines 459 - 463 and 476 - 485) in the paper.\n\nSecond, we further validate MAGO on MedQA-USMLE, a medical question-answering benchmark. MAGO achieves 53.2% accuracy with 2.0× efficiency improvement over Think-Only baseline, demonstrating effective transfer to safety-critical domains. This finding is particularly significant for safety-critical applications where computational efficiency must be balanced with reasoning quality\n\n- These details have been added to the Supplementary Material in Appendix A.18 Cross-Domain Generalization on Medical Reasoning (at lines 1836 - 1864).\n\n---\n\n**Q4:** Overall, I think the most important thing is the proposed method still lack of the novelty and the performance is mainly tested on math problem which increases the concerns that whether the proposed method can be generalized to other domains.\n\n**A4:** We respectfully address this concern by clarifying the specific problems we identified and how we address them.\n\nCurrent hybrid reasoning methods (Fang et al., 2025; Shao et al., 2024) rely on fixed hyperparameters that fail to adapt across varying task complexities. Our motivation in Section 2 demonstrates that optimal weight values differ by nearly two orders of magnitude across datasets (0.001 for GSM8K vs. 0.01 for AIME), and exhaustive hyperparameter search is computationally prohibitive. Additionally, when objectives exhibit correlations, traditional single-weight scalarization constrains optimization to narrow cone-shaped regions (Song et al., 2024; Albeaik et al., 2024), preventing exploration of potentially superior solutions.\n\nTo address these issues, we reformulate hybrid reasoning as optimizing three competing objectives, accuracy, efficiency, and decision calibration, simultaneously. To our knowledge, this is the first work to introduce decision calibration as an explicit optimization target in hybrid reasoning, ensuring that mode selection aligns with the model's actual problem-solving capabilities. We maintain a diverse set of non-dominated weight configurations and dynamically select optimal weights using a correlation-adaptive scoring function, which addresses the cone entrapment problem. Unlike router-based approaches requiring external models (Ong et al., 2024), MAGO embeds adaptive mode selection directly into the trained policy with zero additional inference overhead.\n\nRegarding generalization, the CommonsenseQA results in Section 5.2 demonstrate that MAGO transfers effectively to non-mathematical reasoning without additional fine-tuning, achieving both accuracy improvements and efficiency gains.\n\n- We have updated the paper's Abstract (at lines 034 - 036) and added Subsection 5.2 Cross-domain Generalization (at lines 459 - 463 and 476 - 485) in the paper.\n\nSecond, we further validate MAGO on MedQA-USMLE, a medical question-answering benchmark. MAGO achieves 53.2% accuracy with 2.0× efficiency improvement over Think-Only baseline, demonstrating effective transfer to safety-critical domains. This finding is particularly significant for safety-critical applications where computational efficiency must be balanced with reasoning quality\n- These details have been added to the Supplementary Material in Appendix A.18 Cross-Domain Generalization on Medical Reasoning (at lines 1836-1864).\n\n## References\n- Fang, T., et al. (2025). Thinkless: LLMs Can Learn When to Think. arXiv preprint arXiv:2505.13379.\n- Shao, Z., et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv preprint arXiv:2402.03300.\n- Song, Y., et al. (2024). Optimal scalarization for multi-objective reinforcement learning. In Proceedings of the International Conference on Machine Learning (ICML).\n- Albeaik, S., et al. (2024). Pareto optimization for active learning under class imbalance. In Proceedings of the AAAI Conference on Artificial Intelligence.\n- Ong, C., et al. (2024). RouteLLM: Learning to Route LLMs with Preference Data. arXiv preprint arXiv:2406.18665."},"title":{"value":"MAGO: Beyond Fixed Hyperparameters with Multi-Objective Pareto Optimization for Hybrid LLM Reasoning"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764688722258,"tcdate":1763883072431,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7761/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission7761/Authors"],"forum":"i8vZvBFNJg","number":11,"license":"CC BY 4.0","cdate":1763883072431,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7761/-/Official_Comment"],"mdate":1764688722258,"domain":"ICLR.cc/2026/Conference","replyto":"BshWqCpSFD","id":"bZTuCUpKah","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Multi-objective optimization","Pareto optimization","Large language models","Hybrid reasoning","Chain-of-thought reasoning","Reinforcement learning"]},"supplementary_material":{"value":"/attachment/e5cc4fc1b865c508f8666486a55bae0648952c0a.pdf"},"primary_area":{"value":"reinforcement learning"},"abstract":{"value":"Large language models (LLMs) with advanced step-by-step reasoning capabilities have achieved remarkable performance in complex problem-solving through chain-of-thought (CoT) reasoning. However, uniformly applying elaborate reasoning to all queries creates substantial computational inefficiency, as many problems can be solved directly without extended reasoning chains. Current hybrid reasoning approaches rely on static hyperparameters and heuristic single-objective optimization, leading to suboptimal trade-offs and poor adaptation to varying task complexities. To address these limitations, we propose a multi-objective adaptive generation optimization (MAGO) framework, which integrates multi-objective optimization with dynamic adaptive weighting into hybrid reasoning. MAGO optimizes three competing objectives simultaneously: accuracy (maintaining solution correctness), efficiency (minimizing computational costs through appropriate mode selection), and calibration (ensuring mode selection aligns with model capabilities). The framework employs Pareto frontier maintenance with correlation-aware optimization to automatically explore the full trade-off space, avoiding the spatial constraints that limit fixed-weight approaches to narrow cone-shaped regions of the objective space. Unlike existing methods requiring manual hyperparameter tuning, MAGO's Pareto optimization dynamically adapts weights based on task complexity and training progress, achieving principled and adaptive decision-making across varying problem complexities. Comprehensive evaluation on mathematical reasoning benchmarks including AIME, Minerva Algebra, MATH-500, and GSM-8K shows $2.2\\times$ to $3\\times$ token-efficiency gains and relative accuracy improvements of $0.6\\%$ to $9.4\\%$ over heuristic baselines, while remaining competitive with the strongest task-specific models. Additional experiments on CommonsenseQA and MedQA further confirm the framework's generalizability beyond mathematics, achieving $1$ to $2\\%$ higher accuracy and approximately $2\\times$ efficiency improvement without additional fine-tuning."},"_bibtex":{"value":"@inproceedings{\nding2026mago,\ntitle={{MAGO}: Beyond Fixed Hyperparameters with Multi-Objective Pareto Optimization for Hybrid {LLM} Reasoning},\nauthor={Hongcheng Ding and Xuanze Zhao and Ruiting Deng and Shamsul Nahar Abdullah and Deshinta Arrova Dewi and Qingyu Liu},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=i8vZvBFNJg}\n}"},"title":{"value":"MAGO: Beyond Fixed Hyperparameters with Multi-Objective Pareto Optimization for Hybrid LLM Reasoning"},"pdf":{"value":"/pdf/c52228bf90679932f80d6962c056c6179c54af47.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"ding|mago_beyond_fixed_hyperparameters_with_multiobjective_pareto_optimization_for_hybrid_llm_reasoning"},"authorids":{"value":["~Hongcheng_Ding1","~Xuanze_Zhao1","~Ruiting_Deng1","~Shamsul_Nahar_Abdullah1","~Deshinta_Arrova_Dewi1","~Qingyu_Liu6"]},"authors":{"value":["Hongcheng Ding","Xuanze Zhao","Ruiting Deng","Shamsul Nahar Abdullah","Deshinta Arrova Dewi","Qingyu Liu"]}},"version":2},{"content":{"comment":{"value":"**Dear AC,**\n\nThank you for accepting our paper as a re-assigned submission. We truly appreciate the additional workload this may have caused. We are also grateful for the reviewers’ thoughtful and constructive feedback.\n\n**We appreciate that the reviewers acknowledged the contributions of our work:**\n\n- **Reviewer JRcd** highlighted that the problem formulation is clear and that our **\"idea will be of interest to the community\".**\n- **Reviewer xhvr** praised the \"originality\" of shifting from centralized to distributed routing and noted the \"rigorous\" experimental design.\n- **Reviewer qdad** recognized that DiSRouter achieves a utility score \"significantly superior\" to baselines and appreciated its \"plug-and-play\" modularity.\n- **Reviewer dfTT** acknowledged the \"concrete technical contribution\" of our Self-Awareness Training pipeline.\n\n---\n### **Summary of Key Concerns and Our Responses**\nWe consolidated the feedback into four core questions and present our responses below.\n\n**Core Concern A: Cross-Architecture Generalization (Homogeneity)**\n\n*(Raised by Reviewers xhvr [W1], dfTT [W2])*\n\n**Reviewer Comment:** Concerns were raised that our experiments relied solely on the Qwen series, questioning applicability to heterogeneous agent pools.\n\n**Our Response:** We conducted **new cross-architecture experiments** (as presented in Appendix A.10 of the revised paper) by integrating `Gemma3-1B-Instruct` and `Phi4-mini-Instruct` into the routing chain. The results confirm that our Self-Awareness Training is **architecture-agnostic**. The heterogeneous system (Gemma/Phi/Qwen) maintained high scenario adaptability and superior utility, demonstrating that DiSRouter generalizes effectively to diverse model families.\n\n---\n**Core Concern B: Routing Overhead and Utility Metric Formulation**\n\n*(Raised by Reviewers dfTT [W1], qdad [Q2], JRcd [W2])*\n\n**Reviewer Comment:** Reviewer dfTT argued that our Utility metric was flawed for ignoring cumulative latency (routing overhead) in the cascade. Reviewers qdad and JRcd are also concerned about the overhead of multiple rejection responses.\n\n**Our Response:** We clarified that the omission of the routing overhead term in the original formula was a **deliberate simplification for mathematical conciseness**, predicated on our analysis that this term is statistically negligible as presented in Appendix A.1. To validate this:\n\n1. **Empirical Verification:** We measured that the cumulative \"routing cost\" (time spent rejecting) constitutes **<5%** of the inference cost even in worst-case paths.\n2. **Adjusted Metric:** We re-calculated our results using a stricter Utility metric that explicitly subtracts routing overhead ($Cost_{routing}$). As shown in Table R2 (Response to dfTT), the utility drop is marginal (~1.5%) and DiSRouter retains its superiority over baselines.\n3. **Time-to-First-Token:** We demonstrated that while DiSRouter incurs slightly higher latency than predictive routers (e.g.,s RouteLLM) which use lightweight classifier, it is **significantly faster** than generative baselines that rely on full model outputs or verification steps (e.g., FrugalGPT, AutoMix).\n\n---\n**Core Concern C: Training Efficiency and Overhead**\n\n*(Raised by Reviewers xhvr [W2], qdad [W1])*\n\n**Reviewer Comment:** Reviewers expressed concern that the two-stage training approach (SFT+RL) is computationally heavy compared to training a lightweight centralized router.\n\n**Our Response:** We argue that the training overhead is justified and optional:\n\n1. **Optionality (GPT Series Experiments):** We demonstrated that high-capability models (e.g., GPT-4o-mini) possess **intrinsic self-awareness** and do not require training. We constructed a **\"Training-Free DiSRouter\"** using GPT models, which achieved comparable utility on in-domain tasks and much higher utility on OOD tasks without any gradient updates.\n2. **Amortized Cost:** Unlike centralized routers that require full retraining whenever the model pool and tasks change, DiSRouter is decoupled. Training is a one-time investment per agent; once trained, an agent is a modular \"plug-and-play\" component.\n---\n**Core Concern D: Generalization to Open-Ended Tasks**\n\n*(Raised by Reviewers dfTT [W3], qdad [W2])*\n\n**Reviewer Comment:** Questions regarding whether the binary \"reject/answer\" training applies to open-ended tasks like summarization where correctness is non-binary.\n\n**Our Response:** We extended our framework to **Text Summarization** (SAMSum, XSum) in Appendix A.11. We adapted the reward function to use continuous metrics (BERTScore) and percentile-based thresholds. The results show that DiSRouter successfully distinguishes high-quality generations from low-quality ones, proving the method is **transferable to open-ended generation tasks.**"},"title":{"value":"Summary of Reviews & New Experiments: Addressing All Reviewer Concerns with Definitive Evidence"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764670644614,"tcdate":1764670644614,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7635/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission7635/Authors"],"forum":"KDcwXKr0NU","number":16,"license":"CC BY 4.0","cdate":1764670644614,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7635/-/Official_Comment"],"mdate":1764670644614,"domain":"ICLR.cc/2026/Conference","replyto":"KDcwXKr0NU","id":"oiG3tNvUb3","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"TLDR":{"value":"We introduce DiSRouter, a distributed system where LLMs leverage \"self-awareness\" to route queries among themselves, outperforming conventional centralized routers."},"keywords":{"value":["query routing","model selection","distributed system","self-awareness of LLM"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"The proliferation of Large Language Models (LLMs) has created a diverse ecosystem of models with highly varying performance and costs, necessitating effective query routing to balance performance and expense. Current routing systems often rely on a centralized external router trained on a fixed set of LLMs, making them inflexible and prone to poor performance since the small router can not fully understand the knowledge boundaries of different LLMs. We introduce DiSRouter (Distributed Self-Router), a novel paradigm that shifts from centralized control to distributed routing. In DiSRouter, a query traverses a network of LLM agents, each independently deciding whether to answer or route to other agents based on its own self-awareness—its ability to judge its competence. This distributed design offers superior flexibility, scalability, and generalizability. To enable this, we propose a two-stage Self-Awareness Training pipeline that enhances each LLM's self-awareness. Extensive experiments demonstrate that DiSRouter significantly outperforms existing routing methods in utility across various scenarios, effectively distinguishes between easy and hard queries, and shows strong generalization to out-of-domain tasks. Our work validates that leveraging an LLM's intrinsic self-awareness is more effective than external assessment, paving the way for more modular and efficient multi-agent systems."},"_bibtex":{"value":"@inproceedings{\nzheng2026disrouter,\ntitle={Di{SR}outer: Distributed Self-Routing for {LLM} Selections},\nauthor={Hang Zheng and Hongshen Xu and Yongkai.lin and Shuai Fan and Lu Chen and Kai Yu},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=KDcwXKr0NU}\n}"},"title":{"value":"DiSRouter: Distributed Self-Routing for LLM Selections"},"pdf":{"value":"/pdf/0190f98d94890b700d03febdee678e22a2a5e380.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"zheng|disrouter_distributed_selfrouting_for_llm_selections"},"authorids":{"value":["~Hang_Zheng3","~Hongshen_Xu1","~Yongkai.lin1","~Shuai_Fan1","~Lu_Chen3","~Kai_Yu3"]},"authors":{"value":["Hang Zheng","Hongshen Xu","Yongkai.lin","Shuai Fan","Lu Chen","Kai Yu"]}},"version":2},{"content":{"comment":{"value":"Regarding the parameter K, the curves are obtained by varying K, which controls the trade-off between the proportion of small-model inference and overall accuracy. In practice, K can be selected using the validation curves to achieve a desired ratio–accuracy balance. For all values of K, our cascading method consistently and significantly outperforms the baseline, demonstrating its robustness.\n\n**Q5: “Please clarify that if involving more than two models helps.”**\n\n**A5:** We were not sure whether the suggestion referred to involving more than two models in **model cascading** or in **data cleaning**, so we conducted experiments for both settings.\n\n- **Model cascading (more than two models).**  \n  Numerous extensions beyond a two-model configuration are possible. As an initial exploration, we evaluate a simple three-model variant: a small model (Coat_small) paired with two large models (Eva-in22k and Eva-m38m) of comparable scale and accuracy.  \n  The small model first processes inputs following the standard cascade; unresolved samples are then routed to one of the two large models based on confidence—the batch is sent to the large model with the higher predicted confidence.  \n  With this three-model cascading scheme, we achieve **89.1% accuracy**, while the small model (only 7% of the parameters of the large model) handles **over 60%** of the samples. This performance is comparable to Eva-in22k alone (**89.7%**), showing that multi-model cascading can retain strong accuracy while improving efficiency.\n\n- **Data cleaning (more than two models).**  \n  Introducing more than three models tends to **reduce detection rate** while **slightly increasing filtering accuracy**. This trade-off can largely be achieved by the confidence-based filtering mechanism described in our method (see Figure 4 in the main paper), suggesting that adding extra models provides limited additional benefit.  \n  Empirically, using **three models** yields a filtering rate of **7.9%** with **86.4%** accuracy.  \n  For comparison, using **two models** with linear interpolation achieves **85.3%** accuracy at the same filtering rate, indicating only a **modest improvement** when more models are added.\n\n**Q6: “Does self-reported confidence work in language tasks?”**\n\n**A6:** This is an interesting question but is beyond the scope of the present study.\n\n- **General applicability.**  \n  Our framework is model-agnostic and can naturally incorporate self-reported confidence for language tasks, as it only requires a confidence score (regardless of whether it is model-generated or self-reported).\n\n- **Scope considerations.**  \n  To ensure a clean and controlled comparison with prior work, we intentionally focus on probability-based confidence and do not explore self-reported confidence in this manuscript.\n\nWe agree that this is a promising direction and plan to investigate it in future work.\n\n**Q7: “How sensitive are the cascading and cleaning results to the number of bins or the top-K selection thresholds?”**\n\n**A7:** We added experiments analyzing how the hyperparameter **number of bins** influences the cascading performance. The results show that the method remains **stable across a wide range of bin choices (5–100)**, as summarized in Table 10.\n\n**[Table 10] Effect of number of bins on model cascading.**\n\n| Dataset | 5      | 15     | 50     | 100    |\n|--------|--------|--------|--------|--------|\n| mmlu   | 0.6917 | 0.7013 | 0.7044 | 0.7035 |\n| gsm8k  | 0.7378 | 0.7496 | 0.7547 | 0.7551 |\n\nRegarding the parameter **K**, as discussed in **A4**, the cascading curves are generated by varying *K*, which determines the trade-off between small-model usage and overall accuracy. In practice, *K* can be selected using validation curves to meet a desired ratio–accuracy balance.\n\n**Across all values of K**, the proposed cascading method **consistently and significantly outperforms the baseline**, demonstrating its **robustness**.\n\n[1] RouteLLM: Learning to Route LLMs with Preference Data. ICLR 2025\n\n[2] Confident Learning: Estimating Uncertainty in Dataset Labels. JAIR\n\n[3] Efficient Data Driven Mixture-of-Expert Extraction from Trained Networks. CVPR 2025\n\n[4] Do Deep Generative Models Know What They Don't Know? ICLR 2019\n\n[5] A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. ACM \nTransactions on Information Systems Jan 2024\n\n[6] Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers. ICML 2001\n\n[7] Why Normalizing Flows Fail to Detect Out-of-Distribution Data? NeurIPS 2020\n\n[8] WAIC, but Why? Generative Ensembles for Robust Anomaly Detection\n\n[9] Input complexity and out-of-distribution detection with likelihood-based generative models. ICLR 2020"},"title":{"value":"Author Response to Reviewer mxpA (4/4)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764381981748,"tcdate":1764381775000,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission8659/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission8659/Authors"],"forum":"imzDS4N8CW","number":9,"license":"CC BY 4.0","cdate":1764381775000,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission8659/-/Official_Comment"],"mdate":1764381981748,"domain":"ICLR.cc/2026/Conference","replyto":"G2x3OMyPHc","id":"tEuynNcdSu","forumContent":{"TLDR":{"value":"a general framework that enables a model to know when it does not know and to make use of this property"},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Model Calibration","Model Cascades","Data Cleaning"]},"primary_area":{"value":"alignment, fairness, safety, privacy, and societal considerations"},"abstract":{"value":"When a model knows when it does not know, many possibilities emerge. The first question is how to enable a model to recognize that it does not know. A promising approach is to use confidence, computed from the model’s internal signals, to reflect its ignorance. Prior work in specific domains has shown that calibration can provide reliable confidence estimates. In this work, we propose a simple, effective, and universal training-free method that applies to both vision and language models, performing model calibration, cascading, and data cleaning to better exploit a model's ability to recognize when it does not know. We first highlight two key empirical observations: higher confidence corresponds to higher accuracy within a single model, and models calibrated on the validation set remain calibrated on a held-out test set. These findings empirically establish the reliability and comparability of calibrated confidence. Building on this, we introduce two applications: 1. Model cascading with calibrated advantage routing and 2. Data cleaning based on mixture of experts. Using the routing signal derived from the comparability of calibrated confidences, we cascade large and small models to improve efficiency with almost no compromise in accuracy, and we further cascade two large state-of-the-art models to achieve performance beyond either model alone. Leveraging multiple experts and their calibrated confidences, we design a simple yet effective data-cleaning method that balances precision and detection rate to identify mislabeled samples in ImageNet and Massive Multitask Language Understanding (MMLU) datasets. Our results demonstrate that enabling models to recognize when they do not know is a practical step toward more efficient, reliable, and trustworthy AI."},"_bibtex":{"value":"@misc{\nhao2026when,\ntitle={When Models Know When They Do Not Know: Calibration, Cascading and Cleaning},\nauthor={Chenjie Hao and Weyl Lu and Yuko Ishiwaka and ZENGYI LI and Weier Wan and Yubei Chen},\nyear={2026},\nurl={https://openreview.net/forum?id=imzDS4N8CW}\n}"},"title":{"value":"When Models Know When They Do Not Know: Calibration, Cascading and Cleaning"},"pdf":{"value":"/pdf/15f153d32eedccd6d5def2e3c98e8ad6d40eaffb.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"hao|when_models_know_when_they_do_not_know_calibration_cascading_and_cleaning"},"authorids":{"value":["~Chenjie_Hao1","~Weyl_Lu1","~Yuko_Ishiwaka1","~ZENGYI_LI1","~Weier_Wan1","~Yubei_Chen1"]},"authors":{"value":["Chenjie Hao","Weyl Lu","Yuko Ishiwaka","ZENGYI LI","Weier Wan","Yubei Chen"]}},"version":2},{"content":{"comment":{"value":"We thank the reviewer for your detailed review and recognition. Here, we address your concerns and questions.\n\n---\n\n\n## **W1. Cost normalization bias & Beta sensitivity ------ We provide dynamic normalization & beta tuning**\n\n> W1. The log2 cost normalization with fixed (c_min, c_max) and beta = 0.1 may bias rankings toward certain price bands; no sensitivity analysis shown in Sec. 5.\n\nIn our framework, $c_{max}$ and $c_{min}$ are not intended to be static global constants; instead, they are **tunable for flexibility**. To prevent the most expensive router from being normalized to zero, for now, we set $c_{max}$ and $c_{min}$ to the prices of the most expensive and cheapest models on the market. However, whenever a model's actual cost really falls below $c_{min}$ or exceeds $c_{max}$, the only thing we need to do is to adjust the $c_{min}$ and $c_{max}$, and re-calculate the ArenaScore so that all normalized costs remain within $(0,1)$. Note that this **doesn't require re-running the experiment**.\n\nThe parameter $\\beta$ is **also adjustable** to modulate how much weight is assigned to cost compared to accuracy. Specifically, we made the $\\beta$ tunable on our leaderboard for users (will have a website open to users). Here we give three example settings:\n\n* Example 1: $\\beta$ = 0.01, the ratio of Accuracy to Cost is 100, accuracy-dominated ranking for users who do not consider cost to be a key constraint.\n* Example 2: $\\beta$ = 0.1, the ratio of Accuracy to Cost is 10, trade-off balanced ranking for users who primarily care about accuracy but still operate under budget constraints.\n* Example 3: $\\beta$ = 1, the ratio of Accuracy to Cost is 1, equally weighted ranking for users who have a limited budget.\n\n---\n\n### Table 1: Leaderboard by different values of Beta.\n|Router|Rank & ArenaScore @ β=0.01|Rank & ArenaScore @ β=0.1|Rank & ArenaScore @ β=1|Accuracy|Norm.Cost|\n|-----------------|------------------------------|-----------------------------|---------------------------|--------|---------|\n|GPT-5|1 (0.7282)|5 (0.6272)|12 (0.3688)|0.7428|0.2453|\n|Azure-Router|2 (0.6780)|2 (0.6640)|2 (0.6010)|0.6798|0.5386|\n|MIRT-BERT|3 (0.6731)|1 (0.6729)|1 (0.6718)|0.6731|0.6705|\n|CARROT|4 (0.6682)|3 (0.6386)|6 (0.5219)|0.6720|0.4266|\n|vLLM-SR|5 (0.6633)|4 (0.6385)|5 (0.5368)|0.6665|0.4494|\n|NotDiamond|6 (0.6565)|8 (0.5935)|11 (0.3998)|0.6651|0.2858|\n|RouteLLM|7 (0.6175)|9 (0.5800)|10 (0.4440)|0.6224|0.3451|\n|NIRT-BERT|8 (0.6151)|6 (0.6083)|4 (0.5764)|0.6159|0.5416|\n|MLP|9 (0.6143)|10 (0.5780)|9 (0.4449)|0.6191|0.3472|\n|GraphRouter|10 (0.6070)|7 (0.6054)|3 (0.5976)|0.6072|0.5884|\n|KNN|11 (0.5867)|11 (0.5577)|8 (0.4464)|0.5905|0.3588|\n|RouterDC|12 (0.3362)|12 (0.3522)|7 (0.4627)|0.3344|0.7507|\n\nThanks for your suggestion again! We have revised the paper to incorporate the changes in Appendix H.\n\n---\n\n## **W2. Lack of human validation ------ LLM-as-a-judge only for guidance; difficulty is defined empirically; human study provided.**\n\n>W2. LLM-as-judge labelling (DeepSeek-V3.1) lacks human validation studies or inter-rater checks.\n\nWe agree that the role of LLM-as-judge should be clearly specified, and we have revised Section 3 accordingly. Bloom's Taxonomy in our benchmark is **used only to characterise the cognitive skill** of each query and to guide stratified sampling; it is **not used as the definition of difficulty** in our analyses. **Difficulty is defined empirically**: for each query, we compute the number of models (out of 42 models with various sizes and capabilities used by existing routers, details in Appendix C.4) that answer it correctly and sort queries by this empirical difficulty. We then group them into three bands—hard (≤4/42 models correct), medium (5–19/42), and easy (>19/42)—yielding 23.4%, 29.1%, and 47.5% of the dataset, respectively, with smooth difficulty curves overall and per domain (Fig. 3–4).\n\nWe also quantified the reliability of LLMs as judges via a **human validation study** on 5.4% of the data (450 queries), stratified by category and Bloom level. Eighteen volunteers (undergraduate and graduate students) labelled Bloom levels under the same protocol as the LLM. We observe 54.9% exact agreement and 76.7% agreement within ±1 Bloom level with the human majority vote, indicating that most disagreements are between adjacent levels in an inherently subjective taxonomy. Since Bloom labels only guide dataset construction, while all router evaluations rely on empirical accuracy across 42 LLMs, any bias from the LLM-as-judge has **limited impact** on our reported results."},"title":{"value":"Thank you! (1/2)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764362126891,"tcdate":1763963259043,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission13871/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission13871/Authors"],"forum":"9HsaIi4ngF","number":1,"license":"CC BY 4.0","cdate":1763963259043,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission13871/-/Official_Comment"],"mdate":1764362126891,"domain":"ICLR.cc/2026/Conference","replyto":"OLfcvgNCZH","id":"1CQPLAY716","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"TLDR":{"value":"We proposed a comprehensive LLM router evaluation framework, with a new dataset, metrics, and ranking system."},"keywords":{"value":["LLM Router","Evaluation"]},"primary_area":{"value":"datasets and benchmarks"},"abstract":{"value":"Today's LLM ecosystem comprises a wide spectrum of models that differ in size, capability, and cost. No single model is optimal for all scenarios; hence, LLM routers have become essential for selecting the most appropriate model under varying circumstances.\nHowever, the rapid emergence of various routers has led to fragmented evaluation practices and inconsistent metrics, making it difficult to systematically assess progress in this space. To address this problem, we need a comprehensive router comparison and a standardized leaderboard, similar to those available for models. In this work, we introduce RouterArena, the first open platform enabling comprehensive comparison of LLM routers. RouterArena has (1) a principally constructed dataset with broad knowledge domain coverage, (2) distinguishable difficulty levels for each domain, (3) an extensive list of evaluation metrics, and (4) an automated framework for evaluation and leaderboard updates. Leveraging this framework, we have produced the initial leaderboard with detailed metrics comparison.\nFigure1 provides a preview of the leaderboard. The complete framework and the latest router leaderboard are publicly available at https://routeworks.github.io/"},"_bibtex":{"value":"@inproceedings{\nlu2026routerarena,\ntitle={RouterArena: An Open Platform for Comprehensive Comparison of {LLM} Routers},\nauthor={Yifan Lu and Rixin Liu and Jiayi Yuan and Xingqi Cui and Shenrun Zhang and Hongyi Liu and Jiarong Xing},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=9HsaIi4ngF}\n}"},"title":{"value":"RouterArena: An Open Platform for Comprehensive Comparison of LLM Routers"},"pdf":{"value":"/pdf/03e5d4326116c387dbd69c38e194211db0736cda.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"lu|routerarena_an_open_platform_for_comprehensive_comparison_of_llm_routers"},"authorids":{"value":["~Yifan_Lu8","~Rixin_Liu1","~Jiayi_Yuan1","~Xingqi_Cui1","~Shenrun_Zhang1","~Hongyi_Liu5","~Jiarong_Xing1"]},"authors":{"value":["Yifan Lu","Rixin Liu","Jiayi Yuan","Xingqi Cui","Shenrun Zhang","Hongyi Liu","Jiarong Xing"]}},"version":2},{"content":{"comment":{"value":"## **Memory Overhead in Asynchronous Sequential Inference**\n\nWe appreciate the reviewer’s insightful comments. MoT does not require synchronous execution across experts. Only the primary expert performs cross-attention. All non-primary experts can be evaluated asynchronously and sequentially, exactly as in RouteLLM/Avengers: each auxiliary expert computes its hidden states, projects them into the shared latent space, and these projected hidden states are stored for later computation by the primary expert within its interaction layer.\n\nFor an auxiliary expert, the stored projected hidden states have size:\n\n$$\n\\text{Overhead} = Q \\cdot h \\cdot (L_{\\text{prefill}} + L_{\\text{gen}}) \\cdot \\text{bitwidth},\n$$\n\nwhere \\(Q\\) is the number of interaction layers, $h$ is hidden size, $L_{\\text{prefill}}$ and $L_{\\text{gen}}$ are prefill and generation lengths, and bitwidth = 2 bytes (fp16/bf16). Using representative values $(L_{\\text{prefill}} = 256$, $L_{\\text{gen}} = 128$, total = 384 tokens), the total overhead per auxiliary expert is:\n\n| Q | h=1024 | h=2048 |\n|---|--------|---------|\n| 2 | 1.57 MB | 3.14 MB |\n| 4 | 3.14 MB | 6.29 MB |\n| 6 | 4.71 MB | 9.43 MB |\n| 8 | 6.29 MB | 12.57 MB |\n\nEven in the heaviest configuration $(Q = 8, h = 2048\\)$, one auxiliary expert contributes only **12.6 MB** of stored projections, i.e., **<0.09%** of a 7B model’s fp16 parameter memory (≈14 GB). With $K = 3$, the total overhead is ≈25 MB.\n\nAt the stack level, a 7B model with 32 layers corresponds to ≈1.75 GB per stack (Q=8). If executed sequentially, the primary expert would only require an additional **3.12 MB** of data from the two auxiliary experts (K=3). This overhead scales linearly with sequence length; even for sequence lengths on the order of $10^4$, the memory overhead remains under 100 MB.\n\nWe agree that for long-context inference, this overhead becomes substantial—this is a current limitation of MoT that we will address in upcoming work.\n\n---\n\n## **Communication Overhead in Synchronous Multi-GPU MoT Inference**\n\nFor synchronous deployment, we place each expert on a separate GPU. We analyze the scenario where experts are located on distinct devices—the latency results reported in Table 1 follow this setting.\n\nAt each interaction layer during prefill, the primary expert must receive projected hidden states from the \\(K-1\\) auxiliary experts. This requires transferring:\n\n$$\n(K-1) \\cdot L_{\\text{prefill}} \\cdot h \\cdot \\text{bitwidth}.\n$$\n\nUsing representative values $(L_{\\text{prefill}} = 256$, $h = 2048$, bitwidth = 2 bytes), this gives a per-auxiliary-expert, per-stack overhead of **1.05 MB**. For $K = 3$, the total per-interaction-layer overhead is **2.1 MB**.\n\n**Transfer times for 2.1 MB:**\n\n| Interconnect | Transfer Volume | Transfer Time |\n|--------------|------------------|----------------|\n| PCIe 4.0     | 2.1 MB           | 0.06 ms        |\n| NVLink 3     | 2.1 MB           | 0.02 ms        |\n\nDuring generation, per-token communication is even smaller and independent of sequence length:\n\n$$\n(K-1)\\cdot h \\cdot \\text{bitwidth}.\n$$\n\nFor $h = 2048$ and fp16, this is **4 KB per expert**, or **8 KB total**.\n\nEven when accounting for PCIe and NVLink startup latencies—typically ≈1–3 µs for PCIe transactions and under 1 µs for NVLink link-level traversal [NVIDIA GPUDirect RDMA; NVIDIA DGX A100 Architecture]—the communication cost at these message sizes remains negligible and does not change the conclusions of the above analysis.\n\nCompared to both the prefill and decode latency, these overheads are negligible. This is reflected in our results in Table 1, where MoT has **essentially identical inference time to Avengers**.\n\nStandard pipelining can further hide communication latency:\n\n1. each expert computes its current stack $s$;\n2. auxiliary experts asynchronously send projected hidden states to the primary expert, which performs the interaction-layer collaboration;\n3. simultaneously, auxiliary experts proceed to stack $s+1$ and prepare the next projections.\n\nSuch system-level scheduling optimizations are promising directions for future work.  \nWe will also include a detailed breakdown of prefill vs. generation latency in the revised draft.\n\n---\n\n## **Inference Setup and GPU Memory**\n\nWe appreciate the reviewer’s question and clarify that our inference setup follows the standard multi-LLM configuration used in prior work (such as RouterDC). Each expert is hosted on its own **RTX A6000** GPU, and the router dynamically selects the top-K experts per query.\n\nWe do **not** use activation checkpointing, CPU offloading, quantization, or other memory-saving techniques. Each expert runs as a standard fp16/bf16 model on a single GPU.\n\nWe will add these details to the draft for clarity."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764095610423,"tcdate":1764050774520,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission6329/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission6329/Authors"],"forum":"x9tSyvnD8o","number":7,"license":"CC BY 4.0","cdate":1764050774520,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission6329/-/Official_Comment"],"mdate":1764095610423,"domain":"ICLR.cc/2026/Conference","replyto":"9K30upPGMn","id":"bv2PRRNZAs","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"TLDR":{"value":"Multi-LLM collaboration via latent-space interactions"},"keywords":{"value":["Large Language Models","LLM","LLM routing","LLM aggregation","LLM ensemble"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Open-source Large Language Models (LLMs) increasingly specialize by domain (e.g., math, code, general reasoning), motivating systems that leverage complementary strengths across models. Prior multi-LLM approaches either (i) route a query to one or a few experts and generate independently, (ii) aggregate outputs from each model via costly multi-turn exchanges, or (iii) fuse weights into a single model—typically requiring architectural homogeneity. We introduce Mixture of Thoughts (MoT), a simple method for latent-level collaboration among heterogeneous experts under a global routing scheme. For each query, a lightweight router selects top-$K$ experts and designates a primary expert; uniformly placed interaction layers project hidden states into a shared latent space where the primary expert performs cross-attention over its active (selected) peers. Pre-trained experts remain frozen; only the router and the lightweight interaction layers are trained with a novel joint training objective that improves both the expert selection and inter-expert collaboration. Across five in-distribution (ID) and three out-of-distribution (OOD) benchmarks, MoT surpasses the current routing and aggregation-based state-of-the-art, Avengers, by $+0.38\\%$ and $+2.92\\%$, respectively. Further, MoT significantly outperforms the best-performing single model. It achieves this with single-pass inference, runtime comparable to routing baselines, and none of the overheads of iterative aggregation. MoT offers a simple latent-space mechanism for combining heterogeneous LLMs, a practical step toward broader multi-LLM collaboration. Our code is publicly available at \\url{https://anonymous.4open.science/r/mot-5B4B/}."},"_bibtex":{"value":"@misc{\nfein-ashley2026mixture,\ntitle={Mixture of Thoughts: Learning to Aggregate What Experts Think, Not Just What They Say},\nauthor={Jacob Fein-Ashley and Dhruv Parikh and Rajgopal Kannan and Viktor Prasanna},\nyear={2026},\nurl={https://openreview.net/forum?id=x9tSyvnD8o}\n}"},"title":{"value":"Mixture of Thoughts: Learning to Aggregate What Experts Think, Not Just What They Say"},"pdf":{"value":"/pdf/a857f7bf89f375d5f31ac949d50eea649304f16b.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"feinashley|mixture_of_thoughts_learning_to_aggregate_what_experts_think_not_just_what_they_say"},"authorids":{"value":["~Jacob_Fein-Ashley1","~Dhruv_Parikh1","~Rajgopal_Kannan1","~Viktor_Prasanna1"]},"authors":{"value":["Jacob Fein-Ashley","Dhruv Parikh","Rajgopal Kannan","Viktor Prasanna"]}},"version":2},{"content":{"comment":{"value":">Q3. Training uses samples from GSM8K/MATH/CommonsenseQA, then evaluates on same benchmark test sets. Router may learn dataset artifacts rather than complexity. Missing: 1) Code generation (HumanEval, MBPP); 2)Long-context reasoning (QuALITY, NarrativeQA); Multi-turn dialogue (MT-Bench); Factual QA with retrieval (Natural Questions, TriviaQA); Multimodal tasks (ScienceQA, MMMU). \n\n**A3.** We thank the reviewer for raising this concern about overfitting to specific benchmarks and for suggesting a broader evaluation. We respond from three aspects and have added the corresponding results in the revised version (**Sec. 5.3**).\n* **Evidence that the CogER goes beyond dataset-specific artifacts.** In the main paper, CogER is trained on GSM8K, MATH, Com-QA, and MedQA, but is evaluated not only on their test sets, but also on out-of-domain math datasets such as MAWPS and College. CogER consistently improves both ID and OOD tasks under a reduced compute budget, which suggests that the learned level-selection policy generalizes across related but distribution-shifted tasks rather than memorizing idiosyncratic patterns of a single benchmark.\n* **New experiments on code, long-context, and retrieval-augmented QA.** We further evaluate CogER on three additional benchmarks that were not used for training, covering code generation, long-context reasoning, and retrieval-augmented factual QA. From Table B, CogER matches or surpasses strong baselines on MBPP (code generation, Pass@3), QuALITY (long-context multiple-choice QA), and Natural Questions (retrieval-based QA). These results indicate that the cognitive hierarchy and elastic level selection transfer to tasks with very different formats and reasoning requirements, rather than being tied to the GSM8K/MATH/Com-QA domains.\n* **Scope and limitations regarding multi-turn dialogue and multimodal tasks.** We agree that multi-turn dialogue and multimodal reasoning are highly relevant and valuable directions. In this work, however, we intentionally scope CogER to single-turn, text-only settings to cleanly study cognitive-inspired elastic reasoning for LLMs under controlled conditions and to avoid confounding factors from dialogue state tracking or visual perception. This design choice and its limitations are explicitly discussed in our “Discussions and Future Works” section, where we outline extending CogER to interactive and multi-modal settings as an important avenue for future research. \n\nTable B:  Performance of CogER and baseline models on additional benchmarks, including code generation (MBPP, Pass@3), long-context reasoning (QuALITY, accuracy), and retrieval-augmented factual QA (Natural Questions, F1-score).\n\n|Baseline|MBPP (pass@3)|QuALITY (ACC)|Natural Questions (F1)|\n|-|-|-|-|\n|Math-72B|75.16|55.77|45.52|\n|DS-R1-DQ-7B|77.30|35.21|15.41|\n|DS-R1-DQ-14B|79.88|73.33|61.14|\n|DS-R1-DQ-32B|91.22|81.22|64.45|\n|L1-MAX|38.30|25.45|2.96|\n|S1-32B|86.00|81.41|66.99|\n|ReasonFlux-32B|91.70|81.88|63.37|\n|RouteLLM|90.68|**82.97**|67.06|\n|CogER (Ours)|**91.76**|**82.97**|**67.25**|\n\n\n>Q4. Only 8K training samples total; no ablation showing this suffices. \n\n**A4.** We thank the reviewer for raising the question about whether 8K training samples are sufficient. We added a data-scale ablation to make this explicit (**see Appendix F**).\n* **Ablation over training set size.** We vary the number of training samples while keeping the data mixture and all other settings fixed. As shown in Table C, CogER already achieves strong performance with as few as 4K samples, and that performance is relatively stable across different data scales. The 8K configuration attains the best accuracy on both ID and OOD benchmarks, which supports our choice in the main paper.\n* **Sample efficiency and supervision signals.** Each training instance provides rich supervision for the policy: a binary correctness signal from the final answer and the hierarchical preference encoded by $L_{\\min}(S)$ in $\\mathcal{R}_{\\text{hierarchy}}$. The policy only needs to choose among four fixed reasoning modes rather than learn the underlying tasks from scratch, so the sample complexity is modest. As shown in Table C, performance already plateaus around 8K examples, and the small fluctuations at 10K are within the variance one would expect from our limited training budget and random initialization.\n\nTable C: Effect of training set size on performance of CogER (EM, %).\n|Dataset size|4K|6K|8K|10K|\n|-|-|-|-|-|\n|ID|86.48|86.51|**89.28**|86.48|\n|OOD|92.73|92.75|**93.56**|92.69|"},"title":{"value":"Response to Reviewer opSX (Part 2)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763797563266,"tcdate":1763796302397,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission705/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission705/Authors"],"forum":"z9HUMon2Br","number":9,"license":"CC BY 4.0","cdate":1763796302397,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission705/-/Official_Comment"],"mdate":1763797563266,"domain":"ICLR.cc/2026/Conference","replyto":"M93xVmPWSg","id":"cWirQzPO3k","forumContent":{"TLDR":{"value":"We propose the  Cognitive-Inspired Elastic Reasoning framework for efficient scaling of language model reasoning, which dynamically selects the most appropriate processing mode for each query."},"venue":{"value":"ICLR 2026 Conference Desk Rejected Submission"},"keywords":{"value":["Large language models; Reinforcement Learning; Markov decision process"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Large language models (LLMs) have demonstrated impressive performance across various language tasks. However, existing LLM reasoning strategies mainly rely on the LLM itself with fast or slow mode (like o1 thinking) and thus struggle to balance reasoning efficiency and accuracy across queries of varying difficulties. In this paper, we propose Cognitive-Inspired Elastic Reasoning (CogER), a framework inspired by human hierarchical reasoning that dynamically selects the most suitable reasoning strategy for each query. Specifically, CogER first assesses the complexity of incoming queries and assigns them to one of several predefined levels, each corresponding to a tailored processing strategy, thereby addressing the challenge of unobservable query difficulty. To achieve automatic strategy selection, we model the process as a Markov Decision Process and train a CogER-Agent using reinforcement learning. The agent is guided by a reward function that balances solution quality and computational cost, ensuring resource-efficient reasoning. Moreover, for queries requiring external tools, we introduce Cognitive Tool-Assisted Reasoning, which enables the LLM to autonomously invoke external tools within its chain-of-thought. Extensive experiments demonstrate that CogER outperforms state‑of‑the‑art Test‑Time scaling methods, achieving at least a 13% relative improvement in average exact match on In‑Domain tasks and an 8% relative gain on Out‑of‑Domain tasks."},"_bibtex":{"value":"@misc{\nanonymous2026beyond,\ntitle={Beyond Fast and Slow: Cognitive-Inspired Elastic Reasoning for Large Language Models},\nauthor={Anonymous},\nyear={2026},\nurl={https://openreview.net/forum?id=z9HUMon2Br}\n}"},"title":{"value":"Beyond Fast and Slow: Cognitive-Inspired Elastic Reasoning for Large Language Models"},"pdf":{"value":"/pdf/1252c0b740227fa4d6c59edd0735bd20fd07cbe8.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Desk_Rejected_Submission"},"paperhash":{"value":"hu|beyond_fast_and_slow_cognitiveinspired_elastic_reasoning_for_large_language_models"},"authorids":{"value":["~Jinwu_Hu1","~Dongjin_Yang1","~LangYu_Bian1","~Zhiquan_Wen1","~Yufeng_Wang5","~Yaofo_Chen1","~Bin_Xiao4","~Yuanqing_Li2","~Mingkui_Tan2"]},"authors":{"value":["Jinwu Hu","Dongjin Yang","LangYu Bian","Zhiquan Wen","Yufeng Wang","Yaofo Chen","Bin Xiao","Yuanqing Li","Mingkui Tan"]}},"version":2},{"content":{"comment":{"value":"We sincerely thank you for the careful reading and candid feedback. Your review specifically questions whether the benchmark is “a recombination of existing resources” and whether the routing component is sufficiently useful. We address these two points directly below.\n\n# 1. Why R2Eval is not a simple recombination of prior resources\n\nWe agree that some questions are sourced from prior high-quality datasets. However, R2Eval is not formed by simply concatenating those resources. What is new is the benchmark protocol built on top of them:\n\n* it is the first benchmark explicitly targeting **LRM-as-a-Judge**, rather than only verdict-level LLM judging;\n* it provides paired **Think / Non-Think** variants for every item, enabling controlled study of when explicit reasoning helps, misleads, or becomes unnecessary;\n* it defines a unified interface in which LLM judges return verdict-only outputs, while LRM judges return `{verdict, confidence, rationale, thinking}`, making thinking–verdict mismatch analysis possible as a benchmarked object rather than an anecdotal observation;\n* and it standardizes all final pairwise responses ((A,B)) to a shared dual-source protocol using GPT-4o and Claude 3.5, instead of inheriting heterogeneous prompt–response conventions directly from different benchmarks.\n\nSo while some source questions are borrowed, the evaluated object is new: not prompts alone, but judge behavior under explicit reasoning traces, controlled response pairs, and cost-aware deployment conditions. That is substantially different from simple dataset recombination.\n\n# 2. Comparison with prior work\n\nWe agree that difficulty-aware design alone is not new, and we do not intend to claim it as an isolated novelty. Our claim is about the **new combination of benchmark target, protocol, and deployment formulation**.\n\nMore specifically, compared with the works you cited:\n\n* **tinyBenchmarks** focuses on efficient benchmarking with fewer examples; it is not a judge benchmark, does not study LLM/LRM judges, and does not analyze thinking traces vs. verdict correctness.\n* **Easy2Hard-Bench** standardizes difficulty labels for profiling model performance and generalization; however, it is not designed for judge evaluation, does not benchmark LRM-as-a-Judge, and does not connect difficulty to routing among heterogeneous judges.\n* **Beyond the Singular** studies the value of multiple generations in benchmark evaluation; this is related to response diversity, but it does not define a benchmark where the central object is a reasoning judge with explicit thoughts, nor does it study thinking–verdict mismatches.\n* **RouteLLM**, **Prompt-to-Leaderboard**, and **ThinkSwitcher** are all routing/model-selection style works, but their routing targets are generation models or task-solving models. In contrast, our routing target is judge models, whose utility depends not only on answer quality, but also on verdict reliability, latency, token cost, and reasoning behavior.\n\nThis distinction is also reflected in the development of our own work. Our starting point was not “let us build a routing paper.” The starting point was the absence of a benchmark for **LRM-as-a-Judge**. After benchmarking these judges, we observed that using LRMs alone is often too expensive and not uniformly beneficial, even though they are stronger on some reasoning-heavy subsets. That is precisely **why the routing problem emerges in our paper**: routing is not an add-on borrowed from elsewhere, but a direct consequence of the empirical finding that judge heterogeneity matters and pure LRM judging is inefficient. This logic is already reflected in the paper’s framing and conclusions.\n\n# 3. Why this matters in practice\n\nThe practical value of this design is that it enables questions that prior resources cannot answer directly:\n\n* When should one invoke an **LRM judge** rather than a cheaper LLM judge?\n* In which domains does explicit reasoning improve verdict reliability, and in which domains is it less cost-effective?\n* How should a system allocate judges when **accuracy, latency, and token cost** must be balanced jointly?\n\nThis is why our benchmark and routing components are coupled. The benchmark is not intended as a static collection of prompts, but as a framework that yields **actionable judge utility signals**. To further support this practical angle, we added routing comparisons against prior baselines on our dataset and cross-domain generalization results in the rebuttal (Q1 and Q2). These additions are meant to show that the routing component is not just conceptually attached, but operationally meaningful once judge heterogeneity is measured.\n\nWe respectfully hope that, in reassessing the paper, you may consider our contribution and novelty in terms of the new benchmark protocol, reasoning-trace-centric judge evaluation, and routing-aware deployment formulation, rather than focusing primarily on the strong base resources we build upon."}},"parentInvitations":"ICML.cc/2026/Conference/-/Reply_Rebuttal_Comment","tmdate":1782367875436,"tcdate":1775186186600,"writers":["ICML.cc/2026/Conference","ICML.cc/2026/Conference/Submission9427/Authors"],"signatures":["ICML.cc/2026/Conference/Submission9427/Authors"],"forum":"d0dDhLR19Y","number":1,"license":"CC BY 4.0","cdate":1775186186600,"readers":["everyone"],"invitations":["ICML.cc/2026/Conference/Submission9427/Official_Review1/Rebuttal1/Rebuttal_Acknowledgement1/-/Reply_Rebuttal_Comment","ICML.cc/2026/Conference/-/Edit"],"mdate":1782367875436,"domain":"ICML.cc/2026/Conference","replyto":"I4wPGbZBkU","id":"eSW4cJez8r","forumContent":{"venue":{"value":"ICML 2026 regular"},"keywords":{"value":["Route; Judge; Large Reasoning Models; Benchmark"]},"_bibtex":{"value":"@inproceedings{\ntie2026routing,\ntitle={Routing and Reasoned Evaluation with Large Language Models},\nauthor={Guiyao Tie and Tianyao Luo and Xueyang Zhou and Chaoran Hu and Yunhong He and Junran Wu and Yuanfan Yao and Pan Zhou and Lichao Sun},\nbooktitle={Forty-third International Conference on Machine Learning},\nyear={2026},\nurl={https://openreview.net/forum?id=d0dDhLR19Y}\n}"},"title":{"value":"Routing and Reasoned Evaluation with Large Language Models"},"paperhash":{"value":"tie|routing_and_reasoned_evaluation_with_large_language_models"},"originally_submitted_PDF":{"value":"/pdf/361279dc4307d37f24539f8e72aaf74ca6e1648d.pdf"},"TLDR":{"value":"We introduce R$^2$Eval, a routing-aware automated assessment framework that formulates evaluation as a resource allocation and aggregation problem rather than relying on a single monolithic evaluator."},"primary_area":{"value":"general_machine_learning->evaluation"},"abstract":{"value":"Large language models (LLMs) are increasingly used to provide automated assessment signals for evaluating model-generated outputs. However, practical deployment faces three persistent challenges: heterogeneous reliability across models, substantial latency and token costs, and the absence of principled strategies for allocating evaluation resources. We introduce R$^2$Eval, a routing-aware automated assessment framework that formulates evaluation as a resource allocation and aggregation problem rather than relying on a single monolithic evaluator. R$^2$Eval combines difficulty-aware routing with reasoned evaluation signals to dynamically select evaluator models on a per-instance basis under explicit accuracy, latency, and cost constraints. Our study makes three contributions. First, we construct six difficulty-aware datasets spanning both reasoning-intensive (mathematics, logic, code) and non-reasoning (knowledge, roleplay, writing) tasks, with human-annotated reference assessments. Second, we provide a systematic empirical analysis of how reasoning traces produced by different evaluator models correlate with assessment outcomes, revealing substantial variance and systematic mismatches across difficulty regimes. Third, we develop and evaluate both offline and online routing strategies that adaptively allocate evaluation queries, achieving substantially improved accuracy–efficiency trade-offs compared to static baselines. Experiments across 19 language models demonstrate that R$^2$Eval significantly reduces evaluation cost and latency while maintaining close alignment with human assessments. These results highlight the importance of routing-aware automated assessment and establish R$^2$Eval as a scalable and reliable framework for large-scale model evaluation."},"pdf":{"value":"/pdf/e1375bbaeb1762b438502b1df1d706c22d638093.pdf"},"lay_summary":{"value":"Large language models are increasingly used to judge or compare answers produced by other AI systems, but using a single powerful model as the evaluator can be expensive, slow, and unreliable across different types of tasks. This paper studies how to choose the right evaluator for each evaluation case instead of always relying on one fixed model. We introduce R$^2$Eval, a framework for comparing and routing automated evaluators. It evaluates both standard language models and reasoning models on tasks such as mathematics, logic, code, knowledge, roleplay, and writing. The framework also studies when a model’s intermediate reasoning helps its final judgment and when the reasoning can be misleading. Our results show that different evaluator models have different strengths: some are better for structured reasoning tasks, while others are more efficient or better suited for subjective writing and roleplay tasks. By routing each evaluation instance to an appropriate evaluator, R$^2$Eval can reduce cost and latency while maintaining close agreement with human judgments. This makes automated evaluation more practical for large-scale use."},"venueid":{"value":"ICML.cc/2026/Conference"},"authorids":{"value":["~Guiyao_Tie1","~Tianyao_Luo2","~Xueyang_Zhou2","~Chaoran_Hu3","~Yunhong_He1","~Junran_Wu3","~Yuanfan_Yao1","~Pan_Zhou5","~Lichao_Sun1"]},"authors":{"value":["Guiyao Tie","Tianyao Luo","Xueyang Zhou","Chaoran Hu","Yunhong He","Junran Wu","Yuanfan Yao","Pan Zhou","Lichao Sun"]}},"version":2},{"content":{"comment":{"value":"### **On limited baseline scope (W1)**\n Thanks for rasing these points. We want to note that different from the classic causal estimators for average treatment effect (ATE), e.g., the IPW and DR estimators, the R-learner and DR-learner (meta-learners) considered in this work, target at a more challenging heterogeneous causal functionals: the conditional average treatment effect (CATE). The meta-learners are very flexible frameworks for the estimation of CATE function, which could be diriven by any off-the-shelf ML algorithms. Therefore, we exploit them for the flexible estimation of possibly complex GS-PB shift function $\\Delta(q)$. Moreover, our meta-router is a generally framework does not tie on any specific CATE estimation approach.\n\nThe routers trained on directly pooled GS–PB data or only PB data in our numerical experiments, follow the methodological framework of RouteLLM (Ong et al. (2024)) and other reward-model-based routers, and thus serve as the state-of-the-art baselines in our study. Consequently, evaluating Meta-Router against the PB-only and directly pooled GS–PB routers in each numerical experiment amounts to benchmarking against the SOTA baselines. In both HealthBench experiment and the newly added PRBench experiment (Akyrek et al., 2025), Meta-Router achieves superior efficiency gains over these baselines.\n    \n&nbsp;\n\n### **On scalability (W2)**\n\n  Thank you for rasing this point. In the revision, we have added a new theoretical investigation of the multi-model setting (Section A.2). Briefly, we find that we could naturally extend our causal formulation and Meta-Router to a pool of $K$ LLMs by modeling pairwise GS and PB quality differences between models and using a structured aggregation (e.g., a Bradley–Terry–Luce model) to recover a global GS-aligned ordering over the model pool. The key observation is that the debiasing step still operates at the level of pairwise GS–PB shifts: for each model pair $(M_k, M_\\ell)$, we define a shift function $\\Delta_{k,\\ell}(q)$ between GS and PB evaluations and estimate it via CATE meta-learning exactly as in the two-model case. These pairwise corrections can be learned in parallel across model pairs and then combined into a single, GS-aligned routing policy. As a result, the *training* complexity grows with the number of model pairs (on the order of $K^2$), while the *inference* cost for a given query remains linear in $K$, similar to other multi-model routing approaches that score each candidate model once.\n\n  Our theoretical analysis also clarifies what practical challenges emerge as $K$ grows. First, the main bottleneck is not computation but **label coverage**: collecting enough GS and PB comparisons across sufficient model pairs, could guarantee the learned shifts $\\Delta_{k\\ell}(q)$ are well-identified. Second, when different models exhibit qualitatively different PB biases (e.g., length bias or self-preference that is model-specific), one must decide whether to estimate separate shifts per pair $\\Delta_{k\\ell}(q)$ or impose a more structured ranking model (e.g., assuming model-specific and query-specific components) to control sample complexity. We now explicitly discuss these trade-offs and their implications in Section A.2. \n\n  Empirically validating Meta-Router on very large model pools is an important and largely orthogonal engineering challenge that requires substantial additional API budget; we see our new multi-model analysis as a first step toward that direction and plan to explore such large-pool evaluations in future work.\n\n&nbsp;\n\n### **Missing Error Analysis (W3)**\n\nThank you. The meta-routers (and other routers) misroute queries due to the biases introduced by the preference-based evaluations. We have presented the related discussions in the revised numerical experiments\n&nbsp;\n\n### **On computational overhead (W4 and Q1)**\n \nThank you for raising this point. In practice, the computational cost of CATE meta-learners is modest. The overall complexity is essentially the same order as training the underlying machine-learning models used within the learner.\n\nMore concretely, the computation consists of: (i) fitting the nuisance models (propensity score and outcome regressions) and the final CATE regression, and (ii) for certain meta-learners, constructing pseudo-outcomes. Step (i) has the same computational order as training the chosen ML algorithm for nuisance and CATE function approximations. Step (ii) requires only a single pass through the data (e.g., computing R-learner or DR-learner pseudo-outcomes), which is linear in the sample size. Therefore, the additional overhead introduced by meta-learning is mild relative to the ML models used.\n\nWe have noted such computational overhead in Appendix A.3."},"title":{"value":"Response to Reviewer uL2P"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764412523279,"tcdate":1763763985249,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission18138/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission18138/Authors"],"forum":"r0BFucF2dH","number":8,"license":"CC BY 4.0","cdate":1763763985249,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission18138/-/Official_Comment"],"mdate":1764412523279,"domain":"ICLR.cc/2026/Conference","replyto":"CsULiMnpE2","id":"tj40QQbqLo","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Causal learning","Meta-learner","Large Language Model","query routing"]},"supplementary_material":{"value":"/attachment/9164932b61c808ca473aa568fa79122a32342bda.pdf"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"In language tasks requiring extensive  human-model interaction, the inference cost of large language models (LLMs) can be substantial. To reduce expenses while preserving the quality of the responses, an LLM router selects among candidate models to balance between the expected response quality and the inference cost. A central challenge in router training is the accuracy and accessibility of reliable supervision. Gold-standard data, obtained from domain experts or benchmark labels, provide accurate quality evaluations of LLM responses but are costly and difficult to scale. In contrast, preference-based data, collected via crowdsourcing or LLM-as-a-judge systems, are cheaper and more scalable, yet often biased in reflecting the true quality of responses. We cast the problem of LLM router training with combined Gold-standard and preference-based data into a causal inference framework by viewing the response evaluation mechanism as the treatment assignment. This perspective further reveals that the bias in preference-based data corresponds to the well-known causal estimand: the conditional average treatment effect (CATE). Based on this new perspective, we develop an integrative causal router training framework that corrects preference-data bias, addresses imbalances between two data sources, and improves routing robustness and efficiency.  Numerical experiments demonstrate that our approach delivers more accurate routing and improves the trade-off between cost and quality. Illustrative code to reproduce our main experiment is available at https://github.com/yichistat/Meta-router."},"_bibtex":{"value":"@inproceedings{\nzhang2026metarouter,\ntitle={Meta-Router: Bridging Gold-standard and Preference-based Evaluations in {LLM} Routing},\nauthor={Yichi Zhang and Fangzheng Xie and Shu Yang and Chong Wu},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=r0BFucF2dH}\n}"},"title":{"value":"Meta-Router: Bridging Gold-standard and Preference-based Evaluations in LLM Routing"},"pdf":{"value":"/pdf/b5b19f14eb1393098b973a891e7dde44f0e0ef87.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"zhang|metarouter_bridging_goldstandard_and_preferencebased_evaluations_in_llm_routing"},"authorids":{"value":["~Yichi_Zhang14","~Fangzheng_Xie1","~Shu_Yang4","~Chong_Wu1"]},"authors":{"value":["Yichi Zhang","Fangzheng Xie","Shu Yang","Chong Wu"]}},"version":2},{"content":{"comment":{"value":"Thank you for taking the time to review our work, and we sincerely appreciate your critical but valuable feedback for improving it. We have made every effort to address your concerns and questions, as detailed below.\n\n&nbsp;\n\n### **Practical Implementation Guidance (W1)**\nThank you for raising this important point. In the appendix, we have added a new section **\"Practical deployment and end-to-end workflow\"** which provides a concrete, end-to-end workflow for using Meta-Router in LLM systems. This section now:\n\n- clarifies how to choose the gold-standard quality objective, for example, how to define $r_i$ through a rubric-based system or as the difference between internal reward-model scores;\n\n- explains how to collect gold-standard and preference-based data in practice, e.g., by annotating a few hundred queries with expert GS labels to form $\\mathcal{D}_G$ and using cheaper crowd or LLM-as-a-judge comparisons on thousands or more queries to form $\\mathcal{D}_P$;\n\n- specifies workflow choices inside Meta-Router, including reusing a text embedding service, training causal meta-learners and the final router with lightweight tabular models (e.g., random forests); We have also added a new flowchart (Figure S1) visualizing the overall end-to-end workflow;\n\n- discusses additional deployment details, e.g., training separate or shared routers for multiple domains (e.g., medical vs. legal), the fraction of queries sent to $\\mathcal{M}_p$, realized gold-standard quality to adjust $w$ or retrain the router when distributions or budgets change.\n\nWe also hope to use this opportunity to share our motivation of this study in a way that ties theory and practice together. Existing routers typically rely either on gold-standard data alone, on preference data alone, or on *direct pooling* of gold-standard and preference data. By contrast, our results show that there is a systematic, query-dependent shift between GS and PB evaluations: simply pooling $\\mathcal{D}_G$ and $\\mathcal{D}_P$ can be worse than using $\\mathcal{D}_G$ alone. This empirical finding motivates Meta-Router’s design: **learn the query-dependent GS–PB shift via CATE meta-learners, debias the PB labels into pseudo-GS labels, and then train the router on this bias-corrected fused dataset**.  \n\n&nbsp;\n\n### **Expanded experimental scope (W2)**\n\n- **PB-only router**  \n We now include a PB-only router trained purely on $\\mathcal{D}_P$. Concretely, we learn a predictor of $\\eta(q) = \\mathbb{E}[y \\mid q]$ from the PB labels only and then derive a routing rule based on this predicted PB gain. We have reported the performance of the PB-only router across all original and newly added numerical experiments. As the GS sample size increases, the efficiency gains of all routers improve, except for the PB-only router, highlighting the value of incorporating GS data for debiasing.\n  \n- **SOTA baselines**\n\n  The routers trained on directly pooled GS–PB data or only PB data, follow the methodological framework of RouteLLM (Ong et al. (2024)) and other reward-model-based routers, and thus serve as the state-of-the-art baselines in our study. Consequently, evaluating Meta-Router against the PB-only and directly pooled GS–PB routers in each numerical experiment amounts to benchmarking against the SOTA baselines. In both HealthBench experiment and the newly added PRBench experiment (Akyrek et al., 2025), Meta-Router achieves superior efficiency gains over these baselines.\n  \n- **Non-healthcare domain and multi-model routing.**  \n  To assess generalization beyond HealthBench, we have added an additional experiment on a non-healthcare benchmark (i.e., PRBench) with a different query distribution focusing on law and finance reasoning (Section 4.2). The DR-learner-based meta-router consistently achieves the best efficiency gains in these new experiments.    \n  \n&nbsp;\n\n### **Ablation studies (W3)**\n\nWe conduct additional numerical ablation studies in Section 4.1 of the revision, each time modifying one key element of the main numerical experiment while keeping all other settings fixed.\n\n- We consider a simple linear-scaling debiasing method, by shifting PB data using the overall sample mean difference between PB and GS data. This simply debiased router performs similarly to directly pooled GS–PB router, whereas Meta-Router continues to outperform both. This highlights that PB–GS bias is heterogeneous across queries and that meta-learners are essential for effective debiasing.\n\n- When GS data are used without pre-normalization, Meta-Router does not significantly outperform the GS-only router, confirming the importance of the normalization step.\n- Changing the regression method from random forest to XGBoost shows that the R-learner–based router remains the strongest across algorithms, although overall EGs decline, underscoring the importance of evaluating different regression method choices for router training and demonstrating the robusteness of our proposed router."},"title":{"value":"Response to Reviewer K5JD (Part I)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763719367549,"tcdate":1763711204699,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission18138/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission18138/Authors"],"forum":"r0BFucF2dH","number":3,"license":"CC BY 4.0","cdate":1763711204699,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission18138/-/Official_Comment"],"mdate":1763719367549,"domain":"ICLR.cc/2026/Conference","replyto":"526mivGp1G","id":"ueEx7yYWfZ","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["Causal learning","Meta-learner","Large Language Model","query routing"]},"supplementary_material":{"value":"/attachment/9164932b61c808ca473aa568fa79122a32342bda.pdf"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"In language tasks requiring extensive  human-model interaction, the inference cost of large language models (LLMs) can be substantial. To reduce expenses while preserving the quality of the responses, an LLM router selects among candidate models to balance between the expected response quality and the inference cost. A central challenge in router training is the accuracy and accessibility of reliable supervision. Gold-standard data, obtained from domain experts or benchmark labels, provide accurate quality evaluations of LLM responses but are costly and difficult to scale. In contrast, preference-based data, collected via crowdsourcing or LLM-as-a-judge systems, are cheaper and more scalable, yet often biased in reflecting the true quality of responses. We cast the problem of LLM router training with combined Gold-standard and preference-based data into a causal inference framework by viewing the response evaluation mechanism as the treatment assignment. This perspective further reveals that the bias in preference-based data corresponds to the well-known causal estimand: the conditional average treatment effect (CATE). Based on this new perspective, we develop an integrative causal router training framework that corrects preference-data bias, addresses imbalances between two data sources, and improves routing robustness and efficiency.  Numerical experiments demonstrate that our approach delivers more accurate routing and improves the trade-off between cost and quality. Illustrative code to reproduce our main experiment is available at https://github.com/yichistat/Meta-router."},"_bibtex":{"value":"@inproceedings{\nzhang2026metarouter,\ntitle={Meta-Router: Bridging Gold-standard and Preference-based Evaluations in {LLM} Routing},\nauthor={Yichi Zhang and Fangzheng Xie and Shu Yang and Chong Wu},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=r0BFucF2dH}\n}"},"title":{"value":"Meta-Router: Bridging Gold-standard and Preference-based Evaluations in LLM Routing"},"pdf":{"value":"/pdf/b5b19f14eb1393098b973a891e7dde44f0e0ef87.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"zhang|metarouter_bridging_goldstandard_and_preferencebased_evaluations_in_llm_routing"},"authorids":{"value":["~Yichi_Zhang14","~Fangzheng_Xie1","~Shu_Yang4","~Chong_Wu1"]},"authors":{"value":["Yichi Zhang","Fangzheng Xie","Shu Yang","Chong Wu"]}},"version":2}],"count":120}