{"notes":[{"content":{"summary":{"value":"The paper proposes FrugalGPT, a pipeline for saving API costs for a given task, by progressively calling more capable APIs and stoping when a judge (verifier) determines the answer is good enough. On 5 different benchmarks, it is shown that FrugalGPT can save costs by 52%-98% depending on the task."},"presentation":{"value":"3 good"},"contribution":{"value":"2 fair"},"soundness":{"value":"2 fair"},"strengths":{"value":"* FrugalGPT is a simple and effective method for saving API costs. It has practical utilities.\n* The paper is in general well-written."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"* The benefit of FrugalGPT over simple methods such as fine-tuning is unclear. \n  * Experiments are mainly conducted on QA and text classification tasks. From my understanding, a baseline of fine-tuning RoBERTa will achieve 90%+ accuracy on AGNews and CoQA. \n  * In terms of data efficiency, FrugalGPT also requires a training set to train the judge and the permutation function, and need extra information to compute MPI.\n  * From my understanding, calling API is most common when there are only few training examples, or the task is complex. It would be more convincing to show that FrugalGPT works well in these cases.\n* Many details are missing in the current version of the paper, see questions below."},"confidence":{"value":"3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked."},"questions":{"value":"Questions:\n* Is FrugalGPT a task-specific or task-agnostic method? In the current experiments, are the parameters (permutation, judge model, threshold) trained separately for each task?\n* What's the cost of estimating MPI? Does that mean you need to run all LLMs over all training examples?\n* What data is used to train the DistilBERT judge?\n* For classification tasks, is it possible to use the DistilBERT judge directly for inference?\n* Since the judge model is doing a regression problem, what's the reason of using a 2-d output, and taking the maximum value of the last layer (normalized by softmax)?\n* As generation diversity is mentioned as a reason for improving performance, would it be possible to sample outputs from the same LLM multiple times, and ensemble their answers. As shown in https://arxiv.org/abs/2203.11171 (Figure 2) this would also create a cost-performance curve, and the performance grows with more calls. To demonstrate the contribution of FrugalGPT, such curves should also be included for comparison.\n* In Figure 1(d), I'm a little confused why the leftmost dot of FrugalGPT is still better than the cheap models. At this budget point, I would assume FrugalGPT makes one single call to one API, and thus the dot of FrugalGPT should overlap with one of the model. Please help me understand this better.\n\nSuggestion:\n* Regarding the distribution shift experiments, currently it's mainly done by changing the label distribution in the test set. It would be interesting to study and evaluate how FrugalGPT performs when there is distribution shift on the input text."},"rating":{"value":"5: marginally below the acceptance threshold"},"code_of_conduct":{"value":"Yes"}},"nonreaders":[],"tmdate":1701885180896,"tcdate":1698799620128,"writers":["ICLR.cc/2024/Conference","ICLR.cc/2024/Conference/Submission6216/Reviewer_3Fst"],"signatures":["ICLR.cc/2024/Conference/Submission6216/Reviewer_3Fst"],"forum":"XUZ2S0JVJP","number":3,"license":"CC BY 4.0","cdate":1698799620128,"readers":["everyone"],"invitations":["ICLR.cc/2024/Conference/Submission6216/-/Official_Review","ICLR.cc/2024/Conference/-/Edit"],"mdate":1701885180896,"domain":"ICLR.cc/2024/Conference","replyto":"XUZ2S0JVJP","id":"xqtrftUhly","forumContent":{"TLDR":{"value":"We present FrugalGPT, an algorithmic framework that adaptively selects which generative LLM APIs to use in order to reduce cost and improve performance."},"venue":{"value":"Submitted to ICLR 2024"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["LLM; GPT-4; joint optimization of performance-cost; MLaaS"]},"supplementary_material":{"value":"/attachment/b75b8be632d4844c128eedead8de2bab645f8c5c.pdf"},"primary_area":{"value":"generative models"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide."},"abstract":{"value":"The rapid adoption of large language models (LLMs) has led to an growing number of companies offering generative LLMs as callable services at varying costs. We find that popular generative LLM APIs, such as GPT-4, ChatGPT, and J1-Jumbo, exhibit heterogeneous pricing structures, with fees that can differ by two orders of magnitude, and heterogeneous performance across tasks and input queries. This makes it challenging for users to decide which generative LLM APIs to utilize for their applications and budget. Motivated by these findings, we propose FrugalGPT, an algorithmic framework that adaptively selects which generative LLMs to use for different queries to reduce cost and improve accuracy. Our experiments demonstrate that, for a range of natural language tasks including news classification, reading comprehension, and scientific question answering, FrugalGPT can match the performance of the best individual generative LLM (e.g., GPT-4) with up to a 98% cost reduction or improve the accuracy over GPT-4 by 4% at the same cost. The ideas and findings presented in this paper lay a foundation for using LLMs sustainably and efficiently."},"_bibtex":{"value":"@misc{\nchen2024frugalgpt,\ntitle={Frugal{GPT}: How to Use Large Language Models While Reducing Cost and Improving Performance},\nauthor={Lingjiao Chen and Matei Zaharia and James Zou},\nyear={2024},\nurl={https://openreview.net/forum?id=XUZ2S0JVJP}\n}"},"title":{"value":"FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance"},"pdf":{"value":"/pdf/44f9acd4b03577917238a232b64b9793f40a8b74.pdf"},"venueid":{"value":"ICLR.cc/2024/Conference/Rejected_Submission"},"paperhash":{"value":"chen|frugalgpt_how_to_use_large_language_models_while_reducing_cost_and_improving_performance"},"authorids":{"value":["~Lingjiao_Chen1","~Matei_Zaharia1","~James_Zou1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors' identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Lingjiao Chen","Matei Zaharia","James Zou"]}},"version":2},{"content":{"submission_length":{"value":"Regular submission (no more than 12 pages of main content)"},"venue":{"value":"Accepted by TMLR"},"supplementary_material":{"value":"/attachment/9eb12830d754ac278eaaab46ba9193ca09dee881.pdf"},"abstract":{"value":"The rapid adoption of large language models (LLMs) has led to a growing number of companies offering generative LLMs as callable services at varying costs. We find that popular generative LLM APIs, such as GPT-4, Gemini 1.5, and Claude 3.5, exhibit heterogeneous pricing structures, with fees that can differ by two orders of magnitude and heterogeneous performance across tasks and input queries. This makes it challenging for users to decide which generative LLM APIs to utilize for their applications and budget. Motivated by these findings, we propose FrugalGPT, an algorithmic framework that adaptively selects which generative LLMs to use for different queries to reduce cost and improve accuracy. Our experiments demonstrate that, for a range of natural language tasks including news classification, reading comprehension, and scientific question answering, FrugalGPT can match the performance of the best individual generative LLM (e.g., GPT-4) with up to a 98% cost reduction or improve the accuracy over GPT-4 by 4% at the same cost. The ideas and findings presented in this paper lay a foundation for using LLMs sustainably and efficiently."},"_bibtex":{"value":"@article{\nchen2024frugalgpt,\ntitle={Frugal{GPT}: How to Use Large Language Models While Reducing Cost and Improving Performance},\nauthor={Lingjiao Chen and Matei Zaharia and James Zou},\njournal={Transactions on Machine Learning Research},\nissn={2835-8856},\nyear={2024},\nurl={https://openreview.net/forum?id=cSimKw5p6R},\nnote={Featured Certification}\n}"},"title":{"value":"FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance"},"certifications":{"value":["Featured Certification"]},"pdf":{"value":"/pdf/3bc37598b9fc1a12afe49c84e43e6beb9d8e2c6a.pdf"},"venueid":{"value":"TMLR"},"paperhash":{"value":"chen|frugalgpt_how_to_use_large_language_models_while_reducing_cost_and_improving_performance"},"authorids":{"value":["~Lingjiao_Chen1","~Matei_Zaharia1","~James_Zou1"]},"assigned_action_editor":{"value":"~Yu-Xiong_Wang1"},"authors":{"value":["Lingjiao Chen","Matei Zaharia","James Zou"]}},"tmdate":1764287603352,"pdate":1734677259186,"tcdate":1719592761351,"writers":["TMLR"],"signatures":["TMLR/Paper2934/Authors"],"forum":"cSimKw5p6R","license":"CC BY 4.0","number":2934,"cdate":1719592761351,"readers":["everyone"],"invitations":["TMLR/-/Submission","TMLR/-/Edit","TMLR/-/Under_Review","TMLR/Paper2934/-/Revision","TMLR/Paper2934/-/Camera_Ready_Revision","TMLR/-/Accepted","TMLR/Paper2934/-/EIC_Revision"],"mdate":1764287603352,"odate":1720114064449,"domain":"TMLR","id":"cSimKw5p6R","version":2},{"content":{"TLDR":{"value":"We present FrugalGPT, an algorithmic framework that adaptively selects which generative LLM APIs to use in order to reduce cost and improve performance."},"venue":{"value":"Submitted to ICLR 2024"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["LLM; GPT-4; joint optimization of performance-cost; MLaaS"]},"supplementary_material":{"value":"/attachment/b75b8be632d4844c128eedead8de2bab645f8c5c.pdf"},"primary_area":{"value":"generative models"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide."},"abstract":{"value":"The rapid adoption of large language models (LLMs) has led to an growing number of companies offering generative LLMs as callable services at varying costs. We find that popular generative LLM APIs, such as GPT-4, ChatGPT, and J1-Jumbo, exhibit heterogeneous pricing structures, with fees that can differ by two orders of magnitude, and heterogeneous performance across tasks and input queries. This makes it challenging for users to decide which generative LLM APIs to utilize for their applications and budget. Motivated by these findings, we propose FrugalGPT, an algorithmic framework that adaptively selects which generative LLMs to use for different queries to reduce cost and improve accuracy. Our experiments demonstrate that, for a range of natural language tasks including news classification, reading comprehension, and scientific question answering, FrugalGPT can match the performance of the best individual generative LLM (e.g., GPT-4) with up to a 98% cost reduction or improve the accuracy over GPT-4 by 4% at the same cost. The ideas and findings presented in this paper lay a foundation for using LLMs sustainably and efficiently."},"_bibtex":{"value":"@misc{\nchen2024frugalgpt,\ntitle={Frugal{GPT}: How to Use Large Language Models While Reducing Cost and Improving Performance},\nauthor={Lingjiao Chen and Matei Zaharia and James Zou},\nyear={2024},\nurl={https://openreview.net/forum?id=XUZ2S0JVJP}\n}"},"title":{"value":"FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance"},"pdf":{"value":"/pdf/44f9acd4b03577917238a232b64b9793f40a8b74.pdf"},"venueid":{"value":"ICLR.cc/2024/Conference/Rejected_Submission"},"paperhash":{"value":"chen|frugalgpt_how_to_use_large_language_models_while_reducing_cost_and_improving_performance"},"authorids":{"value":["~Lingjiao_Chen1","~Matei_Zaharia1","~James_Zou1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors' identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Lingjiao Chen","Matei Zaharia","James Zou"]}},"tmdate":1707625734207,"tcdate":1695413452599,"writers":["ICLR.cc/2024/Conference","ICLR.cc/2024/Conference/Submission6216/Authors"],"signatures":["ICLR.cc/2024/Conference/Submission6216/Authors"],"forum":"XUZ2S0JVJP","number":6216,"cdate":1695413452599,"readers":["everyone"],"invitations":["ICLR.cc/2024/Conference/-/Submission","ICLR.cc/2024/Conference/-/Post_Submission","ICLR.cc/2024/Conference/Submission6216/-/Revision","ICLR.cc/2024/Conference/Submission6216/-/Rebuttal_Revision","ICLR.cc/2024/Conference/-/Edit"],"mdate":1707625734207,"odate":1697213872796,"domain":"ICLR.cc/2024/Conference","id":"XUZ2S0JVJP","version":2},{"content":{"venue":{"value":"CoRR 2023"},"pdf":{"value":"http://arxiv.org/pdf/2305.05176v1"},"venueid":{"value":"dblp.org/journals/CORR/2023"},"paperhash":{"value":"chen|frugalgpt_how_to_use_large_language_models_while_reducing_cost_and_improving_performance"},"authorids":{"value":["~Lingjiao_Chen1","https://dblp.org/search/pid/api?q=author:Matei_Zaharia:","https://dblp.org/search/pid/api?q=author:James_Zou_0001:"]},"html":{"value":"https://doi.org/10.48550/arXiv.2305.05176"},"_bibtex":{"value":"@article{DBLP:journals/corr/abs-2305-05176,\n  publtype={informal},\n  author={Lingjiao Chen and Matei Zaharia and James Zou},\n  title={FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance},\n  year={2023},\n  cdate={1672531200000},\n  journal={CoRR},\n  volume={abs/2305.05176},\n  url={https://doi.org/10.48550/arXiv.2305.05176}\n}\n"},"abstract":{"value":"There is a rapidly growing number of large language models (LLMs) that users can query for a fee. We review the cost associated with querying popular LLM APIs, e.g. GPT-4, ChatGPT, J1-Jumbo, and find that these models have heterogeneous pricing structures, with fees that can differ by two orders of magnitude. In particular, using LLMs on large collections of queries and text can be expensive. Motivated by this, we outline and discuss three types of strategies that users can exploit to reduce the inference cost associated with using LLMs: 1) prompt adaptation, 2) LLM approximation, and 3) LLM cascade. As an example, we propose FrugalGPT, a simple yet flexible instantiation of LLM cascade which learns which combinations of LLMs to use for different queries in order to reduce cost and improve accuracy. Our experiments show that FrugalGPT can match the performance of the best individual LLM (e.g. GPT-4) with up to 98% cost reduction or improve the accuracy over GPT-4 by 4% with the same cost. The ideas and findings presented here lay a foundation for using LLMs sustainably and efficiently."},"title":{"value":"FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance"},"authors":{"value":["Lingjiao Chen","Matei Zaharia","James Zou"]}},"tmdate":1727539947790,"pdate":1672531200000,"tcdate":1727539932228,"writers":["~"],"signatures":["~Lingjiao_Chen1"],"forum":"4UeopY7Ogl","license":"CC BY-SA 4.0","number":102665,"cdate":1672531200000,"readers":["everyone"],"invitations":["DBLP.org/-/Record","DBLP.org/-/Edit"],"mdate":1727539947790,"domain":"DBLP.org","id":"4UeopY7Ogl","version":2},{"content":{"comment":{"value":"Thank you very much for your time and feedback. \n\n***[The benefit of FrugalGPT over simple methods such as fine-tuning]***: Fine-tuning requires more data. For example, 127k data points are needed to fine-tune a ROBERTa model on COQA [R1], while only a few hundred samples are needed for FrugalGPT. In addition, FrugalGPT is complementary to fine-tuning an individual LLM since we can wrap FrugalGPT on top of the fine-tuned LLM and other LLMs to reduce cost. \n\n[R1], I. Staliūnaitė et al, Compositional and Lexical Semantics in RoBERTa, BERT and DistilBERT: A Case Study on CoQA, 2020.\n\n***[Experiments are mainly conducted on QA and text classification tasks]***: These tasks are selected partially because they are relatively easy to evaluate, which is why these datasets are commonly used to evaluate LLM research. In addition, several of our datasets are quite challenging. For example, on the COQA and SCIQ datasets, even GPT-4 is only able to achieve 40%-70% accuracy. \n\n***[FrugalGPT requires a training set to train the judge and the permutation function, and need extra information to compute MPI]***: FrugalGPT is designed for applications involving a large number of LLM queries. Consider, for example, Yabble (https://openai.com/customer-stories/yabble), which analyzes customer posts for sentiment,  or, Jasper (https://www.jasper.ai/the-prompt/gpt4-future-of-gen-ai-business), which offers chatbots for varying domains. Here, as in many other industry applications of LLM, many tens of thousands of new queries per day are expected. In these cases, the cost of tuning FrugalGPT on a few thousand examples is well worth the reduction in cost when applying it to millions over time. We also conducted an additional study on the effects of FrugalGPT training set size in the HEADLINES dataset.  Specifically, we varied the data used to train FrugalGPT from 500 to 5000, and measured the accuracy of the FrugalGPT LLM cascade on the test set with 5000 samples. As shown in the following table, FrugalGPT’s performance was consistently better than GPT-4 (85.5%). We have added this in the revised paper (see Section B.5 in the Appendix). \n\n| Training set size            | Accuracy |\n| :----------------: | :------: | \n| 500        |   85.9   |\n| 1000           |  86.8  |\n| 2000   | 86.8 |\n| 3000   | 86.8 |\n| 4000   | 87.2 |\n| 5000   | 87.0 |\n \n\n***[From my understanding, calling API is most common when there are only few training examples, or the task is complex]***: Many enterprises are interested in using LLMs on a large number of relatively repetitive tasks. For example, financial companies such as Stripe are using LLMs to perform large scale fraud detection (https://www.techcircle.in/2023/03/16/11-companies-using-gpt-4-in-consumer-products), and Yabble analyzes customer posts for sentiment (https://openai.com/customer-stories/yabble), which is similar to our evaluation datasets AGNEWS. Insurance companies are also using LLMs to process millions of claims. Also, FrugalGPT requires fewer labeled data points than fine tuning a model for the task would (since it only needs to train a router that can still use the big model).\n\n***[Is FrugalGPT a task-specific or task-agnostic method]***: The FrugalGPT paradigm is task-agnostic, but different tasks do require additional training for better performance. For example, our experiments show the cost savings due to FrugalGPT in diverse tasks including price prediction, reading comprehension and news classification.\n\n***[What's the cost of estimating MPI]***: MPI was estimated based on each API’s performance on the training partitions. FrugalGPT was designed for applications requiring numerous and continuous queries to LLMs. In such applications, the cost of estimating MPI is often small (less than 5%) compared to processing queries during deployment.\n\n***[What data is used to train the DistilBERT judge]***: The judge was trained on the training partition of each evaluation dataset. The judge training is data efficient: a few hundred examples are enough to obtain a high quality judge.\n\n***[Is it possible to use the DistilBERT judge directly for inference for classification tasks]: Yes. One can use the judge to process a query with each possible label and pick the label with the maximum score returned by the judge. However, this performance is much worse than FrugalGPT. For example, on the HEADLINES dataset, directly using the judge for inference gives an accuracy of 60% performance compared to the FrugalGPT’s accuracy of 87%. This is because judging is easier than prediction. For example, for a given query, the judge may assign low scores to all possible labels. In this case, FrugalGPT would defer the query to the next LLM, which may give a high-quality answer. Since all assigned scores are low, directly using the judge for labeling can give poor predictions."},"title":{"value":"Thank you ver much for your time and feedback! We have answered your questions as below (1/2)"}},"tmdate":1700551993711,"tcdate":1700551740580,"writers":["ICLR.cc/2024/Conference","ICLR.cc/2024/Conference/Submission6216/Authors"],"signatures":["ICLR.cc/2024/Conference/Submission6216/Authors"],"forum":"XUZ2S0JVJP","number":3,"license":"CC BY 4.0","cdate":1700551740580,"readers":["everyone"],"invitations":["ICLR.cc/2024/Conference/Submission6216/-/Official_Comment"],"mdate":1700551993711,"domain":"ICLR.cc/2024/Conference","replyto":"xqtrftUhly","id":"5au4y9YdOl","forumContent":{"TLDR":{"value":"We present FrugalGPT, an algorithmic framework that adaptively selects which generative LLM APIs to use in order to reduce cost and improve performance."},"venue":{"value":"Submitted to ICLR 2024"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["LLM; GPT-4; joint optimization of performance-cost; MLaaS"]},"supplementary_material":{"value":"/attachment/b75b8be632d4844c128eedead8de2bab645f8c5c.pdf"},"primary_area":{"value":"generative models"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide."},"abstract":{"value":"The rapid adoption of large language models (LLMs) has led to an growing number of companies offering generative LLMs as callable services at varying costs. We find that popular generative LLM APIs, such as GPT-4, ChatGPT, and J1-Jumbo, exhibit heterogeneous pricing structures, with fees that can differ by two orders of magnitude, and heterogeneous performance across tasks and input queries. This makes it challenging for users to decide which generative LLM APIs to utilize for their applications and budget. Motivated by these findings, we propose FrugalGPT, an algorithmic framework that adaptively selects which generative LLMs to use for different queries to reduce cost and improve accuracy. Our experiments demonstrate that, for a range of natural language tasks including news classification, reading comprehension, and scientific question answering, FrugalGPT can match the performance of the best individual generative LLM (e.g., GPT-4) with up to a 98% cost reduction or improve the accuracy over GPT-4 by 4% at the same cost. The ideas and findings presented in this paper lay a foundation for using LLMs sustainably and efficiently."},"_bibtex":{"value":"@misc{\nchen2024frugalgpt,\ntitle={Frugal{GPT}: How to Use Large Language Models While Reducing Cost and Improving Performance},\nauthor={Lingjiao Chen and Matei Zaharia and James Zou},\nyear={2024},\nurl={https://openreview.net/forum?id=XUZ2S0JVJP}\n}"},"title":{"value":"FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance"},"pdf":{"value":"/pdf/44f9acd4b03577917238a232b64b9793f40a8b74.pdf"},"venueid":{"value":"ICLR.cc/2024/Conference/Rejected_Submission"},"paperhash":{"value":"chen|frugalgpt_how_to_use_large_language_models_while_reducing_cost_and_improving_performance"},"authorids":{"value":["~Lingjiao_Chen1","~Matei_Zaharia1","~James_Zou1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors' identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Lingjiao Chen","Matei Zaharia","James Zou"]}},"version":2},{"content":{"comment":{"value":"Thank you for your helpful feedback and support of the paper! We answer your questions as follows.\n\n***[How much does the search space pruning help in practice, versus just ordering the APIs in increasing order of cost?]***: Pruning does help identify the most effective set of LLMs to query. As an ablation study, we compare the performance of FrugalGPT and that of using three APIs in increasing order of cost: GPT-J, GPT-Neo, and GPT-4. We chose these three APIs because they roughly lie on the surface of the convex hull of all APIs in Figure 1. As shown below, pruning (FrugalGPT) brings additional 1-2% performance improvements compared to using a fixed ordering of APIs in increasing cost.\n\n| Cost (fraction of GPT-4) | FrugalGPT | FrugalGPT with fixed API order|\n| :----------------: | :------: | :------: | \n| 10%  | 86.8 |85.1 |\n| 20%  | 86.9 |85.6 |\n| 100%  | 87.2 | 85.8|\n\n***[How much changing the threshold T affected results? Is the method very sensitive to T?]***: The threshold T does affect the performance, as it controls when a query is routed to an API. Fortunately, practitioners do not need to tune it themselves: the threshold is learned automatically during FrugalGPT’s training process (the optimization problem on the bottom of page 4). \n\n***[Writing a discussion on the constraints of the design space]***: Thank you for the helpful suggestion! We have added a short discussion in the revised draft (see Section D in the appendix).\n\n\n\n***[If APIs give probability/logit outputs, is a scorer still necessary?]***: The scorer is a flexible module to measure the quality of a generation; it does not necessarily need to be the particular form we used. If the APIs give probability/logit outputs that truly capture the quality of their generations, then leveraging these signals can certainly simplify the scorer design. However, in practice, the API’s probability and logit are often not calibrated, which is why having a scorer helps.  \n\n\n\n***[Do you believe that additionally optimizing for this can result in similar improvements to latency and an improvement in cost (ie, either replacing cost budget with latency budget or combining the two)?]***: Yes. Replacing the cost budget cost by the latency budget should work. For latency-critical applications, one can set up a small latency budget and always call the API with the fastest response rate."},"title":{"value":"Thank you for your helpful feedback and support of the paper!"}},"tmdate":1700551948109,"tcdate":1700551948109,"writers":["ICLR.cc/2024/Conference","ICLR.cc/2024/Conference/Submission6216/Authors"],"signatures":["ICLR.cc/2024/Conference/Submission6216/Authors"],"forum":"XUZ2S0JVJP","number":5,"license":"CC BY 4.0","cdate":1700551948109,"readers":["everyone"],"invitations":["ICLR.cc/2024/Conference/Submission6216/-/Official_Comment"],"mdate":1700551948109,"domain":"ICLR.cc/2024/Conference","replyto":"7GqKIUN7Fn","id":"8ueZmIwe3V","forumContent":{"TLDR":{"value":"We present FrugalGPT, an algorithmic framework that adaptively selects which generative LLM APIs to use in order to reduce cost and improve performance."},"venue":{"value":"Submitted to ICLR 2024"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["LLM; GPT-4; joint optimization of performance-cost; MLaaS"]},"supplementary_material":{"value":"/attachment/b75b8be632d4844c128eedead8de2bab645f8c5c.pdf"},"primary_area":{"value":"generative models"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide."},"abstract":{"value":"The rapid adoption of large language models (LLMs) has led to an growing number of companies offering generative LLMs as callable services at varying costs. We find that popular generative LLM APIs, such as GPT-4, ChatGPT, and J1-Jumbo, exhibit heterogeneous pricing structures, with fees that can differ by two orders of magnitude, and heterogeneous performance across tasks and input queries. This makes it challenging for users to decide which generative LLM APIs to utilize for their applications and budget. Motivated by these findings, we propose FrugalGPT, an algorithmic framework that adaptively selects which generative LLMs to use for different queries to reduce cost and improve accuracy. Our experiments demonstrate that, for a range of natural language tasks including news classification, reading comprehension, and scientific question answering, FrugalGPT can match the performance of the best individual generative LLM (e.g., GPT-4) with up to a 98% cost reduction or improve the accuracy over GPT-4 by 4% at the same cost. The ideas and findings presented in this paper lay a foundation for using LLMs sustainably and efficiently."},"_bibtex":{"value":"@misc{\nchen2024frugalgpt,\ntitle={Frugal{GPT}: How to Use Large Language Models While Reducing Cost and Improving Performance},\nauthor={Lingjiao Chen and Matei Zaharia and James Zou},\nyear={2024},\nurl={https://openreview.net/forum?id=XUZ2S0JVJP}\n}"},"title":{"value":"FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance"},"pdf":{"value":"/pdf/44f9acd4b03577917238a232b64b9793f40a8b74.pdf"},"venueid":{"value":"ICLR.cc/2024/Conference/Rejected_Submission"},"paperhash":{"value":"chen|frugalgpt_how_to_use_large_language_models_while_reducing_cost_and_improving_performance"},"authorids":{"value":["~Lingjiao_Chen1","~Matei_Zaharia1","~James_Zou1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors' identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Lingjiao Chen","Matei Zaharia","James Zou"]}},"version":2},{"content":{"comment":{"value":"We thank the reviewer for their valuable feedback. Below we respond to all of the reviewer's concerns/questions in detail.\n- **Fig 2 could provide more elaborated or distinguished details for Sequential Bandits to make it friendly to the audiences from other fields.** : We have updated Figure 2 to make it more friendly for readers from other fields by simplifying the language and also adding the online cost model training. We would be happy to make further revisions if the reviewer thinks that any aspects of the revised figure are confusing or difficult to understand for non-experts.\n- **As provided in related works, it seems there exist other recent studies for LLM selection pipeline, while the experiment only employed adapted traditional MAB baselines.** : We have compared our approach to another LLM selection baseline, FrugalGPT (which uses a model cascade) for the Medical experiments and included the results in Figure 3. FrugalGPT is an offline cascade-building method, and we re-implemented it within our own prompting setup, Azure API calls, evaluation pipeline, and datasets so that comparisons would be fair. Specifically, we replaced FrugalGPT’s built-in completion and scoring components with our Azure inference wrapper and our task-specific reward function. We also added a round-based evaluation loop that applies the fixed cascade to each query in sequence, which lets us compute cumulative reward and cost curves, even though FrugalGPT itself does not perform online learning. Finally, because different cascades depend on the allowed budget, we standardized this by limiting the budget to three times the diagnosis subtask cost for all FrugalGPT variants. We also aim to add the results for the Telecom experiments before the end of the discussion period. These results are for the non-pipelined version of FrugalGPT (which is why FrugalGPT obtains the lowest cost among all baselines in Figure 3), and we will also try to extend it to include a pipelined version before the end of the discussion period and share the results once they are available.\n- **Although illustrate reward and cost for each baseline, a more detailed results on task accuracy, success rate, etc. are not provided to give a more clear review of the method effectiveness or usability.** : We have included some results about task accuracy in the Appendix other than Figure 1 in the main paper, which we detail below. The task accuracy for the medical dataset is defined according to whether the LLM pipeline successfully identified a correct diagnosis, and that for the telecom data is defined according to whether the LLM pipeline successfully identified the correct answer to a multiple-choice question and explained the answer it gave correctly. Figure 17 shows the average accuracy of models for the diagnosis subtask in 3 subtask medical setting given the summarizer is selected at random, Figure 19-21 display the average accuracy of models for the summarization, answering and explanation subtasks given all models are selected at random (equivalent to Random baseline) for the Telecom setting. Figure 25 shows the average accuracy of models for the summarization subtask given the summarizer is selected at random for the medical setting. Figure 26 shows the diagnosis accuracies given the summarizer is selected as the model shown for random diagnoser selection for the medical setting and Figure 27 is a plot of the average accuracy of models for the diagnosis subtask for random summarizer and diagnoser selection in the medical setting."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763700349901,"tcdate":1763700349901,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission13581/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission13581/Authors"],"forum":"EiJbxZTtDj","number":4,"license":"CC BY 4.0","cdate":1763700349901,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission13581/-/Official_Comment"],"mdate":1763700349901,"domain":"ICLR.cc/2026/Conference","replyto":"Awfb2yJ1EW","id":"QWyZ5iqp51","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["neural bandits","online LLM selection"]},"supplementary_material":{"value":"/attachment/a9a80a4aff5ba151c2f87476eb660c59f0e0a5da.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"With the increasing popularity of large language models (LLMs) for a variety of tasks, there has been a growing interest in strategies that can predict which out of a set of LLMs will yield a successful answer at low cost. This problem promises to become more and more relevant as providers like Microsoft allow users to easily create custom LLM \"assistants'' specialized to particular types of queries. However, some tasks (i.e., queries) may benefit from breaking down the task into smaller subtasks, each of which can then be executed by a LLM expected to perform well on that specific subtask. For example, in extracting a diagnosis from medical records, one can first select an LLM to summarize the record, select another to validate the summary, and then select another, possibly different, LLM to extract the diagnosis from the summarized record. Unlike existing LLM selection or routing algorithms, this setting requires selecting a sequence of LLMs, with the output of each LLM feeding into the next and potentially influencing its success. Thus, unlike single LLM selection, the quality of each subtask's output directly affects the inputs, and hence the cost and success rate, of downstream LLMs, creating complex performance dependencies that must be learned during selection. We propose a neural contextual bandit-based algorithm that trains neural networks that model LLM success on each subtask in an online manner, thus learning to guide the LLM selections for the different subtasks, even in the absence of historical LLM performance data. Experiments on telecommunications question answering and medical diagnosis prediction datasets illustrate the effectiveness of our proposed approach compared to other LLM selection algorithms."},"_bibtex":{"value":"@misc{\natalar2026neural,\ntitle={Neural Bandit Based Optimal {LLM} Selection for a Pipeline of Tasks},\nauthor={Baran Atalar and Eddie Zhang and Carlee Joe-Wong},\nyear={2026},\nurl={https://openreview.net/forum?id=EiJbxZTtDj}\n}"},"title":{"value":"Neural Bandit Based Optimal LLM Selection for a Pipeline of Tasks"},"pdf":{"value":"/pdf/0ea0038c815706f637e52253b61de3097ff3ae5b.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"atalar|neural_bandit_based_optimal_llm_selection_for_a_pipeline_of_tasks"},"authorids":{"value":["~Baran_Atalar1","~Eddie_Zhang3","~Carlee_Joe-Wong1"]},"authors":{"value":["Baran Atalar","Eddie Zhang","Carlee Joe-Wong"]}},"version":2},{"content":{"comment":{"value":"We thank the reviewer for their valuable feedback. Below we respond to all of the reviewer's concerns/questions in detail.\n- **The details of the cost prediction model, particularly in terms of output token length prediction and overall cost estimation, are not fully explained.** : We would like to note that we give details about the cost prediction model, output token length prediction and overall cost estimation in lines 249-254, 387-394 in the main paper and sections B.3 and B.4 in the Appendix (lines 1115-1151, 1178-1187). We have moved some of these details to the main body of the paper (lines 387-394) and are happy to provide further details if desired.\n- **The paper compares the method to a few baselines but does not offer a comprehensive comparison with other state-of-the-art LLM selection algorithms.** : We have compared our approach to another LLM selection baseline, FrugalGPT (which uses a model cascade) for the Medical experiments and included the results in Figure 3. FrugalGPT is an offline cascade-building method, and we re-implemented it within our own prompting setup, Azure API calls, evaluation pipeline, and datasets so that comparisons would be fair. Specifically, we replaced FrugalGPT’s built-in completion and scoring components with our Azure inference wrapper and our task-specific reward function. We also added a round-based evaluation loop that applies the fixed cascade to each query in sequence, which lets us compute cumulative reward and cost curves, even though FrugalGPT itself does not perform online learning. Finally, because different cascades depend on the allowed budget, we standardized this by limiting the budget to three times the diagnosis subtask cost for all FrugalGPT variants. We also aim to add the results for the Telecom experiments before the end of the discussion period. These results are for the non-pipelined version of FrugalGPT (which is why FrugalGPT obtains the lowest cost among all baselines in Figure 3), and we will also try to extend it to include a pipelined version before the end of the discussion period and share the results once they are available.\n- **The approach uses separate neural networks for each subtask, which could lead to scalability challenges for larger systems.** : We agree that the scalability challenges for larger systems is a valid concern and important point. In particular, scaling to tasks with a large number of subtasks may appear to be a significant challenge, as we train separate neural networks for each subtask. However, we would like to note that the neural networks we use are quite small and shallow with a hidden dimension of 50 and a total parameter count of approximately 19,300 for each of the neural networks. Thus, keeping track of the network parameters for each subtask is likely to have tolerable memory and storage overhead. Training these neural networks during model selection also takes negligible time compared to the LLM API calls that we make in every iteration. In fact, we conducted some recent experiments on the Telecom dataset where we measured the runtime of a fixed baseline like Random (which does not run any sophisticated LLM selection logic) and compared it to the runtime of our algorithm Sequential Bandits. Random achieved an average of 5657 seconds per run averaged over 5 runs while Sequential Bandits achieved a comparable runtime of 5666 seconds per run averaged over 5 runs.  We include a paragraph about runtime and scalability in the updated paper, including these results and a discussion (lines 506-515)."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763699752421,"tcdate":1763699752421,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission13581/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission13581/Authors"],"forum":"EiJbxZTtDj","number":2,"license":"CC BY 4.0","cdate":1763699752421,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission13581/-/Official_Comment"],"mdate":1763699752421,"domain":"ICLR.cc/2026/Conference","replyto":"ERbaiKcByR","id":"9N9U3jJORB","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["neural bandits","online LLM selection"]},"supplementary_material":{"value":"/attachment/a9a80a4aff5ba151c2f87476eb660c59f0e0a5da.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"With the increasing popularity of large language models (LLMs) for a variety of tasks, there has been a growing interest in strategies that can predict which out of a set of LLMs will yield a successful answer at low cost. This problem promises to become more and more relevant as providers like Microsoft allow users to easily create custom LLM \"assistants'' specialized to particular types of queries. However, some tasks (i.e., queries) may benefit from breaking down the task into smaller subtasks, each of which can then be executed by a LLM expected to perform well on that specific subtask. For example, in extracting a diagnosis from medical records, one can first select an LLM to summarize the record, select another to validate the summary, and then select another, possibly different, LLM to extract the diagnosis from the summarized record. Unlike existing LLM selection or routing algorithms, this setting requires selecting a sequence of LLMs, with the output of each LLM feeding into the next and potentially influencing its success. Thus, unlike single LLM selection, the quality of each subtask's output directly affects the inputs, and hence the cost and success rate, of downstream LLMs, creating complex performance dependencies that must be learned during selection. We propose a neural contextual bandit-based algorithm that trains neural networks that model LLM success on each subtask in an online manner, thus learning to guide the LLM selections for the different subtasks, even in the absence of historical LLM performance data. Experiments on telecommunications question answering and medical diagnosis prediction datasets illustrate the effectiveness of our proposed approach compared to other LLM selection algorithms."},"_bibtex":{"value":"@misc{\natalar2026neural,\ntitle={Neural Bandit Based Optimal {LLM} Selection for a Pipeline of Tasks},\nauthor={Baran Atalar and Eddie Zhang and Carlee Joe-Wong},\nyear={2026},\nurl={https://openreview.net/forum?id=EiJbxZTtDj}\n}"},"title":{"value":"Neural Bandit Based Optimal LLM Selection for a Pipeline of Tasks"},"pdf":{"value":"/pdf/0ea0038c815706f637e52253b61de3097ff3ae5b.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"atalar|neural_bandit_based_optimal_llm_selection_for_a_pipeline_of_tasks"},"authorids":{"value":["~Baran_Atalar1","~Eddie_Zhang3","~Carlee_Joe-Wong1"]},"authors":{"value":["Baran Atalar","Eddie Zhang","Carlee Joe-Wong"]}},"version":2},{"content":{"summary":{"value":"This paper presents AutoMix, a method that routes query to language models (LMs) of various size and capabilities to optimize the performance within a cost budget. AutoMix has two key technical features, a few-shot self-verification mechanism which estimates the reliability of its own outputs, and a POMDP-based router that can select a model to optimize the reward. Experiments across five LMs and five datasets show that AutoMix outperform two baselines (i.e., FrugalGPT, HybridLLM)."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"See my comments in \"Weaknesses\"."},"rating":{"value":5},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. Formulating the query routing optimization problem as a Partially Observable Markov Decision Process (POMDP) is interesting. Treating performance of different models as $S$ is quite smart since the most challenging part of the optimization problem is we want to make as less computation as possible while we cannot observe whether an LM's output is correct or not since the optimization happens in the testing stage. The property of the optimization problem matches very well with POMDP.\n2. The experimental results show that AutoMix outperforms FrugalGPT and HybridLLM which are two recent methods in this setup."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The experimental setup is not very rigorous. The major metric in this paper is $\\Delta_{IBC}$ which will be influenced by how you set the cost of different LMs in your experiment. Although the arguments provided in Line 216-219 is legitimate, setting the cost as 1, 60, 100, 200 respectively feels like setting \"hyperparameters\" to me. Given the results on different datasets and models in Table 1 are not consistent, I don't have confident with the conclusion from the current results. For example, will the trend change if I set the cost as 10, 60, 100, 200?\n2. While I praise the integration of POMDP as interesting and well-motived, I am not sure whether such integration adds much to the performance given the main results in Figure 4. It seems like \"AutoMix+POMDP\" and \"AutoMix+Threshold\" achieve similar results while the latter excels in simplicity. I am also wondering how much adding POMDP will increase the overhead since the system needs to run the AdaOps POMDP solver.\n3. The explanation of \"domain-specific training\" could be clearer. Since this phrase in LM literature usually refers to post-train LM on domain-specific data, it's necessary to clearly define what it means in your method. Although I believe I understand the training of router in your method, I have a clarification question: Does computing `P(o|s)` require running the inference on all the samples in the dataset and also running the self-verification step? If this is the case, should this part of the cost be considered?\n\n**Update:** Some of these points are addressed by the author response, so I updated the score from 4 (Borderline Reject) to 5 (Borderline Accept)."},"limitations":{"value":"The paper has a \"Limitations\" section."}},"nonreaders":[],"tmdate":1730880104321,"tcdate":1719834752327,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission20326/Reviewer_RAHf"],"signatures":["NeurIPS.cc/2024/Conference/Submission20326/Reviewer_RAHf"],"forum":"e6WrwIvgzX","number":2,"license":"CC BY 4.0","cdate":1719834752327,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission20326/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730880104321,"domain":"NeurIPS.cc/2024/Conference","replyto":"e6WrwIvgzX","id":"MuluzeFRBR","forumContent":{"TLDR":{"value":"AutoMix robustly routes queries among language models of varying sizes, efficiently balancing computational cost and solution accuracy"},"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Few-shot learning","Zero-shot learning","Self-Verification","cost-quality optimization","Decision making","Prompting","LLMs"]},"primary_area":{"value":"natural_language_processing"},"abstract":{"value":"Large language models (LLMs) are now available from cloud API providers in various sizes and configurations. While this diversity offers a broad spectrum of choices, effectively leveraging the options to optimize computational cost and performance remains challenging. In this work, we present AutoMix, an approach that strategically routes queries to larger LMs, based on the approximate correctness of outputs from a smaller LM. Central to AutoMix are two key technical contributions. First, it has a few-shot self-verification mechanism, which estimates the reliability of its own outputs without requiring extensive training. Second, given that self-verification can be noisy, it employs a POMDP based router that can effectively select an appropriately sized model, based on answer confidence. Experiments across five language models and five challenging datasets show that Automix consistently surpasses strong baselines, reducing computational cost by over 50\\% for comparable performance."},"_bibtex":{"value":"@inproceedings{\naggarwal2024automix,\ntitle={AutoMix: Automatically Mixing Language Models},\nauthor={Pranjal Aggarwal and Aman Madaan and Ankit Anand and Srividya Pranavi Potharaju and Swaroop Mishra and Pei Zhou and Aditya Gupta and Dheeraj Rajagopal and Karthik Kappaganthu and Yiming Yang and Shyam Upadhyay and Manaal Faruqui and Mausam .},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=e6WrwIvgzX}\n}"},"title":{"value":"AutoMix: Automatically Mixing Language Models"},"pdf":{"value":"/pdf/f99d5e58580a4ddb7a4d60eef66db492cfafb4d9.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"aggarwal|automix_automatically_mixing_language_models"},"authorids":{"value":["~Pranjal_Aggarwal1","~Aman_Madaan1","~Ankit_Anand4","~Srividya_Pranavi_Potharaju1","~Swaroop_Mishra1","~Pei_Zhou1","~Aditya_Gupta2","~Dheeraj_Rajagopal1","~Karthik_Kappaganthu1","~Yiming_Yang1","~Shyam_Upadhyay1","~Manaal_Faruqui2","~Mausam_.1"]},"authors":{"value":["Pranjal Aggarwal","Aman Madaan","Ankit Anand","Srividya Pranavi Potharaju","Swaroop Mishra","Pei Zhou","Aditya Gupta","Dheeraj Rajagopal","Karthik Kappaganthu","Yiming Yang","Shyam Upadhyay","Manaal Faruqui","Mausam ."]}},"version":2},{"content":{"comment":{"value":"We thank the reviewer for their valuable feedback. Below we respond to all of the reviewer's concerns/questions in detail.\n- **Limited Baseline Comparisons** : We have compared our approach to another LLM selection baseline, FrugalGPT (which uses a model cascade) for the Medical experiments and included the results in Figure 3. FrugalGPT is an offline cascade-building method, and we re-implemented it within our own prompting setup, Azure API calls, evaluation pipeline, and datasets so that comparisons would be fair. Specifically, we replaced FrugalGPT’s built-in completion and scoring components with our Azure inference wrapper and our task-specific reward function. We also added a round-based evaluation loop that applies the fixed cascade to each query in sequence, which lets us compute cumulative reward and cost curves, even though FrugalGPT itself does not perform online learning. Finally, because different cascades depend on the allowed budget, we standardized this by limiting the budget to three times the diagnosis subtask cost for all FrugalGPT variants. We also aim to add the results for the Telecom experiments before the end of the discussion period. These results are for the non-pipelined version of FrugalGPT (which is why FrugalGPT obtains the lowest cost among all baselines in Figure 3), and we will also try to extend it to include a pipelined version before the end of the discussion period and share the results once they are available.\n- **Task Decomposition Assumption** : We fully acknowledge that we do not address how the task decomposition should be done in this paper. However, our goal in this work was to study the decision problem that arises once a task decomposition is specified to us and not to solve the task decomposition itself. Automatic decomposition is indeed an important problem, but we believe that is a separate research direction. In fact, there are some works which focus on solving task decomposition for LLMs such as “ADaPT: As-Needed Decomposition and Planning with Language Models;” these algorithms could be applied to a given task before our sequential bandits approach is used to select which LLMs to use for each subtask in the decomposition. We cited this paper in the related work section (172-175). We do believe that an interesting direction of future work includes incorporating a framework similar to ADaPT in the online training loop of our sequential bandits algorithm, and we briefly discuss this idea in our paper’s conclusion. Regarding the concern that using a pipeline may hurt task performance compared to not breaking a task into subtasks, we agree that this is a possibility. However, it is unlikely to be the case for complex tasks; indeed, the potential benefit of this approach motivates much of the existing literature on task decomposition for LLMs. Since our approach is an online learning approach, our algorithm is able to learn whether a step in the pipeline provides a benefit in terms of task success. If it does not, based on our selection rule, the cost term will govern the model selections, prioritizing cheaper models for that step as the output of this subtask does not significantly contribute to task success.\n- **Lack of Theoretical Analysis** : As mentioned by the reviewer, we do not have any theoretical guarantees in this work, and we agree that this is an important direction of future work. However, we believe this is a nontrivial future direction meriting its own paper, as the typical NeuralUCB guarantees do not easily translate to our sequential pipeline setting. One of the key challenges is that the context of the next arm to be selected in the pipeline depends on the arm choice in the previous step of the pipeline, which effectively makes part of the decision context (i.e., the context of the next arm to be selected) dependent on previous decisions. In standard contextual bandit and neural bandit settings, this is not the case, and we have not found a straightforward way to incorporate such dependence. Our problem does have analogues in the combinatorial bandit setting, in the sense that selecting multiple arms (models) for the different subtasks at the beginning of the task can be modeled as a combinatorial bandit problem, but in this formulation an arm choice made in that round does not affect another arm’s choice. In addition, the selection rule of our algorithm is different from other neural bandit works, as it also includes a cost term which is also updated over time, presenting another challenge."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763700720701,"tcdate":1763700720701,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission13581/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission13581/Authors"],"forum":"EiJbxZTtDj","number":5,"license":"CC BY 4.0","cdate":1763700720701,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission13581/-/Official_Comment"],"mdate":1763700720701,"domain":"ICLR.cc/2026/Conference","replyto":"C4VZVDflGQ","id":"Hk0umtUUlr","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["neural bandits","online LLM selection"]},"supplementary_material":{"value":"/attachment/a9a80a4aff5ba151c2f87476eb660c59f0e0a5da.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"With the increasing popularity of large language models (LLMs) for a variety of tasks, there has been a growing interest in strategies that can predict which out of a set of LLMs will yield a successful answer at low cost. This problem promises to become more and more relevant as providers like Microsoft allow users to easily create custom LLM \"assistants'' specialized to particular types of queries. However, some tasks (i.e., queries) may benefit from breaking down the task into smaller subtasks, each of which can then be executed by a LLM expected to perform well on that specific subtask. For example, in extracting a diagnosis from medical records, one can first select an LLM to summarize the record, select another to validate the summary, and then select another, possibly different, LLM to extract the diagnosis from the summarized record. Unlike existing LLM selection or routing algorithms, this setting requires selecting a sequence of LLMs, with the output of each LLM feeding into the next and potentially influencing its success. Thus, unlike single LLM selection, the quality of each subtask's output directly affects the inputs, and hence the cost and success rate, of downstream LLMs, creating complex performance dependencies that must be learned during selection. We propose a neural contextual bandit-based algorithm that trains neural networks that model LLM success on each subtask in an online manner, thus learning to guide the LLM selections for the different subtasks, even in the absence of historical LLM performance data. Experiments on telecommunications question answering and medical diagnosis prediction datasets illustrate the effectiveness of our proposed approach compared to other LLM selection algorithms."},"_bibtex":{"value":"@misc{\natalar2026neural,\ntitle={Neural Bandit Based Optimal {LLM} Selection for a Pipeline of Tasks},\nauthor={Baran Atalar and Eddie Zhang and Carlee Joe-Wong},\nyear={2026},\nurl={https://openreview.net/forum?id=EiJbxZTtDj}\n}"},"title":{"value":"Neural Bandit Based Optimal LLM Selection for a Pipeline of Tasks"},"pdf":{"value":"/pdf/0ea0038c815706f637e52253b61de3097ff3ae5b.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"atalar|neural_bandit_based_optimal_llm_selection_for_a_pipeline_of_tasks"},"authorids":{"value":["~Baran_Atalar1","~Eddie_Zhang3","~Carlee_Joe-Wong1"]},"authors":{"value":["Baran Atalar","Eddie Zhang","Carlee Joe-Wong"]}},"version":2},{"content":{"comment":{"value":"Thank you very much for your acceptance and suggestions! We have revised the paper accordingly and uploaded the updated paper. In particular, we have evaluated FrugalGPT's performance on several datasets using 10 recent models including Claude 3.5 Sonnet, GPT-4 Turbo, GPT-4o, and Google Gemini 1.5 Pro, and highlighted the results in the main paper (Figure 1, Table 3, and Figure 4, and Figure 5). The abstract and introduction have also been updated to reflect the more recent results. \n\nThank you again for your acceptance and constructive suggestions!"},"title":{"value":"Thank you for your acceptance and suggestions!"}},"tmdate":1733197907998,"tcdate":1733197907998,"writers":["TMLR","TMLR/Paper2934/Authors"],"signatures":["TMLR/Paper2934/Authors"],"forum":"cSimKw5p6R","number":7,"license":"CC BY 4.0","cdate":1733197907998,"readers":["everyone"],"invitations":["TMLR/Paper2934/-/Official_Comment"],"mdate":1733197907998,"domain":"TMLR","replyto":"rh9wD1889m","id":"NMjci79P6y","forumContent":{"submission_length":{"value":"Regular submission (no more than 12 pages of main content)"},"venue":{"value":"Accepted by TMLR"},"supplementary_material":{"value":"/attachment/9eb12830d754ac278eaaab46ba9193ca09dee881.pdf"},"abstract":{"value":"The rapid adoption of large language models (LLMs) has led to a growing number of companies offering generative LLMs as callable services at varying costs. We find that popular generative LLM APIs, such as GPT-4, Gemini 1.5, and Claude 3.5, exhibit heterogeneous pricing structures, with fees that can differ by two orders of magnitude and heterogeneous performance across tasks and input queries. This makes it challenging for users to decide which generative LLM APIs to utilize for their applications and budget. Motivated by these findings, we propose FrugalGPT, an algorithmic framework that adaptively selects which generative LLMs to use for different queries to reduce cost and improve accuracy. Our experiments demonstrate that, for a range of natural language tasks including news classification, reading comprehension, and scientific question answering, FrugalGPT can match the performance of the best individual generative LLM (e.g., GPT-4) with up to a 98% cost reduction or improve the accuracy over GPT-4 by 4% at the same cost. The ideas and findings presented in this paper lay a foundation for using LLMs sustainably and efficiently."},"_bibtex":{"value":"@article{\nchen2024frugalgpt,\ntitle={Frugal{GPT}: How to Use Large Language Models While Reducing Cost and Improving Performance},\nauthor={Lingjiao Chen and Matei Zaharia and James Zou},\njournal={Transactions on Machine Learning Research},\nissn={2835-8856},\nyear={2024},\nurl={https://openreview.net/forum?id=cSimKw5p6R},\nnote={Featured Certification}\n}"},"title":{"value":"FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance"},"certifications":{"value":["Featured Certification"]},"pdf":{"value":"/pdf/3bc37598b9fc1a12afe49c84e43e6beb9d8e2c6a.pdf"},"venueid":{"value":"TMLR"},"paperhash":{"value":"chen|frugalgpt_how_to_use_large_language_models_while_reducing_cost_and_improving_performance"},"authorids":{"value":["~Lingjiao_Chen1","~Matei_Zaharia1","~James_Zou1"]},"assigned_action_editor":{"value":"~Yu-Xiong_Wang1"},"authors":{"value":["Lingjiao Chen","Matei Zaharia","James Zou"]}},"version":2},{"content":{"comment":{"value":"***[What’s the reason of a 2-d output for training the judge as a regression problem]***: This is because there is no numerical label for quality, and thus we cannot use the standard regression training paradigm directly. Instead, we train a model to predict correctness, which is binary and can be curated from the ground-truth labels. Then we use the model’s confidence as the judge’s quality estimation. We have added a detailed discussion in the revision (Section D in the appendix).\n\n***[Compare with sampling outputs from the same LLM multiple times, and ensembling their answers]***: We have conducted an additional experiment for the model ensembles. In particular, we query ChatGPT 3 times with different temperatures (0, 0.1, 0.2) and then take the mode of the generation to be the final output. As shown in Table 5 in the appendix, sampling multiple times leads to marginal performance gains on the HEADLINES dataset. Yet, the gain is relatively smaller than FrugalGPT, whose accuracy can be 87% at the same cost.  We have added this in the appendix (Section B) too.\n\n***[Why the leftmost dot of FrugalGPT is still better than the cheap models in Figure 1(d)]***: This is because the leftmost dot of FrugalGPT actually used a bit more budget than the cheapest model. In fact, it routed 2% of queries to ChatGPT. This leads to an accuracy gain of 0.7% (=78.7%-78%).\n\n***[It would be interesting to study and evaluate how FrugalGPT performs when there is distribution shift on the input text]***: We conducted new experiments for distribution shift on the input text. Specifically, we asked ChatGPT to rephrase each question without changing its  meaning. On the HEADLINES dataset, we observe that FrugalGPT is quite robust to such shifts. For example, GPT-4 achieves 84% overall performance on the shifted dataset, and FrugalGPT can reach the same accuracy with only 33% of the cost. We have added this analysis in the revised draft (see Section B.3 in the Appendix).\n\nThank you again for your time! We hope you would consider increasing your score if we have answered your questions. Please let us know if you have additional questions and we are happy to follow up. Thanks!"},"title":{"value":"Thank you ver much for your time and feedback! We have answered your questions as below (2/2)"}},"tmdate":1700552007982,"tcdate":1700551752385,"writers":["ICLR.cc/2024/Conference","ICLR.cc/2024/Conference/Submission6216/Authors"],"signatures":["ICLR.cc/2024/Conference/Submission6216/Authors"],"forum":"XUZ2S0JVJP","number":4,"license":"CC BY 4.0","cdate":1700551752385,"readers":["everyone"],"invitations":["ICLR.cc/2024/Conference/Submission6216/-/Official_Comment"],"mdate":1700552007982,"domain":"ICLR.cc/2024/Conference","replyto":"xqtrftUhly","id":"NsNSwuRYM2","forumContent":{"TLDR":{"value":"We present FrugalGPT, an algorithmic framework that adaptively selects which generative LLM APIs to use in order to reduce cost and improve performance."},"venue":{"value":"Submitted to ICLR 2024"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["LLM; GPT-4; joint optimization of performance-cost; MLaaS"]},"supplementary_material":{"value":"/attachment/b75b8be632d4844c128eedead8de2bab645f8c5c.pdf"},"primary_area":{"value":"generative models"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide."},"abstract":{"value":"The rapid adoption of large language models (LLMs) has led to an growing number of companies offering generative LLMs as callable services at varying costs. We find that popular generative LLM APIs, such as GPT-4, ChatGPT, and J1-Jumbo, exhibit heterogeneous pricing structures, with fees that can differ by two orders of magnitude, and heterogeneous performance across tasks and input queries. This makes it challenging for users to decide which generative LLM APIs to utilize for their applications and budget. Motivated by these findings, we propose FrugalGPT, an algorithmic framework that adaptively selects which generative LLMs to use for different queries to reduce cost and improve accuracy. Our experiments demonstrate that, for a range of natural language tasks including news classification, reading comprehension, and scientific question answering, FrugalGPT can match the performance of the best individual generative LLM (e.g., GPT-4) with up to a 98% cost reduction or improve the accuracy over GPT-4 by 4% at the same cost. The ideas and findings presented in this paper lay a foundation for using LLMs sustainably and efficiently."},"_bibtex":{"value":"@misc{\nchen2024frugalgpt,\ntitle={Frugal{GPT}: How to Use Large Language Models While Reducing Cost and Improving Performance},\nauthor={Lingjiao Chen and Matei Zaharia and James Zou},\nyear={2024},\nurl={https://openreview.net/forum?id=XUZ2S0JVJP}\n}"},"title":{"value":"FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance"},"pdf":{"value":"/pdf/44f9acd4b03577917238a232b64b9793f40a8b74.pdf"},"venueid":{"value":"ICLR.cc/2024/Conference/Rejected_Submission"},"paperhash":{"value":"chen|frugalgpt_how_to_use_large_language_models_while_reducing_cost_and_improving_performance"},"authorids":{"value":["~Lingjiao_Chen1","~Matei_Zaharia1","~James_Zou1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors' identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Lingjiao Chen","Matei Zaharia","James Zou"]}},"version":2},{"content":{"summary":{"value":"Authors propose a technique called FrugalGPT which selects which sequence of model APIs to call in order to reduce costs.  FrugalGPT has 3 different trainable components: router, answer-scorer and user budget. In experiments, they find that there is upto a 98% cost reduction, and has similar or slightly better results than using the strongest API used for experiments. In addition, they investigate the effect of data distribution, scorer quality changes and the incidental latency improvements that come along with cost."},"presentation":{"value":"4 excellent"},"contribution":{"value":"4 excellent"},"soundness":{"value":"2 fair"},"strengths":{"value":"- The authors spend time motivating a novel problem faced by practitioners in a clear and engaging manner.\n- The paper is in general well-written and I did not have trouble following the proposed method.\n- The results are compelling and exciting.\n- The analysis done is useful to practitioners (latency improvements, distribution shift, scorer quality)"},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"The main weakness of the paper is the lack of analysis on what makes the method works. This method is relatively complex, with three different components, and there are several natural ablations that have not been done. While it is understandable that there are API costs to contend with, even using open source models like Llama with synthetically assigned costs for ablations could have been done. However, I still believe that the paper should be accepted.\n\nGiven that all related papers that are not speculative decoding are too recent for the authors to take into account (around/after ICLR deadline), it is fine not to have external baselines to compare to (speculative decoding provides better latency, but does not have the same cost improvements, afaik). However, some ablations on the method would be useful:\n- How much does the search space pruning help in practice, versus just ordering the APIs in increasing order of cost? (vs just anecdotes shown in the experimental section)\n- How much changing the threshold T effected results? Is the method very sensitive to T? This would be useful to practitioners.\n\nSome of the design space is indeed constrained by only having access to the output (see comments on scorer in Questions section). I believe writing this in a discussion somewhere in the draft would be helpful in answering questions a reader would have on a first reading."},"confidence":{"value":"5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully."},"questions":{"value":"- Why is a scorer necessary? Given closed APIs don't give probability/logit outputs generally, this is not an option. However, when there is an option, is a simpler proxy like confidence or probability thresholding enough? (like using a much smaller learned calibration module at the query level Eg. https://arxiv.org/abs/2207.07411 https://arxiv.org/abs/2207.05221. A logit based score is more useful for routing at the token level). \n- While interesting, the latency improvments are modest relative to what is possible by combining the use of expensive/larger and cheaper/smaller models (speculative decoding, and https://arxiv.org/abs/2310.12126 get a 2-3x improvement in latency). Do you believe that additionally optimizing for this can result in similar improvements to latency and an improvement in cost (ie, either replacing cost budget with latency budget, or combining the two)? Or is this a limitation of relying on closed source models?"},"rating":{"value":"6: marginally above the acceptance threshold"},"code_of_conduct":{"value":"Yes"}},"nonreaders":[],"tmdate":1700611895207,"tcdate":1699660297232,"writers":["ICLR.cc/2024/Conference","ICLR.cc/2024/Conference/Submission6216/Reviewer_QsiX"],"signatures":["ICLR.cc/2024/Conference/Submission6216/Reviewer_QsiX"],"forum":"XUZ2S0JVJP","number":4,"license":"CC BY 4.0","cdate":1699660297232,"readers":["everyone"],"invitations":["ICLR.cc/2024/Conference/Submission6216/-/Official_Review"],"mdate":1700611895207,"domain":"ICLR.cc/2024/Conference","replyto":"XUZ2S0JVJP","id":"7GqKIUN7Fn","forumContent":{"TLDR":{"value":"We present FrugalGPT, an algorithmic framework that adaptively selects which generative LLM APIs to use in order to reduce cost and improve performance."},"venue":{"value":"Submitted to ICLR 2024"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["LLM; GPT-4; joint optimization of performance-cost; MLaaS"]},"supplementary_material":{"value":"/attachment/b75b8be632d4844c128eedead8de2bab645f8c5c.pdf"},"primary_area":{"value":"generative models"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide."},"abstract":{"value":"The rapid adoption of large language models (LLMs) has led to an growing number of companies offering generative LLMs as callable services at varying costs. We find that popular generative LLM APIs, such as GPT-4, ChatGPT, and J1-Jumbo, exhibit heterogeneous pricing structures, with fees that can differ by two orders of magnitude, and heterogeneous performance across tasks and input queries. This makes it challenging for users to decide which generative LLM APIs to utilize for their applications and budget. Motivated by these findings, we propose FrugalGPT, an algorithmic framework that adaptively selects which generative LLMs to use for different queries to reduce cost and improve accuracy. Our experiments demonstrate that, for a range of natural language tasks including news classification, reading comprehension, and scientific question answering, FrugalGPT can match the performance of the best individual generative LLM (e.g., GPT-4) with up to a 98% cost reduction or improve the accuracy over GPT-4 by 4% at the same cost. The ideas and findings presented in this paper lay a foundation for using LLMs sustainably and efficiently."},"_bibtex":{"value":"@misc{\nchen2024frugalgpt,\ntitle={Frugal{GPT}: How to Use Large Language Models While Reducing Cost and Improving Performance},\nauthor={Lingjiao Chen and Matei Zaharia and James Zou},\nyear={2024},\nurl={https://openreview.net/forum?id=XUZ2S0JVJP}\n}"},"title":{"value":"FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance"},"pdf":{"value":"/pdf/44f9acd4b03577917238a232b64b9793f40a8b74.pdf"},"venueid":{"value":"ICLR.cc/2024/Conference/Rejected_Submission"},"paperhash":{"value":"chen|frugalgpt_how_to_use_large_language_models_while_reducing_cost_and_improving_performance"},"authorids":{"value":["~Lingjiao_Chen1","~Matei_Zaharia1","~James_Zou1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors' identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Lingjiao Chen","Matei Zaharia","James Zou"]}},"version":2},{"content":{"summary":{"value":"FrugalGPT proposes a layer on top of competing LLMs designed to minimize costs without hurting performance or latency, by adaptively selecting LLMs and evaluating their answers with smaller models."},"presentation":{"value":"3 good"},"contribution":{"value":"4 excellent"},"soundness":{"value":"3 good"},"strengths":{"value":"* This is a simple and brilliant idea, with tremendous potential impact for affordable accessibility of AI, minimized environmental impact, and general efficiency.\n* The authors leverage an intuitive observation (answer validation is easier than answer generation) and hierarchy of models to save costs and improve performance together.\n* Demonstration of improved latency is an added benefit"},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"* No justification for the chosen datasets. Do these map to diverse and realistic domains, tasks, and applications? The evaluation selection is fairly important to understand what realistic cost saving ranges are.\n* Evaluation datasets are all short outputs: classification or question answering. This is a slightly limited setting for the claims."},"confidence":{"value":"4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work."},"questions":{"value":"* Are any of the datasets speculated to be in the training set of some of the models? Is there a way to test this?"},"rating":{"value":"8: accept, good paper"},"code_of_conduct":{"value":"Yes"}},"nonreaders":[],"tmdate":1699636677986,"tcdate":1698782349953,"writers":["ICLR.cc/2024/Conference","ICLR.cc/2024/Conference/Submission6216/Reviewer_q9b6"],"signatures":["ICLR.cc/2024/Conference/Submission6216/Reviewer_q9b6"],"forum":"XUZ2S0JVJP","number":2,"license":"CC BY 4.0","cdate":1698782349953,"readers":["everyone"],"invitations":["ICLR.cc/2024/Conference/Submission6216/-/Official_Review","ICLR.cc/2024/Conference/-/Edit"],"mdate":1699636677986,"domain":"ICLR.cc/2024/Conference","replyto":"XUZ2S0JVJP","id":"vYaW36OvzT","forumContent":{"TLDR":{"value":"We present FrugalGPT, an algorithmic framework that adaptively selects which generative LLM APIs to use in order to reduce cost and improve performance."},"venue":{"value":"Submitted to ICLR 2024"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["LLM; GPT-4; joint optimization of performance-cost; MLaaS"]},"supplementary_material":{"value":"/attachment/b75b8be632d4844c128eedead8de2bab645f8c5c.pdf"},"primary_area":{"value":"generative models"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide."},"abstract":{"value":"The rapid adoption of large language models (LLMs) has led to an growing number of companies offering generative LLMs as callable services at varying costs. We find that popular generative LLM APIs, such as GPT-4, ChatGPT, and J1-Jumbo, exhibit heterogeneous pricing structures, with fees that can differ by two orders of magnitude, and heterogeneous performance across tasks and input queries. This makes it challenging for users to decide which generative LLM APIs to utilize for their applications and budget. Motivated by these findings, we propose FrugalGPT, an algorithmic framework that adaptively selects which generative LLMs to use for different queries to reduce cost and improve accuracy. Our experiments demonstrate that, for a range of natural language tasks including news classification, reading comprehension, and scientific question answering, FrugalGPT can match the performance of the best individual generative LLM (e.g., GPT-4) with up to a 98% cost reduction or improve the accuracy over GPT-4 by 4% at the same cost. The ideas and findings presented in this paper lay a foundation for using LLMs sustainably and efficiently."},"_bibtex":{"value":"@misc{\nchen2024frugalgpt,\ntitle={Frugal{GPT}: How to Use Large Language Models While Reducing Cost and Improving Performance},\nauthor={Lingjiao Chen and Matei Zaharia and James Zou},\nyear={2024},\nurl={https://openreview.net/forum?id=XUZ2S0JVJP}\n}"},"title":{"value":"FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance"},"pdf":{"value":"/pdf/44f9acd4b03577917238a232b64b9793f40a8b74.pdf"},"venueid":{"value":"ICLR.cc/2024/Conference/Rejected_Submission"},"paperhash":{"value":"chen|frugalgpt_how_to_use_large_language_models_while_reducing_cost_and_improving_performance"},"authorids":{"value":["~Lingjiao_Chen1","~Matei_Zaharia1","~James_Zou1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors' identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Lingjiao Chen","Matei Zaharia","James Zou"]}},"version":2},{"content":{"comment":{"value":"### 5. Response generation from LLMs\n\nWe believe there may be a misunderstanding regarding our use of the term “*independent sampling*.” By independent sampling, we mean standard **statistical independence conditioned on the input prompt**. Specifically, for a fixed prompt $T=t$, repeated outputs are sampled independently from the conditional distribution $P(X\\vert T=t)$. This is the standard sampling assumption commonly adopted in stochastic and contextual bandit analyses.\n\nImportantly, statistical independence does **not** imply that independently sampled outputs must be different. For example, if $P(X\\vert T=t)$ is a point mass (i.e., has zero variance), independent draws from this distribution will always produce the same outcome, while still being statistically independent. Therefore, similarity or even identity between repeated LLM outputs does not contradict the independent-sampling assumption.\n\nIn our experiments, each repeated query is generated through a **separate API call** using the same fixed prompt and the standard temperature of 1.0. Therefore, our experimental protocol is designed to obtain independent samples from the model’s decoding distribution conditioned on the prompt.\n\n### 6. Experimental Setup\n\nWe have provided the relevant implementation and experimental results through the following anonymous repository: https://anonymous.4open.science/r/tmlr_rebuttal-10000/. We would be happy to provide any additional details about the experimental setup, implementation, or results if the reviewer would find them helpful. \n\n### 7. Including additional baseline\n\nIn addition to the baselines already reported in the text, we report the results of the baseline FrugalGPT. For each task, we train a generation quality scorer using approximately 2.5k prompt-completion pairs. Specifically, the scorer takes as input the CodeBERT representations of the prompt and the generated completion, and predicts the probability that the completion passes the test suite. We adapt the query strategy of FrugalGPT to the cost-aware contextual MAB setup (Protocol 1), where multiple answers can be evaluated for one single prompt. During inference, FrugalGPT first queries the cheapest model in the cascade. If the predicted correctness probability exceeds a selected threshold, the completion is accepted. Otherwise, it queries the next model in the cascade until an answer passes the test suite or the round budget is reached. The results show that FrugalGPT achieves similar performance to the cascade baseline.\n\n\n| Code completion | Success | Cost | Utility ($\\lambda = 0.005$) |\n| --- | --- | --- | --- |\n| **PromptWise-LLR** | 0.96 +- 0.03 | 1.49 +- 0.82 | 0.97 +- 0.01 |\n| **FrugalGPT** | 0.98 +- 0.21 | 3.91 +- 0.83 | 0.95 +- 0.00 |\n\n\n\n### 8. Including results on utility in Table 1\n\nWe have updated Table 1 in the revised paper, where the utility is evaluated at different values of the cost parameter $\\lambda$. The results indicate that PromptWise performs more successfullly compared to the baselines."},"title":{"value":"Response to Reviewer CAxJ (Cont'd)"}},"parentInvitations":"TMLR/-/Official_Comment","tmdate":1786880192059,"tcdate":1786880192059,"writers":["TMLR","TMLR/Paper10000/Authors"],"signatures":["TMLR/Paper10000/Authors"],"forum":"mjkSi2oFLX","number":2,"license":"CC BY 4.0","cdate":1786880192059,"readers":["everyone"],"invitations":["TMLR/Paper10000/-/Official_Comment"],"mdate":1786880192059,"domain":"TMLR","replyto":"iUC8Suzz8z","id":"B2SWIivtdz","forumContent":{"submission_length":{"value":"Regular submission (no more than 12 pages of main content)"},"venue":{"value":"Rejected by TMLR"},"abstract":{"value":"The rapid advancement of generative AI has provided users with a wide range of well-trained models to address diverse prompts, with substantial variation in both performance and service cost. When selecting a model for a given prompt, users should therefore consider not only expected output quality but also the monetary cost of querying the model. Many existing approaches assume fixed or single-shot model assignment per prompt, whereas practical deployments may require online adaptation and cost accumulation across multiple attempts. In this paper, we introduce PromptWise, an online learning framework for cost-aware prompt assignment that learns from sequential interactions between prompts and models. PromptWise maintains and continuously updates estimates of prompt–model compatibility based on observed feedback, and uses these estimates to make adaptive, cost-sensitive decisions under uncertainty. Unlike standard contextual bandits that make a one-shot decision per prompt, PromptWise employs a cost-aware contextual bandit structure that allows sequential model assignments per prompt and explicitly accounts for the cumulative service cost incurred across multiple attempts. Through numerical experiments on tasks such as code generation and translation, we demonstrate that PromptWise achieves performance comparable to baseline selection methods while substantially reducing overall service cost."},"_bibtex":{"value":"@article{\nanonymous2026promptwise,\ntitle={PromptWise: Extending Contextual Bandits to Multi-Round Prompt Routing for Cost-Efficient {LLM} Inference},\nauthor={Anonymous},\njournal={Submitted to Transactions on Machine Learning Research},\nyear={2026},\nurl={https://openreview.net/forum?id=mjkSi2oFLX},\nnote={Rejected}\n}"},"title":{"value":"PromptWise: Extending Contextual Bandits to Multi-Round Prompt Routing for Cost-Efficient LLM Inference"},"pdf":{"value":"/pdf/02ca4464deacbc7d1c17b1fb95095471f47e83d1.pdf"},"venueid":{"value":"TMLR/Rejected"},"assigned_action_editor":{"value":"~Cong_Shen1"}},"version":2},{"content":{"comment":{"value":"We thank you again for your reviews and the positive feedback. We greatly appreciate your support of our ideas and have revised the paper as per your suggestions. Specifically, in Section 2.1 (Page 4), we have added a detailed discussion to compare our approach with respect to FrugalGPT [1] and speculative decoding [2,3].\n\nWe hope we have answered your questions to your satisfaction and hope you would consider increasing the score.\n\nReference:  \n[1] Chen, Lingjiao, Matei Zaharia, and James Zou. \"FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance.\" arXiv preprint arXiv:2305.05176 (2023).  \n[2] Leviathan, Yaniv, Matan Kalman, and Yossi Matias. \"Fast inference from transformers via speculative decoding.\" International Conference on Machine Learning. PMLR, 2023.  \n[3] Kim, Sehoon, et al. \"Speculative Decoding with Big Little Decoder.\" Thirty-seventh Conference on Neural Information Processing Systems. 2023."},"title":{"value":"To Reviewer 9pHQ - Revision is Ready"}},"tmdate":1700683147191,"tcdate":1700682527124,"writers":["ICLR.cc/2024/Conference","ICLR.cc/2024/Conference/Submission7318/Authors"],"signatures":["ICLR.cc/2024/Conference/Submission7318/Authors"],"forum":"02f3mUtqnM","number":7,"license":"CC BY 4.0","cdate":1700682527124,"readers":["everyone"],"invitations":["ICLR.cc/2024/Conference/Submission7318/-/Official_Comment"],"mdate":1700683147191,"domain":"ICLR.cc/2024/Conference","replyto":"01SJ6h5IHh","id":"bsJnrBdOWq","forumContent":{"venue":{"value":"ICLR 2024 poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Large language models","Efficient ML","Query Routing"]},"primary_area":{"value":"general machine learning (i.e., none of the above)"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide."},"abstract":{"value":"Large language models (LLMs) excel in most NLP tasks but also require expensive cloud servers for deployment due to their size, while smaller models that can be deployed on lower cost (e.g., edge) devices, tend to lag behind in terms of response quality. Therefore in this work we propose a hybrid inference approach which combines their respective strengths to save cost and maintain quality. Our approach uses a router that assigns queries to the small or large model based on the predicted query difficulty and the desired quality level. The desired quality level can be tuned dynamically at test time to seamlessly trade  quality for cost as per the scenario requirements. In experiments our approach allows us to make up to 40% fewer calls to the large model, with no drop in response quality."},"_bibtex":{"value":"@inproceedings{\nding2024hybrid,\ntitle={Hybrid {LLM}: Cost-Efficient and Quality-Aware Query Routing},\nauthor={Dujian Ding and Ankur Mallick and Chi Wang and Robert Sim and Subhabrata Mukherjee and Victor R{\\\"u}hle and Laks V. S. Lakshmanan and Ahmed Hassan Awadallah},\nbooktitle={The Twelfth International Conference on Learning Representations},\nyear={2024},\nurl={https://openreview.net/forum?id=02f3mUtqnM}\n}"},"title":{"value":"Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing"},"pdf":{"value":"/pdf/18ea688c270f3fb84befcc014b7228d72ce47eda.pdf"},"venueid":{"value":"ICLR.cc/2024/Conference"},"paperhash":{"value":"ding|hybrid_llm_costefficient_and_qualityaware_query_routing"},"authorids":{"value":["~Dujian_Ding1","~Ankur_Mallick1","~Chi_Wang3","~Robert_Sim1","~Subhabrata_Mukherjee2","~Victor_Rühle1","~Laks_V._S._Lakshmanan1","~Ahmed_Hassan_Awadallah1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors' identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Dujian Ding","Ankur Mallick","Chi Wang","Robert Sim","Subhabrata Mukherjee","Victor Rühle","Laks V. S. Lakshmanan","Ahmed Hassan Awadallah"]}},"version":2},{"content":{"comment":{"value":"We have extended our experiments to include the FrugalGPT baseline and ran the experiments for 150 rounds instead of 100 rounds for both telecom and medical settings. These results could be found in the updated pdf document."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764797248997,"tcdate":1764795620341,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission13581/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission13581/Authors"],"forum":"EiJbxZTtDj","number":7,"license":"CC BY 4.0","cdate":1764795620341,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission13581/-/Official_Comment"],"mdate":1764797248997,"domain":"ICLR.cc/2026/Conference","replyto":"EiJbxZTtDj","id":"JSzsIG1q15","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["neural bandits","online LLM selection"]},"supplementary_material":{"value":"/attachment/a9a80a4aff5ba151c2f87476eb660c59f0e0a5da.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"With the increasing popularity of large language models (LLMs) for a variety of tasks, there has been a growing interest in strategies that can predict which out of a set of LLMs will yield a successful answer at low cost. This problem promises to become more and more relevant as providers like Microsoft allow users to easily create custom LLM \"assistants'' specialized to particular types of queries. However, some tasks (i.e., queries) may benefit from breaking down the task into smaller subtasks, each of which can then be executed by a LLM expected to perform well on that specific subtask. For example, in extracting a diagnosis from medical records, one can first select an LLM to summarize the record, select another to validate the summary, and then select another, possibly different, LLM to extract the diagnosis from the summarized record. Unlike existing LLM selection or routing algorithms, this setting requires selecting a sequence of LLMs, with the output of each LLM feeding into the next and potentially influencing its success. Thus, unlike single LLM selection, the quality of each subtask's output directly affects the inputs, and hence the cost and success rate, of downstream LLMs, creating complex performance dependencies that must be learned during selection. We propose a neural contextual bandit-based algorithm that trains neural networks that model LLM success on each subtask in an online manner, thus learning to guide the LLM selections for the different subtasks, even in the absence of historical LLM performance data. Experiments on telecommunications question answering and medical diagnosis prediction datasets illustrate the effectiveness of our proposed approach compared to other LLM selection algorithms."},"_bibtex":{"value":"@misc{\natalar2026neural,\ntitle={Neural Bandit Based Optimal {LLM} Selection for a Pipeline of Tasks},\nauthor={Baran Atalar and Eddie Zhang and Carlee Joe-Wong},\nyear={2026},\nurl={https://openreview.net/forum?id=EiJbxZTtDj}\n}"},"title":{"value":"Neural Bandit Based Optimal LLM Selection for a Pipeline of Tasks"},"pdf":{"value":"/pdf/0ea0038c815706f637e52253b61de3097ff3ae5b.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"atalar|neural_bandit_based_optimal_llm_selection_for_a_pipeline_of_tasks"},"authorids":{"value":["~Baran_Atalar1","~Eddie_Zhang3","~Carlee_Joe-Wong1"]},"authors":{"value":["Baran Atalar","Eddie Zhang","Carlee Joe-Wong"]}},"version":2},{"content":{"comment":{"value":"**Table3: Comparison between different methods under two simulation strategies in the small-scale experimental setting. The Improvement is measured relative to the best baseline.**\n\n| Scenario&nbsp; | Multi-cost-efficiency Simulation      | LLM-as-a-Judge    |\n|----------------|---------------------------------------|-------------------|\n| Method     | Reward &nbsp;  Improvement (%) | Accuracy &nbsp;  Improvement (%)  |\n| HybridLLM  | 0.141  &emsp;&emsp;&emsp; -36.20           | 0.347  &emsp;&emsp;&emsp;   -14.74            |\n| FrugalGPT  | 0.218  &emsp;&emsp;&emsp; -1.36            | 0.354  &emsp;&emsp;&emsp;   -13.22            |\n| GraphRouter| 0.221  &emsp;&emsp;&emsp; 0.00             | 0.364  &emsp;&emsp;&emsp;   -10.57            |\n| RouterDC   | 0.208  &emsp;&emsp;&emsp; -5.88            | 0.407  &emsp;&emsp;&emsp;   0.00              |\n| Ours       | 0.255  &emsp;&emsp;&emsp; 15.38            | 0.447  &emsp;&emsp;&emsp;   9.83              |\n| Oracle     | 0.304  &emsp;&emsp;&emsp; 37.56            | 1.000  &emsp;&emsp;&emsp;   145.70            |\n\n\n**Table4: Comparison between different methods under simulation strategies in the new user experimental setting. The Improvement is measured relative to the best baseline.** \n\n| Scenario         | multi-cost-efficiency simulation             | LLM-as-a-Judge strategy                |\n|-------------     |----------------------------------------------|----------------------------------------|\n| Method           | Reward  &nbsp; Improvement (%) | Accuracy   Improvement (%)  |\n| HybridLLM        | -0.142   &emsp;&emsp;&emsp;   -240.59         | 0.294  &emsp;&emsp;&emsp;&ensp;    0.00             |\n| FrugalGPT        | 0.044    &emsp;&emsp;&emsp;   -56.44          | 0.192  &emsp;&emsp;&emsp;   -34.69            |\n| GraphRouter      | 0.083    &emsp;&emsp;&emsp;   -17.82          | 0.256  &emsp;&emsp;&emsp;   -12.93            |\n| RouterDC         | 0.101    &emsp;&emsp;&emsp;&ensp;    0.00           | 0.208  &emsp;&emsp;&emsp;   -29.25            |\n| Ours (Few-shots) | 0.070    &emsp;&emsp;&emsp;  -30.69           | 0.313  &emsp;&emsp;&emsp;&ensp;    6.46             |\n| Ours (Trained)   | 0.108    &emsp;&emsp;&emsp;&ensp;   6.93            | 0.326  &emsp;&emsp;&emsp;&ensp;   10.88             |\n| Oracle           | 0.116    &emsp;&emsp;&emsp;&ensp;   14.85           | 1.000  &emsp;&emsp;&emsp;&ensp;   240.14  \n\n**Table5: Comparison between different methods under simulation strategies in the new LLM experimental setting. The Improvement is measured relative to the best baseline.** \n\n| Scenario        | multi-cost-efficiency simulation             | LLM-as-a-Judge strategy                |\n|-------------    |----------------------------------------------|----------------------------------------|\n| Method          | Reward &nbsp;  Improvement (%) | Accuracy &nbsp;  Improvement (%)  |\n| HybridLLM       | 0.137 &emsp;&emsp;&emsp;  -37.16           | 0.273  &emsp;&emsp;&emsp;    0.00             |\n| FrugalGPT       | 0.182 &emsp;&emsp;&emsp;  -16.51           | 0.038  &emsp;&emsp;&emsp;   -86.08            |\n| GraphRouter     | 0.203 &emsp;&emsp;&emsp;  -6.88            | 0.247  &emsp;&emsp;&emsp;   -9.52             |\n| RouterDC        | 0.218 &emsp;&emsp;&emsp;   0.00            | 0.199  &emsp;&emsp;&emsp;  -27.10             |\n| Ours (few-shot) | 0.201 &emsp;&emsp;&emsp;  -7.80            | 0.329  &emsp;&emsp;&emsp;   20.51             |\n| Ours (trained)  | 0.234 &emsp;&emsp;&emsp;   7.34            | 0.383  &emsp;&emsp;&emsp;   40.29             |\n| Oracle          | 0.371 &emsp;&emsp;&emsp;  70.18            | 1.000  &emsp;&emsp;&emsp;  266.30             |"},"title":{"value":"Response to Reviewer h7mj (Part 3)"}},"parentInvitations":"TMLR/-/Official_Comment","tmdate":1758313289909,"tcdate":1758313214242,"writers":["TMLR","TMLR/Paper5500/Authors"],"signatures":["TMLR/Paper5500/Authors"],"forum":"W80eE3ArAl","number":9,"license":"CC BY 4.0","cdate":1758313214242,"readers":["everyone"],"invitations":["TMLR/Paper5500/-/Official_Comment"],"mdate":1758313289909,"domain":"TMLR","replyto":"WfA7D74GQL","id":"KX1BcZex34","forumContent":{"submission_length":{"value":"Regular submission (no more than 12 pages of main content)"},"venue":{"value":"Accepted by TMLR"},"code":{"value":"https://github.com/ulab-uiuc/PersonalizedRouter"},"supplementary_material":{"value":"/attachment/e4bbeecf2accf23bec60f721de3508b2cf488096.zip"},"abstract":{"value":"The growing number of Large Language Models (LLMs) with diverse capabilities and response styles provides users with a wider range of choices, which presents challenges in selecting appropriate LLMs, as user preferences vary in terms of performance, cost, and response style. Current LLM selection methods typically optimize for a single fixed objective, such as performance, cost, or a trade-off between them, and fail to learn individual user preferences from interaction data. To address these limitations, we propose PersonalizedRouter, a graph-based framework that models diverse user profiles and performs personalized LLM selection by leveraging interaction data that includes task context, queries, candidate LLMs, and user decisions. To capture contextual information between user queries and optimal LLMs, PersonalizedRouter converts the interaction data into a heterogeneous graph, where the relationships between different types of nodes are represented by edges. To evaluate adaptability across users, we design two strategies: the multi-cost-efficiency simulation strategy and the LLM-as-a-Judge strategy. In addition, we construct PersonaRoute-Bench, a large-scale benchmark with 1,000 simulated users and 10 LLMs. Experimental results show that PersonalizedRouter significantly outperforms existing LLM selection methods and surpasses the strongest methods by a large margin of 15.38% and 9.83% under two simulation strategies. On the PersonaRoute-Bench with 1,000 users, it further surpasses the best methods by 16.19% and 59.69% while maintaining higher efficiency. Moreover, PersonalizedRouter demonstrates strong few-shot generalization, achieving 64.81% and 85.80% of the fully trained model’s performance when adapting to new users and new LLMs."},"_bibtex":{"value":"@article{\ndai2025personalizedrouter,\ntitle={PersonalizedRouter: Personalized {LLM} Routing via Graph-based User Preference Modeling},\nauthor={Zhongjie Dai and Tao Feng and Jiaxuan You},\njournal={Transactions on Machine Learning Research},\nissn={2835-8856},\nyear={2025},\nurl={https://openreview.net/forum?id=W80eE3ArAl},\nnote={}\n}"},"title":{"value":"PersonalizedRouter: Personalized LLM Routing via Graph-based User Preference Modeling"},"pdf":{"value":"/pdf/1c7d7c4a7a37c3cd71672b102988c3788f67c8e7.pdf"},"venueid":{"value":"TMLR"},"paperhash":{"value":"dai|personalizedrouter_personalized_llm_routing_via_graphbased_user_preference_modeling"},"authorids":{"value":["~Zhongjie_Dai1","~Tao_Feng5","~Jiaxuan_You2"]},"assigned_action_editor":{"value":"~Yen-Chang_Hsu1"},"authors":{"value":["Zhongjie Dai","Tao Feng","Jiaxuan You"]}},"version":2},{"content":{"TLDR":{"value":"We adopt multiple LLM services to match GPT-4's performance with up to 98% cost savings"},"venue":{"value":"ES-FoMO 2023 Poster"},"pdf":{"value":"/pdf/93b99426fb1722bc6c6648f6d1a5e59a81fb880e.pdf"},"keywords":{"value":["LLM","inference","cloud AI market"]},"venueid":{"value":"ICML.cc/2023/Workshop/ES-FoMO"},"paperhash":{"value":"chen|less_is_more_using_multiple_llms_for_applications_with_lower_costs"},"authorids":{"value":["~Lingjiao_Chen1","~Matei_Zaharia1","~James_Zou1"]},"abstract":{"value":"Large language models (LLMs) are increasingly used for querying purposes, but their associated costs vary significantly. This study investigates the pricing structures of popular LLM APIs, such as GPT-4, ChatGPT, and J1-Jumbo, revealing sub- stantial fee differences. To mitigate the expense of using LLMs on extensive queries and text, we propose three strategies: prompt adaptation, LLM approximation, and LLM cascade. We present FrugalGPT, an adaptable LLM cascade that in- telligently selects LLM combinations to reduce costs by up to 98% while matching or improving the accuracy of individual LLMs. This work es- tablishes a foundation for sustainable and efficient LLM utilization, offering valuable insights and practical techniques for users."},"_bibtex":{"value":"@inproceedings{\nchen2023less,\ntitle={Less is More: Using Multiple {LLM}s for Applications with Lower Costs},\nauthor={Lingjiao Chen and Matei Zaharia and James Zou},\nbooktitle={Workshop on Efficient Systems for Foundation Models @ ICML2023},\nyear={2023},\nurl={https://openreview.net/forum?id=TkXjqcwQ4s}\n}"},"title":{"value":"Less is More: Using Multiple LLMs for Applications with Lower Costs"},"authors":{"value":["Lingjiao Chen","Matei Zaharia","James Zou"]}},"tmdate":1689519095385,"pdate":1687252026944,"tcdate":1685599476088,"writers":["ICML.cc/2023/Workshop/ES-FoMO","ICML.cc/2023/Workshop/ES-FoMO/Submission54/Authors"],"signatures":["ICML.cc/2023/Workshop/ES-FoMO/Submission54/Authors"],"forum":"TkXjqcwQ4s","number":54,"cdate":1685599476088,"mdate":1689519095385,"readers":["everyone"],"invitations":["ICML.cc/2023/Workshop/ES-FoMO/-/Submission","ICML.cc/2023/Workshop/ES-FoMO/-/Post_Submission","ICML.cc/2023/Workshop/ES-FoMO/-/Edit"],"odate":1689519095372,"domain":"ICML.cc/2023/Workshop/ES-FoMO","id":"TkXjqcwQ4s","version":2},{"content":{"summary":{"value":"The paper introduces E-Agent, a cost-efficient agentic framework for knowledge-intensive and reasoning tasks. It follows an executor–verifier paradigm, where multiple local-based executors generate candidate answers that are subsequently validated by a cloud-based verifier. Experiments on GSM8K, ALFWorld, HotpotQA, and StrategyQA demonstrate that E-Agent reduces token usage by 10%–50% compared with strong baselines such as ReAct, while maintaining accuracy or even improving it."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"1. How does the framework ensure that executors produce structured outputs compliant with the JSON/Schema format? How does the system robustly handle cases where the output format is incorrect?\n2. When the verifier needs to “regenerate an answer” rather than simply verify one, what are the common failure modes? Could you provide some examples?"},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. Cost-efficient agentic architecture exploiting pricing asymmetry\nThe proposed E-Agent framework explicitly leverages the input–output pricing asymmetry of commercial LLMs to reduce overall token expenditure by shifting computation from expensive output decoding to cheaper input prefilling, achieving 10%–50% cost savings across benchmarks.\n2. Experimental evidence (Table 3) is compelling\n E-Agent consistently reduces aggregate token and monetary costs by a meaningful margin (10–50%) relative to strong baselines, while accuracy is maintained or improved. These results are corroborated by runtime measurements (Figure 3)."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. Lack of Methodological Originality\n\nThe proposed E-Agent framework lacks sufficient originality. Using structured outputs as inputs to cloud models to reduce token usage and improve accuracy is not a novel idea, as similar techniques have been widely adopted in agentic LLM prompt designs (e.g., A-Mem [1]). Likewise, employing multiple local LLMs for cost optimization has already been demonstrated in FrugalGPT [2]. The paper is encouraged to clarify its methodological novelty compared with these prior works.\n\n2. Insufficient Discussion of Related Work\n\nThe “Related Work” section does not adequately compare E-Agent with closely related recent studies (e.g., KnowAgent [3]). As a result, the paper’s positioning within the landscape of cost-efficient agentic framework design remains unclear.\n\n3. Lack of Cost-Efficient Baselines\nAlthough several cost-reduction methods such as LLMLingua[4] and FrugalGPT[2] (which provide open-source implementations) are mentioned in the related work section, they are not included as baselines in the experiments. This omission weakens the empirical validation of E-Agent’s cost efficiency.\n\n4. Missing Theoretical Analysis of Failure Modes\n\nWhile the paper motivates cost reduction by shifting computation to input tokens, it does not model or analyze failure cases when all local executors fail. In such scenarios, the cloud verifier must regenerate answers from scratch. The frequency and impact of these failures are not quantified, nor are worst-case cost or robustness metrics provided.\n\n5. Vague System Description\n\nThe description of the Cost-Efficient Agentic framework remains vague. The equations in Section 3.2 are mostly symbolic process representations and lack formal definitions of key implementation details such as candidate generation, structured output constraints, and task assignment. It would be helpful to elaborate on how user inputs are mapped to executor tasks and how the cloud LLM coordinates with or supervises different executors.\n\n6. Unbalanced Experimental Baselines\n\nAlthough the related work section discusses token-efficient methods like FrugalGPT[2], the experiments only compare E-Agent against reasoning-oriented baselines such as ReAct. Including additional comparisons with token-efficient agentic frameworks would strengthen the empirical evaluation and clarify the claimed advantages.\n\n7. Minor Typographical Errors\n\nThere are small spelling mistakes in the paper, such as “Ouput” → “Output” in Table 5 and “StrategrQA” → “StrategyQA” in Figure 3.\n\nReferences\n\n[1] Wujiang Xu, Zujie Liang, and Kai Mei. A-Mem: Agentic Memory for LLM Agents. NeurIPS, 2025.\n\n[2] Lingjiao Chen, Matei Zaharia, and James Zou. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. Trans. Mach. Learn. Res., 2024.\n\n[3] Yuqi Zhu, Shuofei Qiao, and Yixin Ou. KnowAgent: Knowledge-Augmented Planning for LLM-Based Agents. ACL, 2025.\n\n[4] Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models. EMNLP, 2023."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762927866688,"tcdate":1761910378276,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission18095/Reviewer_u63t"],"signatures":["ICLR.cc/2026/Conference/Submission18095/Reviewer_u63t"],"forum":"qmYLlDg9kQ","number":2,"license":"CC BY 4.0","cdate":1761910378276,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission18095/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762927866688,"domain":"ICLR.cc/2026/Conference","replyto":"qmYLlDg9kQ","id":"hn01YSQerK","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["agent","token","cost","efficient"]},"primary_area":{"value":"other topics in machine learning (i.e., none of the above)"},"abstract":{"value":"The deployment of large language model (LLM)-powered agents for knowledge-intensive and reasoning tasks is often prohibitively expensive, since processing large volumes of evidence incurs massive token costs. Existing techniques such as prompt compression and model routing attempt to reduce token usage, but they often compromise accuracy or fail to capture the fine-grained structure of reasoning tasks. In this work, we introduce E-Agent, a cost-effective framework that leverages the pricing asymmetry of LLMs to significantly reduce monetary cost without sacrificing performance. E-Agent adopts an executor–verifier paradigm: multiple small, locally deployed models act as executors to generate candidate answers, which are then verified by a powerful cloud-based model. This design shifts token consumption from expensive outputs to relatively cheaper inputs. The framework further supports specialized workflows for both retrieval-augmented generation (RAG) and non-RAG tasks, and employs structured outputs to minimize candidate answer length. Experiments on GSM8K, ALFWorld, HotpotQA, and StrategyQA demonstrate that E-Agent reduces token usage by 10\\%–50\\% compared with strong baselines, while maintaining or even improving accuracy."},"_bibtex":{"value":"@misc{\nliu2026eagent,\ntitle={E-Agent: A Cost-Efficient Agentic Framework for Knowledge-Intensive and Reasoning Tasks},\nauthor={Yu Liu and Mengbai Xiao and Jing Qiao and YUAN YUAN and Xiao Zhang and Guanghui Zhang and Dongxiao Yu},\nyear={2026},\nurl={https://openreview.net/forum?id=qmYLlDg9kQ}\n}"},"title":{"value":"E-Agent: A Cost-Efficient Agentic Framework for Knowledge-Intensive and Reasoning Tasks"},"pdf":{"value":"/pdf/60687d4df2cff3455dc8951f1130d1a649a3bae6.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"liu|eagent_a_costefficient_agentic_framework_for_knowledgeintensive_and_reasoning_tasks"},"authorids":{"value":["~Yu_Liu54","~Mengbai_Xiao2","~Jing_Qiao1","~YUAN_YUAN21","~Xiao_Zhang18","~Guanghui_Zhang2","~Dongxiao_Yu1"]},"authors":{"value":["Yu Liu","Mengbai Xiao","Jing Qiao","YUAN YUAN","Xiao Zhang","Guanghui Zhang","Dongxiao Yu"]}},"version":2},{"content":{"summary":{"value":"This paper proposes Selective Deferred Routing, a paradigm that enables cost-efficient collaboration between local SLMs and remote LLMs. In this framework, a user request is first processed by the local SLM, which not only generates a preliminary response but also provides rich semantic representations of the request. A lightweight decider module then leverages this information to either adopt the initial response or route the request in a single step to the most suitable remote LLM for a higher-quality response."},"soundness":{"value":2},"confidence":{"value":3},"questions":{"value":"In GSM8K Figure 4, why using Llama-4-Maverick as the LLM? How about GPT and Deepseek?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. Multi-LLM Routing is important and on time.\n\n2. Selective Deferred Routing balances cost and accuracy.\n\n3. Extensive experiments on 5 LLMs and 3 datasets are provided."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The latency is not reported. Judge after SLM finishes generation can take very long. After SLM generates a few tokens, it might be sufficient to start routing.\n\n2. Fine tuning a BERT before SLM and LLMs may have lower latency and cost.\n\n3. In GSM8K Figure 4, the performance is close to FrugalGPT. Code is also not provided."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762919068544,"tcdate":1762153223874,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission6798/Reviewer_cQJE"],"signatures":["ICLR.cc/2026/Conference/Submission6798/Reviewer_cQJE"],"forum":"uSMttGgyuZ","number":3,"license":"CC BY 4.0","cdate":1762153223874,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission6798/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762919068544,"domain":"ICLR.cc/2026/Conference","replyto":"uSMttGgyuZ","id":"r74htWv0HA","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"TLDR":{"value":"To achieve high performance with limited monetary cost with collaboration of local SLMs and remote LLMs."},"keywords":{"value":["LLM Routing","Cost-Efficiency","Local-Remote Collaboration"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"The rapid advancement of large language models (LLMs) has led to remarkable performance across diverse domains such as question answering, creative writing, programming, etc., making them indispensable assistants in daily life and work. Currently, LLM services are primarily accessed in two ways: (i) paid access to cloud-hosted LLMs, which are powerful but introduce nontrivial cost; and (ii) deployment of small language models (SLMs) on personal devices or small clusters, which, while less powerful, are sufficient for handling relatively simple tasks. To achieve a balanced trade-off between monetary cost and task performance, we propose Selective Deferred Routing, a paradigm that enables cost-efficient collaboration between local SLMs and remote LLMs. In this framework, a user request is first processed by the local SLM, which not only generates a preliminary response but also provides rich semantic representations of the request. A lightweight decider module then leverages this information to either adopt the initial response or route the request in a single step to the most suitable remote LLM for a higher-quality response. Extensive experiments on 5 LLMs and 3 datasets demonstrate that our approach consistently outperforms existing multi-LLM collaboration methods across different cost–performance trade-off preferences."},"_bibtex":{"value":"@misc{\nmiao2026selective,\ntitle={Selective Deferred Routing: Enabling Cost-Efficient Collaboration between Local {SLM}s and Remote {LLM}s},\nauthor={Qijun Miao and Zhixuan Fang},\nyear={2026},\nurl={https://openreview.net/forum?id=uSMttGgyuZ}\n}"},"title":{"value":"Selective Deferred Routing: Enabling Cost-Efficient Collaboration between Local SLMs and Remote LLMs"},"pdf":{"value":"/pdf/53c64af1a280cc8a3b45cf0c65c03f4c04ee44ec.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"miao|selective_deferred_routing_enabling_costefficient_collaboration_between_local_slms_and_remote_llms"},"authorids":{"value":["~Qijun_Miao1","~Zhixuan_Fang1"]},"authors":{"value":["Qijun Miao","Zhixuan Fang"]}},"version":2},{"content":{"comment":{"value":"We sincerely thank Reviewer 6vjE for the thoughtful feedback on our work. Below is our response to the reviewer's comments and questions.\n\n### 1. Clarification of theoretical statements and related proofs\n\n* **Regarding Theorems 1 and 2:** Thank you for pointing this out. Both Theorems 1 and 2 focus on the same regret guarantee for the Epoch-PromptWise (Algorithm 2), the PromptWise algorithm with per-prompt updates. Our intention was to state the regret bound concisely in the main text, while deferring the explicit choices of the parameters, including $\\tau_{\\exp}$, $\\alpha$, and $\\tau_{\\max}$, to the Appendix, which rely on quantities and lemmas introduced in the theoretical analysis. To address the reviewer's comment, we have reorganized the theoretical presentation. There is now a single theorem in the main text and we deferred its proof to Appendix B.1.\n* **Regarding Appendix B.2:** We added detailed derivation and the concavity statement to Appendix B.2.\n* **Regarding Proposition 2:** We note that Proposition 2 is not used in the proof and we have omitted it from the revised text.\n\n\n### 2. Oracle and FrugalGPT baselines\n\nWe have included results for the oracle baseline and the FrugalGPT baseline in the code completion task and the code translation task.\n\n| Code completion (Fig. 3) | Success (%) | Cost | Utility ($\\lambda = 0.005$) |\n| --- | --- | --- | --- |\n| **PromptWise-LLR** | 0.96 +- 0.03 | 1.49 +- 0.72 | 0.97 +- 0.01 |\n| **PromptWise-Oracle (Prop.1)** | 0.98 +- 0.21 | 0.82 +- 0.18 | 0.98 +- 0.00 |\n| **FrugalGPT** | 0.98 +- 0.21 | 3.95 +- 0.88 | 0.96 +- 0.00 |\n\n| Code translation (Fig. 4) | Success (%) | Cost | Utility ($\\lambda = 0.005$) |\n| --- | --- | --- | --- |\n| **PromptWise-LLR** | 0.74 +- 0.17 | 3.11 +- 0.66 | 0.71 +- 0.05 |\n| **PromptWise-Oracle (Prop.1)** | 0.77 +- 0.17 | 1.81 +- 0.37 | 0.76 +- 0.01 |\n| **FrugalGPT** | 0.76 +- 0.17 | 28.00 +- 6.22 | 0.63 +- 0.02 |\n\n### 3. Summary of ablation study on hyper-parameters\n\nWe conduct ablation study the cost parameter and the round budget for the PromptWise algorithm. We summarize our findings as follows, which have been included in the revised manuscript in Appendix C.2.\n\n* **Cost parameter:** The results show that a relatively small cost parameter, i.e., $\\lambda \\le 0.01$, results in a better trade-off between the success rate and model costs. When the cost parameter is too large, e.g., $\\lambda = 0.05$, the algorithm only queries the cheapest model (or even skips the prompt without trying any model), and the success rate can drop substantially.\n* **Round budget:** As our results indicate, a round budget between 4 and 8 can robustly lead to satisfactory results in terms of both the cost and performance metrics.\n\n### 4. Analysis of statistical uncertainty\n\nWe thank the reviewer for the suggestion. We provide the analysis in the following tables for Figures 3 and 4 in the main text.\n\n| Success | Code completion (Fig. 3) | Code translation (Fig. 4) | \n| --- | --- | --- |\n| **PromptWise-LLR** | 0.96 +- 0.03 | 0.74 +- 0.17 |\n| **PromptWise-KLR** | 0.98 +- 0.04 | 0.67 +- 0.14 |\n| **Cascade** | 0.98 +- 0.21 | 0.76 +- 0.17 |\n| **CA PAK-UCB-tS** | 0.92 +- 0.10 | 0.73 +- 0.16 |\n| **Random** | 0.75 +- 0.16 | 0.62 +- 0.13 |\n| **Greedy** | 0.90 +- 0.20 | 0.71 +- 0.16 |\n\n| Cost | Code completion (Fig. 3) | Code translation (Fig. 4) | \n| --- | --- | --- |\n| **PromptWise-LLR** | 1.49 +- 0.82 | 3.11 +- 0.66 |\n| **PromptWise-KLR** | 1.01 +- 0.02 | 2.63 +- 0.59 |\n| **Cascade** | 3.71 +- 0.76 | 27.02 +- 6.01 |\n| **CA PAK-UCB-tS** | 11.12 +- 0.55 | 26.35 +- 6.88 |\n| **Random** | 21.16 +- 1.03 | 21.16 +- 1.03 |\n| **Greedy** | 1.43 +- 0.56 | 10.44 +- 0.32 |\n\n| Utility ($\\lambda=0.005$) | Code completion (Fig. 3) | Code translation (Fig. 4) | \n| --- | --- | --- |\n| **PromptWise-LLR** | 0.97 +- 0.01 | 0.71 +- 0.05 |\n| **PromptWise-KLR** | 0.97 +- 0.00 | 0.66 +- 0.01 |\n| **Cascade** | 0.96 +- 0.00 | 0.63 +- 0.02 |\n| **CA PAK-UCB-tS** | 0.81 +- 0.14 | 0.58 +- 0.11 |\n| **Random** | 0.57 +- 0.01 | 0.52 +- 0.02 |\n| **Greedy** | 0.88 +- 0.03 | 0.65 +- 0.12 |"},"title":{"value":"Response to Reviewer 6vjE"}},"parentInvitations":"TMLR/-/Official_Comment","tmdate":1786880371074,"tcdate":1786880371074,"writers":["TMLR","TMLR/Paper10000/Authors"],"signatures":["TMLR/Paper10000/Authors"],"forum":"mjkSi2oFLX","number":4,"license":"CC BY 4.0","cdate":1786880371074,"readers":["everyone"],"invitations":["TMLR/Paper10000/-/Official_Comment"],"mdate":1786880371074,"domain":"TMLR","replyto":"rs8uoIjC1z","id":"AINOIMw5rX","forumContent":{"submission_length":{"value":"Regular submission (no more than 12 pages of main content)"},"venue":{"value":"Rejected by TMLR"},"abstract":{"value":"The rapid advancement of generative AI has provided users with a wide range of well-trained models to address diverse prompts, with substantial variation in both performance and service cost. When selecting a model for a given prompt, users should therefore consider not only expected output quality but also the monetary cost of querying the model. Many existing approaches assume fixed or single-shot model assignment per prompt, whereas practical deployments may require online adaptation and cost accumulation across multiple attempts. In this paper, we introduce PromptWise, an online learning framework for cost-aware prompt assignment that learns from sequential interactions between prompts and models. PromptWise maintains and continuously updates estimates of prompt–model compatibility based on observed feedback, and uses these estimates to make adaptive, cost-sensitive decisions under uncertainty. Unlike standard contextual bandits that make a one-shot decision per prompt, PromptWise employs a cost-aware contextual bandit structure that allows sequential model assignments per prompt and explicitly accounts for the cumulative service cost incurred across multiple attempts. Through numerical experiments on tasks such as code generation and translation, we demonstrate that PromptWise achieves performance comparable to baseline selection methods while substantially reducing overall service cost."},"_bibtex":{"value":"@article{\nanonymous2026promptwise,\ntitle={PromptWise: Extending Contextual Bandits to Multi-Round Prompt Routing for Cost-Efficient {LLM} Inference},\nauthor={Anonymous},\njournal={Submitted to Transactions on Machine Learning Research},\nyear={2026},\nurl={https://openreview.net/forum?id=mjkSi2oFLX},\nnote={Rejected}\n}"},"title":{"value":"PromptWise: Extending Contextual Bandits to Multi-Round Prompt Routing for Cost-Efficient LLM Inference"},"pdf":{"value":"/pdf/02ca4464deacbc7d1c17b1fb95095471f47e83d1.pdf"},"venueid":{"value":"TMLR/Rejected"},"assigned_action_editor":{"value":"~Cong_Shen1"}},"version":2},{"content":{"comment":{"value":"### **W3. Evaluation is restricted exclusively to mathematical reasoning tasks. The paper completely ignores other important domains like general question answering, commonsense reasoning, coding, summarization, or multi-turn dialogue. This severely limits claims about generalizability, as mathematical problems may have unique characteristics that don't transfer to other collaborative inference scenarios where routing decisions differ fundamentally.**\n\nWe agree with the reviewer that designing reward functions for open-ended or subjective tasks is inherently challenging. However, this challenge is **not specific to our method**, it is a well-recognized limitation across the entire reinforcement-learning-with-verified-rewards (RLVR) family of approaches. Existing RLVR-based post-training studies (e.g., GRPO and GRPO variants) are almost exclusively evaluated on tasks with objective correctness signals, such as mathematics reasoning. Reliable reward design for open-ended tasks remains an open research problem. \n\nIn this context, our work follows the same established practice. We intentionally focus on mathematical and arithmetic reasoning tasks because they provide **deterministic, verifiable reward signals** essential for evaluating the effectiveness of a new collaborative training paradigm. Our contribution is to make the **first attempt** at an end-to-end optimization framework for on-device–cloud collaborative inference, with a particular focus on enabling **collaborative reasoning**.\n\nWe would like to clarify that **we do not claim that our methodology currently applies to all categories of tasks**. Extending collaborative device–cloud RL training to more complex or subjective tasks would require **reward-model-based evaluation** or **LLM-as-a-Judge mechanisms**, which is an important direction for future work.\n\n> The limitation noted by the reviewer is **a limitation of the RLVR regime**, not of our framework specifically. Our method inherits the same scope as other RLVR methods and can be applied to the tasks where deterministic reward signals exist. To clarify this, we revised the manuscript and added a discussion on the potential limitation of our work.\n\n### **W4. Only one routing method, notable omissions include FrugalGPT, RouteLLM variants, context-aware cascading policies, and ensemble-based routing methods**\n\nWe would like to clarify that some works mentioned by the reviewer are **not applicable** to the device–cloud LLM collaboration setting studied in this paper, and some of them are actually **similar** as the baseline we considered in our original manuscript.\n\n**FrugalGPT [R1]** also adopt a router to select LLMs. However, it select multiple LLMs for one query and sequentially ask these LLMs to generate response until a target quality is reached. This is fundamentally different from our device–cloud collaboration setup.\n\n**Context-aware cascading policy [R2]** extend FrugalGPT by adding contextual features but still operate in the same **multi-LLM cascade** regime, where several LLMs may be queried sequentially for the same prompt, making it not an appropriate baseline.\n\n**Ensemble-based routing method [R3]** also assumes access to several diverse LLMs and predicts a routing probability distribution over them. This amounts to a **multiclass classification problem across**, which is not aligned with our binary device–cloud setting. Conceptually, this is similar to the baseline we already included (Task-Tuning&Router).\n\n**RouteLLM [R4]** take BERT-base and LLaMA3-8B as the classifier to route the query between a small model and powerful model, which is essentially similar to the baseline we already included (Task-Tuning&Router).\n\n[R1] \"Frugalgpt: How to use large language models while reducing cost and improving performance.\" *2023*.\n\n[R2] \"Efficient contextual llm cascades through budget-constrained policy learning.\" *2024*.\n\n[R3] \"Routing to the expert: Efficient reward-guided ensemble of large language models.\" *2024*.\n\n[R4] \"Routellm: Learning to route llms with preference data.\" *2024*."},"title":{"value":"Responses (2/4)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763965340795,"tcdate":1763941829110,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission6850/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission6850/Authors"],"forum":"SwHnok3JtD","number":5,"license":"CC BY 4.0","cdate":1763941829110,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission6850/-/Official_Comment"],"mdate":1763965340795,"domain":"ICLR.cc/2026/Conference","replyto":"7ReKF9CHhr","id":"pZQHJAt9cq","forumContent":{"TLDR":{"value":"We propose a unified post-training framework that integrates routing optimization, enabling the on-device LLM to improve its problem-solving ability while learning routing strategies."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Large Language Models","Device-Cloud Collaboration","Post-Training","Routing Optimization","Reinforcement Learning"]},"supplementary_material":{"value":"/attachment/0abec1d957d5dfe99e2938019c58c05fb162a7f5.zip"},"primary_area":{"value":"other topics in machine learning (i.e., none of the above)"},"abstract":{"value":"Device-cloud collaboration has emerged as a promising paradigm for deploying large language models (LLMs), combining the efficiency of lightweight on-device inference with the superior performance of powerful cloud LLMs. An essential problem in this scenario lies in deciding whether a given query is best handled locally or delegated to the cloud. Existing approaches typically rely on external routers, implemented as binary classifiers, which often struggle to determine task difficulty from the prompt's surface pattern. To address these limitations, we propose a framework where the on-device LLM makes routing decisions at the end of its solving process, with this capability instilled through post-training. In particular, we formulate a reward maximization problem with carefully designed rewards that encourage effective problem solving and judicious offloading to the cloud. To solve this problem, we develop a group-adaptive policy gradient algorithm, featuring a group-level policy gradient, designed to yield an unbiased gradient estimator of the reward, and adaptive prompt filtering, developed to provide complementary learning signals. Extensive experiments across models and benchmarks show that the proposed methodology consistently outperforms existing baselines and significantly narrows the gap to full cloud LLM performance."},"_bibtex":{"value":"@misc{\nfang2026collaborative,\ntitle={Collaborative Device-Cloud {LLM} Inference through Reinforcement Learning},\nauthor={Wenzhi Fang and Dong-Jun Han and Liangqi Yuan and Christopher Brinton},\nyear={2026},\nurl={https://openreview.net/forum?id=SwHnok3JtD}\n}"},"title":{"value":"Collaborative Device-Cloud LLM Inference through Reinforcement Learning"},"pdf":{"value":"/pdf/b0a3698c6181d5c8e4245b48f5ea8700da6b0ff7.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"fang|collaborative_devicecloud_llm_inference_through_reinforcement_learning"},"authorids":{"value":["~Wenzhi_Fang1","~Dong-Jun_Han1","~Liangqi_Yuan1","~Christopher_Brinton1"]},"authors":{"value":["Wenzhi Fang","Dong-Jun Han","Liangqi Yuan","Christopher Brinton"]}},"version":2},{"content":{"comment":{"value":"**Q4. While GNN layer count is analyzed, other hyperparameters and embedding dimension are not thoroughly explored, but I think this is a minor problem.** \n\n**Response:** Thanks for your constructive advice. In the paper, we have already explored the impact of different GNN layer counts and GNN embedding sizes on GraphRouter in section 4.3. However, following the reviewers' suggestions, we have expanded the original ablation study to include experiments on the effects of different training rates on GraphRouter's performance. Specifically, we explored the impact of different learning rates on the Reward of GraphRouter, we selected five different learning rates and compared their effects on Reward in the Balance scenario while keeping other hyperparameters consistent. The table below shows that the Reward generally shows a trend of initially increasing and then decreasing as the learning rate increases, achieving the best performance when the learning rate is 1e-4. All the discussions and experimental results mentioned above have been updated in Section C.3 of the Appendix and Table 26 of the revised PDF version.\n\n**Table: Reward values of different learning rates.**\n\n| Learning Rate | 1e-1  | 1e-2  | 1e-3  | 1e-4 | 1e-5 |\n|---------------|-------|-------|-------|------|------|\n| **Reward**    | 0.0725| 0.0639| 0.0964| 0.208| 0.152|\n\n**Q5. How does the performance of GraphRouter change when handling LLMs with similar architectures but different sizes (e.g., different versions of LLaMA)? Is the framework able to effectively distinguish between such similar models?** \n\n**Response:** Thanks for the reviewer’s insightful question. Due to our current LLM setting, which includes many LLMs with similar architectures but different sizes (e.g., different versions of LLaMA), our experimental results also indirectly prove GraphRouter's ability to model and distinguish similar LLMs. However, following the reviewers' suggestions, we conducted experiments on different versions of LLaMA. We extracted and utilized data related to the LLaMA series of LLMs from the interaction dataset introduced in section 3.2 for training and prediction. The selected LLMs include LLaMA-3 (7b), LLaMA-2 (7b), LLaMA-3 (70b), LLaMA-3-Turbo (8b), LLaMA-3-Turbo (70b), and Llama-3.1-Turbo (70b). We compared the performance of GraphRouter and the best-performing baseline, FrugalGPT, on this dataset and reported the specific results in the following table. We observed that, compared to FrugalGPT, GraphRouter improved the Reward by at least 10.8%. These observations demonstrate that GraphRouter can effectively capture differences in the capabilities of different LLMs through interactions, achieving good results. All the discussions and experimental results mentioned above have been updated in Section B.2 of the Appendix and Table 23 of the revised PDF version.\n\n**Comparison of various methods on the LLaMA-series dataset.** Bold denotes the best results. Each metric reflects average values from multiple evaluation rounds.\n\n| Model       |  Performance (Performance First) |  Cost (Performance First) |  Reward (Performance First) |  Performance (Balance) |  Cost (Balance) | Reward (Balance) |  Performance (Cost First) | Cost  (Cost First) |  Reward  (Cost First) |\n|-------------|--------------------------------|-------------------------|--------------------------|---------------------|--------------|----------------|------------------------|-----------------|-------------------|\n| FrugalGPT   | 0.382                          | 0.299                   | 0.382                    | 0.367               | 0.043        | 0.162          | 0.372                  | 0.030           | 0.050             |\n| **GraphRouter** | 0.422                          | 0.307                   | **0.422**                | 0.416               | 0.032        | **0.192**      | 0.416                  | 0.032           | **0.058**         |\n| Oracle      | 0.489                          | 0.376                   | 0.489                    | 0.459               | 0.051        | 0.204          | 0.436                  | 0.032           | 0.062             |"},"title":{"value":"Response to Reviewer c8tQ (3/4)"}},"tmdate":1732304701229,"tcdate":1732304701229,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission10977/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission10977/Authors"],"forum":"eU39PDsZtT","number":5,"license":"CC BY 4.0","cdate":1732304701229,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission10977/-/Official_Comment"],"mdate":1732304701229,"domain":"ICLR.cc/2025/Conference","replyto":"hJGbaSeBcP","id":"D5otISgSFR","forumContent":{"TLDR":{"value":"We propose GraphRouter, a novel graph-based router that improves LLM selection for diverse tasks by leveraging an inductive graph framework to optimize performance-cost trade-offs and generalize across various models and scenarios."},"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["LLM selection","Graph-based router","Contextual interactions","New LLM settings"]},"primary_area":{"value":"learning on graphs and other geometries & topologies"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The rapidly growing number and variety of Large Language Models (LLMs)\npresent significant challenges in efficiently selecting the appropriate LLM for\na given query, especially considering the trade-offs between performance and\ncomputational cost. Current LLM selection methods often struggle to generalize\nacross new LLMs and different tasks because of their limited ability to leverage\ncontextual interactions among tasks, queries, and LLMs, as well as their depen-\ndence on a transductive learning framework. To address these shortcomings, we\nintroduce a novel inductive graph framework, named as GraphRouter, which\nfully utilizes the contextual information among tasks, queries, and LLMs to en-\nhance the LLM selection process. GraphRouter constructs a heterogeneous\ngraph comprising task, query, and LLM nodes, with interactions represented as\nedges, which efficiently captures the contextual information between the query’s\nrequirements and the LLM’s capabilities. Through an innovative edge prediction\nmechanism, GraphRouter is able to predict attributes (the effect and cost of\nLLM response) of potential edges, allowing for optimized recommendations that\nadapt to both existing and newly introduced LLMs without requiring retraining.\nComprehensive experiments across three distinct effect-cost weight scenarios have\nshown that GraphRouter substantially surpasses existing routers, delivering a\nminimum performance improvement of 12.3%. In addition, it achieves enhanced\ngeneralization across new LLMs settings and supports diverse tasks with at least a\n9.5% boost in effect and a significant reduction in computational demands. This\nwork endeavors to apply a graph-based approach for the contextual and adaptive\nselection of LLMs, offering insights for real-world applications."},"_bibtex":{"value":"@inproceedings{\nfeng2025graphrouter,\ntitle={GraphRouter: A Graph-based Router for {LLM} Selections},\nauthor={Tao Feng and Yanzhen Shen and Jiaxuan You},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=eU39PDsZtT}\n}"},"title":{"value":"GraphRouter: A Graph-based Router for LLM Selections"},"pdf":{"value":"/pdf/2452b8ff98e5d19eedb55a812aabf2db832a2ec4.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"feng|graphrouter_a_graphbased_router_for_llm_selections"},"authorids":{"value":["~Tao_Feng5","~Yanzhen_Shen1","~Jiaxuan_You2"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Tao Feng","Yanzhen Shen","Jiaxuan You"]}},"version":2},{"content":{"comment":{"value":"We thank you for your reviews and address your concerns below.\n\nQ1: Please cite these works: https://arxiv.org/abs/2305.05176 - routing on a query level; https://arxiv.org/abs/2211.17192, https://arxiv.org/abs/2302.07863 - latency reduction using small and big models. I believe writing a discussion of the tradeoffs of these approaches would improve the current draft.\n\nA1: Thank you for pointing out these important references. We will add a discussion of the tradeoffs of these approaches in our revision. In brief, \n1. FrugalGPT [1] studies how to reduce inference costs using LLM cascades and early-exit strategies. Specifically, for each user query, FrugalGPT will execute a sequence of LLMs and return answers whenever the confidence scores exceed the predefined thresholds. Though empirically effective, FrugalGPT is at the risk of executing multiple LLMs or even all LLMs in the cascade to answer a single query, which could be prohibitively expensive. Instead, our routing mechanism ensures that only one LLM will be used to generate answers for user queries and therefore is likely to be more efficient especially on challenging queries.\n2. Speculative decoding [2,3] speeds up decoding of expensive models by invoking small-and-efficient decoders on the “easy” decoding steps. Instead, in our work we are interested in query routing which assigns “easy” queries to small models to reduce overall inference costs while maintaining high performance. Clearly, speculative decoding [2,3] optimizes a different objective and is complementary to our approach. A straightforward framework combining the two will be as follows. Users can first use our router to decide if a query should be handled by the small or large model. If the query goes to the small model, we can save significant costs as demonstrated in our paper. If the query goes to the large model, we can still apply speculative decoding to achieve further efficiency improvements.\n\nQ2: In this method, we are able to reduce cost and latency, but not as much latency reduction as methods such as speculative decoding (https://arxiv.org/abs/2211.17192). While there is added cost with speculative decoding, do you think there's any possibility of closing this gap?\n\nA2: Thank you for this question. As discussed in A1, speculative decoding [2, 3] optimizes a different objective than ours, which may explain the difference in achieved cost reduction. Given that the two approaches are complementary, instead of asking how to “close this gap”, it could be more interesting to ask how to achieve further efficiency improvements by combining our approach with speculative decoding. A simple strategy is discussed in A1 and we will investigate other possibilities in the future work.\n\nQ3: Do you think this might be because of scoring query wise vs token-wise? Why not use this method token wise?\n\nA3: Thank you for this question. As discussed above, the difference may largely come from the fact that we are optimizing different objectives than speculative decoding [2,3], with the distinction between query-wise and token-wise objective formulation being one of the contributing factors.\n\nIn this work, we did not consider applying our approach on token level mainly for two reasons: (1) We are interested in developing a generic approach that is agnostic to model implementation and generalizes to both open-sourced as well as black-box LLMs. Token-wise routing requires transferring KV values of previous tokens across models which can be infeasible if these intermediate results are unavailable (e.g., black-box LLM APIs) or the dimensions do not match across open-sourced LLMs. (2) A naive token-wise routing requires executing the router at each decoding step, which may lead to notable compute overheads and weaken the achieved cost reduction. \n\nThank you for your time and consideration. We sincerely hope that you would consider increasing your rating if you find our responses helpful. \n\nReferences:  \n[1] Chen, Lingjiao, Matei Zaharia, and James Zou. \"FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance.\" arXiv preprint arXiv:2305.05176 (2023).  \n[2] Leviathan, Yaniv, Matan Kalman, and Yossi Matias. \"Fast inference from transformers via speculative decoding.\" International Conference on Machine Learning. PMLR, 2023.  \n[3] Kim, Sehoon, et al. \"Speculative Decoding with Big Little Decoder.\" Thirty-seventh Conference on Neural Information Processing Systems. 2023."},"title":{"value":"To Reviewer 9pHQ"}},"tmdate":1700179166019,"tcdate":1700179166019,"writers":["ICLR.cc/2024/Conference","ICLR.cc/2024/Conference/Submission7318/Authors"],"signatures":["ICLR.cc/2024/Conference/Submission7318/Authors"],"forum":"02f3mUtqnM","number":1,"license":"CC BY 4.0","cdate":1700179166019,"readers":["everyone"],"invitations":["ICLR.cc/2024/Conference/Submission7318/-/Official_Comment"],"mdate":1700179166019,"domain":"ICLR.cc/2024/Conference","replyto":"01SJ6h5IHh","id":"0RDByakEoL","forumContent":{"venue":{"value":"ICLR 2024 poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Large language models","Efficient ML","Query Routing"]},"primary_area":{"value":"general machine learning (i.e., none of the above)"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide."},"abstract":{"value":"Large language models (LLMs) excel in most NLP tasks but also require expensive cloud servers for deployment due to their size, while smaller models that can be deployed on lower cost (e.g., edge) devices, tend to lag behind in terms of response quality. Therefore in this work we propose a hybrid inference approach which combines their respective strengths to save cost and maintain quality. Our approach uses a router that assigns queries to the small or large model based on the predicted query difficulty and the desired quality level. The desired quality level can be tuned dynamically at test time to seamlessly trade  quality for cost as per the scenario requirements. In experiments our approach allows us to make up to 40% fewer calls to the large model, with no drop in response quality."},"_bibtex":{"value":"@inproceedings{\nding2024hybrid,\ntitle={Hybrid {LLM}: Cost-Efficient and Quality-Aware Query Routing},\nauthor={Dujian Ding and Ankur Mallick and Chi Wang and Robert Sim and Subhabrata Mukherjee and Victor R{\\\"u}hle and Laks V. S. Lakshmanan and Ahmed Hassan Awadallah},\nbooktitle={The Twelfth International Conference on Learning Representations},\nyear={2024},\nurl={https://openreview.net/forum?id=02f3mUtqnM}\n}"},"title":{"value":"Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing"},"pdf":{"value":"/pdf/18ea688c270f3fb84befcc014b7228d72ce47eda.pdf"},"venueid":{"value":"ICLR.cc/2024/Conference"},"paperhash":{"value":"ding|hybrid_llm_costefficient_and_qualityaware_query_routing"},"authorids":{"value":["~Dujian_Ding1","~Ankur_Mallick1","~Chi_Wang3","~Robert_Sim1","~Subhabrata_Mukherjee2","~Victor_Rühle1","~Laks_V._S._Lakshmanan1","~Ahmed_Hassan_Awadallah1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors' identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Dujian Ding","Ankur Mallick","Chi Wang","Robert Sim","Subhabrata Mukherjee","Victor Rühle","Laks V. S. Lakshmanan","Ahmed Hassan Awadallah"]}},"version":2},{"content":{"summary":{"value":"This paper introduces RADAR (Reasoning–Ability and Difficulty-Aware Routing), a lightweight and interpretable routing framework that optimizes the trade-off between performance and cost when deploying reasoning language models (RLMs) with varying model sizes and reasoning budgets.\nRADAR draws inspiration from Item Response Theory (IRT) in psychometrics to estimate query difficulty and model-budget ability. Using these interpretable parameters, it routes easier queries to cheaper, smaller models and harder ones to more capable configurations. The authors formalize routing as a multi-objective optimization (MOO) problem balancing performance and cost via linear and Chebyshev scalarization.\nExperiments across 8 reasoning benchmarks (AIME, MATH-500, GPQA, LSAT, MMLU, MMLU-Redux, MMLU-Pro, FRAMES) demonstrate that RADAR achieves superior performance-cost Pareto efficiency, generalizes well to out-of-distribution queries, and scales effectively when adding new models. It operates in real-time with ~7 ms routing latency."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"Check weakness above.\n1. A comparison/discussion with previous works on selecting the best answer from different LLMs would have been useful for latency. \ni. Uncertainty-Aware Answer Selection for Improved Reasoning in Multi-LLM Systems.\nii. Scalable best-of-n selection for large language models via self-certainty. \n2. Which embedding model was used to represent queries?\n3. Could RADAR dynamically adjust the user weight w₁ during a session (based on feedback or budget exhaustion) to continuously optimize global cost-performance?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"First to frame reasoning model routing as a Multi-Objective Optimization (MOO) problem. Innovative use of Item Response Theory for modeling reasoning ability and query difficulty, an elegant and interpretable alternative to opaque routing regressors.\n\nThe methodology is rigorous, combining psychometric modeling with MOO and cost modeling in a consistent mathematical framework.\nExperiments are extensive, covering 8 benchmarks and both in-distribution (ID) and out-of-distribution (OOD) settings, with clear comparisons to baselines such as RouterBench and IRT-Router.\n\nThe paper is very well written and structured, with clear motivation.\n\nThe work offers a generalizable and practical approach for efficient LLM deployment under cost constraints."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. While RADAR clearly outperforms IRT-Router and RouterBench, the paper does not compare against mixture-of-experts or test-time adaptation methods that could serve as strong baselines for dynamic routing under cost constraints.\n2. More discussion on relation to cascading methods (e.g., FrugalGPT) would strengthen the positioning.\n3. Although latency is reported (~7 ms), no end-to-end system-level throughput analysis is provided. \n4. Some OOD performance dips (e.g., AIME benchmark) suggest the need for improved handling of unseen high-difficulty queries.\n5. The model for estimating difficulty is too simple and may not capture the real question difficulty."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762921267664,"tcdate":1761952572520,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission9782/Reviewer_EkVW"],"signatures":["ICLR.cc/2026/Conference/Submission9782/Reviewer_EkVW"],"forum":"CB6Ds5T4ae","number":2,"license":"CC BY 4.0","cdate":1761952572520,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission9782/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762921267664,"domain":"ICLR.cc/2026/Conference","replyto":"CB6Ds5T4ae","id":"cC7FC9cuXW","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["routing","adaptive reasoning","item response theory","reasoning models","large language models"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Reasoning language models have demonstrated remarkable performance on many challenging tasks in math, science, and coding. Choosing the right reasoning model for practical deployment involves a performance-cost trade-off at two key levels: model size and reasoning budget, where larger models and higher reasoning budgets lead to better performance but incur greater cost and latency. In this work, we tackle this tradeoff from the angle of model configuration routing for different queries, and present RADAR (Reasoning–Ability and Difficulty-Aware Routing), a lightweight, interpretable, and scalable routing framework. Inspired by psychometrics, RADAR learns an item response model from model responses with different budgets to different queries, with interpretable parameters including query difficulties and model-budget abilities. RADAR then routes queries with higher difficulty to model-budget pairs with higher ability, and vice versa. We conduct extensive experiments on 8 widely used challenging reasoning benchmarks, demonstrating the superior performance of RADAR compared to state-of-the-art model routing methods. RADAR also exhibits query generalization capabilities, achieving strong performance on out-of-distribution queries on all benchmarks. RADAR is also scalable and can efficiently integrate additional models by dynamically selecting a small set of evaluation queries to estimate their abilities."},"_bibtex":{"value":"@inproceedings{\nfernandez2026radar,\ntitle={{RADAR}: Reasoning-Ability and Difficulty-Aware Routing for Reasoning {LLM}s},\nauthor={Nigel Fernandez and Branislav Kveton and Ryan A. Rossi and Andrew Lan and Zichao Wang},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=CB6Ds5T4ae}\n}"},"title":{"value":"RADAR: Reasoning-Ability and Difficulty-Aware Routing for Reasoning LLMs"},"pdf":{"value":"/pdf/0aee0b8194c380f76d5f3af79f768e66adeeb181.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"fernandez|radar_reasoningability_and_difficultyaware_routing_for_reasoning_llms"},"authorids":{"value":["~Nigel_Fernandez1","~Branislav_Kveton1","~Ryan_A._Rossi2","~Andrew_Lan1","~Zichao_Wang1"]},"authors":{"value":["Nigel Fernandez","Branislav Kveton","Ryan A. Rossi","Andrew Lan","Zichao Wang"]}},"version":2},{"content":{"summary":{"value":"AutoMix is an approach designed to optimize the performance and computational cost of LLMs by selecting the appropriate model based on the difficulty of the task. This is achieved through a few-shot self-verification mechanism and a Partially Observable Markov Decision Process (POMDP) based router. The few-shot self-verification estimates the correctness of outputs from a smaller LM, and the POMDP router decides whether to route the query to a larger LM. Experiments demonstrate that AutoMix can reduce computational cost by over 50% while maintaining performance across various datasets and models."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"I think it would be interesting to just randomly pick up a LLM for each question in dataset, and to see the total cost and the quality of results. Have anyone done this before? Because there might be a case where, even without any filtering strategy, just putting several models with different performance and different cost out there is itself effective. And I think may be it’s possible to derive it mathematically？"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"- The author introduce a novel combination of few-shot self-verification and POMDP-based routing to optimize the use of large language models. It is very thoughtful to make SLM itself both the answerer and the judge, and to consider it as the first candidate when selecting a suitable model. And the author successfully uses POMDP to meet the need of selecting a suitable model.\n- They provide a reasonable metric for cost-efficiency analysis. When the cost grows or when the performance decreases, the results of the metric will become small.\n- The methodology is sound, and the results demonstrate the effectiveness of AutoMix in reducing computational costs by over 50% while maintaining same performance. The experiments are robust, and the comparisons with several baselines prove the advantages of AutoMix."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- While the paper demonstrates the effectiveness of AutoMix across various datasets, it primarily focuses on dialogue and context-grounded reasoning tasks. It is not clear how well the approach would generalize to other types of tasks, such as factual question answering or commonsense reasoning.\n- The paper mentions that the POMDP can learn from as few as 50 examples, but it does not elaborate on the conditions or types of data required for effective training.\n- Although the paper compares AutoMix with strong baselines like FrugalGPT and HybridLLM, the comparisons could be more detailed in terms of why AutoMix outperforms these baselines in specific cases."},"limitations":{"value":"The author pointed out the limitations  in the discussion."}},"nonreaders":[],"tmdate":1730880104078,"tcdate":1720834078510,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission20326/Reviewer_E2SA"],"signatures":["NeurIPS.cc/2024/Conference/Submission20326/Reviewer_E2SA"],"forum":"e6WrwIvgzX","number":4,"license":"CC BY 4.0","cdate":1720834078510,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission20326/-/Official_Review","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730880104078,"domain":"NeurIPS.cc/2024/Conference","replyto":"e6WrwIvgzX","id":"THxJwt1atr","forumContent":{"TLDR":{"value":"AutoMix robustly routes queries among language models of varying sizes, efficiently balancing computational cost and solution accuracy"},"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Few-shot learning","Zero-shot learning","Self-Verification","cost-quality optimization","Decision making","Prompting","LLMs"]},"primary_area":{"value":"natural_language_processing"},"abstract":{"value":"Large language models (LLMs) are now available from cloud API providers in various sizes and configurations. While this diversity offers a broad spectrum of choices, effectively leveraging the options to optimize computational cost and performance remains challenging. In this work, we present AutoMix, an approach that strategically routes queries to larger LMs, based on the approximate correctness of outputs from a smaller LM. Central to AutoMix are two key technical contributions. First, it has a few-shot self-verification mechanism, which estimates the reliability of its own outputs without requiring extensive training. Second, given that self-verification can be noisy, it employs a POMDP based router that can effectively select an appropriately sized model, based on answer confidence. Experiments across five language models and five challenging datasets show that Automix consistently surpasses strong baselines, reducing computational cost by over 50\\% for comparable performance."},"_bibtex":{"value":"@inproceedings{\naggarwal2024automix,\ntitle={AutoMix: Automatically Mixing Language Models},\nauthor={Pranjal Aggarwal and Aman Madaan and Ankit Anand and Srividya Pranavi Potharaju and Swaroop Mishra and Pei Zhou and Aditya Gupta and Dheeraj Rajagopal and Karthik Kappaganthu and Yiming Yang and Shyam Upadhyay and Manaal Faruqui and Mausam .},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=e6WrwIvgzX}\n}"},"title":{"value":"AutoMix: Automatically Mixing Language Models"},"pdf":{"value":"/pdf/f99d5e58580a4ddb7a4d60eef66db492cfafb4d9.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"aggarwal|automix_automatically_mixing_language_models"},"authorids":{"value":["~Pranjal_Aggarwal1","~Aman_Madaan1","~Ankit_Anand4","~Srividya_Pranavi_Potharaju1","~Swaroop_Mishra1","~Pei_Zhou1","~Aditya_Gupta2","~Dheeraj_Rajagopal1","~Karthik_Kappaganthu1","~Yiming_Yang1","~Shyam_Upadhyay1","~Manaal_Faruqui2","~Mausam_.1"]},"authors":{"value":["Pranjal Aggarwal","Aman Madaan","Ankit Anand","Srividya Pranavi Potharaju","Swaroop Mishra","Pei Zhou","Aditya Gupta","Dheeraj Rajagopal","Karthik Kappaganthu","Yiming Yang","Shyam Upadhyay","Manaal Faruqui","Mausam ."]}},"version":2},{"content":{"comment":{"value":"**Q5.Generalization gap: Over-claiming robustness based on synthetic users could mislead practitioners into deploying systems that fail in real-world diversity.** \n\n**Response:** Thanks for raising the above concerns. As mentioned in our response to Q3, prior work has also adopted simulated data to get user preferences as a reliable methodology. In particular, LLM-based simulation can mimic latent preferences of real users, thereby providing a strong reference value for validating whether a router is capable of capturing such hidden preferences. Moreover, we collected a small-scale real-user interaction dataset to evaluate the generalization of the models. The results reported in Table 1 further confirm that our approach is effective not only in simulated environments but also in real-user scenarios.  \n\n**Q6.Privacy: If deployed on real interaction data, capturing latent user preferences raises concerns about profiling and potential misuse.** \n\n**Response:** Thanks for the reviewer’s insightful questions. It is worth noting that user profiles are only used as system prompts for LLMs to simulate users. During actual training and inference, the model does not observe these profiles. PersonalizedRouter does not rely on accessing sensitive user profiles. Instead, it learns preferences from query–LLM interaction data. Consequently, the embedding of a user node is not a direct representation of the user’s real profile, but rather an abstract latent representation designed to better predict the most suitable LLM for a given query.\n\n\n**Table3: Comparison between different methods under two simulation strategies. The Improvement is measured relative to the best baseline.** \n\n| Scenario&nbsp; | Multi-cost-efficiency Simulation      | LLM-as-a-Judge    |\n|----------------|---|-------------------|\n| Method     | Reward &nbsp;  Improvement (%) | Accuracy &nbsp;  Improvement (%)  |\n| HybridLLM  | 0.141  &emsp;&emsp;&emsp; -36.20  | 0.347  &emsp;&emsp;&emsp;   -14.74            |\n| FrugalGPT  | 0.218  &emsp;&emsp;&emsp; -1.36  | 0.354  &emsp;&emsp;&emsp;   -13.22            |\n| GraphRouter| 0.221  &emsp;&emsp;&emsp; 0.00    | 0.364  &emsp;&emsp;&emsp;   -10.57            |\n| RouterDC   | 0.208  &emsp;&emsp;&emsp; -5.88   | 0.407  &emsp;&emsp;&emsp;   0.00              |\n| Ours       | 0.255  &emsp;&emsp;&emsp; 15.38   | 0.447  &emsp;&emsp;&emsp;   9.83              |\n| Oracle     | 0.304  &emsp;&emsp;&emsp; 37.56  | 1.000  &emsp;&emsp;&emsp;   145.70            |\n\n**Table4: Comparison between different methods under simulation strategies in the new user experimental setting. The Improvement is measured relative to the best baseline.** \n\n| Scenario         | multi-cost-efficiency simulation             | LLM-as-a-Judge strategy                |\n|-------------     |----------------------------------------------|----------------------------------------|\n| Method           | Reward  &nbsp; Improvement (%) | Accuracy   Improvement (%)  |\n| HybridLLM        | -0.142   &emsp;&emsp;&emsp;   -240.59         | 0.294  &emsp;&emsp;&emsp;&ensp;    0.00             |\n| FrugalGPT        | 0.044    &emsp;&emsp;&emsp;   -56.44          | 0.192  &emsp;&emsp;&emsp;   -34.69            |\n| GraphRouter      | 0.083    &emsp;&emsp;&emsp;   -17.82          | 0.256  &emsp;&emsp;&emsp;   -12.93            |\n| RouterDC         | 0.101    &emsp;&emsp;&emsp;&ensp;    0.00           | 0.208  &emsp;&emsp;&emsp;   -29.25            |\n| Ours (Few-shots) | 0.070    &emsp;&emsp;&emsp;  -30.69           | 0.313  &emsp;&emsp;&emsp;&ensp;    6.46             |\n| Ours (Trained)   | 0.108    &emsp;&emsp;&emsp;&ensp;   6.93            | 0.326  &emsp;&emsp;&emsp;&ensp;   10.88             |\n| Oracle           | 0.116    &emsp;&emsp;&emsp;&ensp;   14.85           | 1.000  &emsp;&emsp;&emsp;&ensp;   240.14            |\n\n**Table5: Comparison between different methods under simulation strategies in the new LLM experimental setting. The Improvement is measured relative to the best baseline.** \n\n| Scenario        | multi-cost-efficiency simulation             | LLM-as-a-Judge strategy                |\n|-------------    |----------------------------------------------|----------------------------------------|\n| Method          | Reward &nbsp;  Improvement (%) | Accuracy &nbsp;  Improvement (%)  |\n| HybridLLM       | 0.137 &emsp;&emsp;&emsp;  -37.16   | 0.273  &emsp;&emsp;&emsp;    0.00             |\n| FrugalGPT       | 0.182 &emsp;&emsp;&emsp;  -16.51   | 0.038  &emsp;&emsp;&emsp;   -86.08     |\n| GraphRouter     | 0.203 &emsp;&emsp;&emsp;  -6.88    | 0.247  &emsp;&emsp;&emsp;   -9.52    |\n| RouterDC        | 0.218 &emsp;&emsp;&emsp;   0.00    | 0.199  &emsp;&emsp;&emsp;  -27.10     |\n| Ours (few-shot) | 0.201 &emsp;&emsp;&emsp;  -7.80   | 0.329  &emsp;&emsp;&emsp;   20.51     |\n| Ours (trained)  | 0.234 &emsp;&emsp;&emsp;   7.34    | 0.383  &emsp;&emsp;&emsp;   40.29    |\n| Oracle          | 0.371 &emsp;&emsp;&emsp;  70.18      | 1.000  &emsp;&emsp;&emsp;  266.30    |"},"title":{"value":"Response to Reviewer 2sp7 (Part 3)"}},"parentInvitations":"TMLR/-/Official_Comment","tmdate":1758313818950,"tcdate":1758310911526,"writers":["TMLR","TMLR/Paper5500/Authors"],"signatures":["TMLR/Paper5500/Authors"],"forum":"W80eE3ArAl","number":4,"license":"CC BY 4.0","cdate":1758310911526,"readers":["everyone"],"invitations":["TMLR/Paper5500/-/Official_Comment"],"mdate":1758313818950,"domain":"TMLR","replyto":"fIBO7UiOvL","id":"oLDRgyAg9c","forumContent":{"submission_length":{"value":"Regular submission (no more than 12 pages of main content)"},"venue":{"value":"Accepted by TMLR"},"code":{"value":"https://github.com/ulab-uiuc/PersonalizedRouter"},"supplementary_material":{"value":"/attachment/e4bbeecf2accf23bec60f721de3508b2cf488096.zip"},"abstract":{"value":"The growing number of Large Language Models (LLMs) with diverse capabilities and response styles provides users with a wider range of choices, which presents challenges in selecting appropriate LLMs, as user preferences vary in terms of performance, cost, and response style. Current LLM selection methods typically optimize for a single fixed objective, such as performance, cost, or a trade-off between them, and fail to learn individual user preferences from interaction data. To address these limitations, we propose PersonalizedRouter, a graph-based framework that models diverse user profiles and performs personalized LLM selection by leveraging interaction data that includes task context, queries, candidate LLMs, and user decisions. To capture contextual information between user queries and optimal LLMs, PersonalizedRouter converts the interaction data into a heterogeneous graph, where the relationships between different types of nodes are represented by edges. To evaluate adaptability across users, we design two strategies: the multi-cost-efficiency simulation strategy and the LLM-as-a-Judge strategy. In addition, we construct PersonaRoute-Bench, a large-scale benchmark with 1,000 simulated users and 10 LLMs. Experimental results show that PersonalizedRouter significantly outperforms existing LLM selection methods and surpasses the strongest methods by a large margin of 15.38% and 9.83% under two simulation strategies. On the PersonaRoute-Bench with 1,000 users, it further surpasses the best methods by 16.19% and 59.69% while maintaining higher efficiency. Moreover, PersonalizedRouter demonstrates strong few-shot generalization, achieving 64.81% and 85.80% of the fully trained model’s performance when adapting to new users and new LLMs."},"_bibtex":{"value":"@article{\ndai2025personalizedrouter,\ntitle={PersonalizedRouter: Personalized {LLM} Routing via Graph-based User Preference Modeling},\nauthor={Zhongjie Dai and Tao Feng and Jiaxuan You},\njournal={Transactions on Machine Learning Research},\nissn={2835-8856},\nyear={2025},\nurl={https://openreview.net/forum?id=W80eE3ArAl},\nnote={}\n}"},"title":{"value":"PersonalizedRouter: Personalized LLM Routing via Graph-based User Preference Modeling"},"pdf":{"value":"/pdf/1c7d7c4a7a37c3cd71672b102988c3788f67c8e7.pdf"},"venueid":{"value":"TMLR"},"paperhash":{"value":"dai|personalizedrouter_personalized_llm_routing_via_graphbased_user_preference_modeling"},"authorids":{"value":["~Zhongjie_Dai1","~Tao_Feng5","~Jiaxuan_You2"]},"assigned_action_editor":{"value":"~Yen-Chang_Hsu1"},"authors":{"value":["Zhongjie Dai","Tao Feng","Jiaxuan You"]}},"version":2},{"content":{"summary":{"value":"This work aims at optimizing the LLM inference cost by running user queries through small models and switching to larger models when the performance is not satisfying. \nIn contrast to other work that trains router models to predict the poor small model performance, the paper proposes to use the margin token prediction as a metric to decide when to switch between small and large models.\nThe proposed method is compared to several supervised router approaches including Routing, FrugalGPT, Hybrid LLM, and random. The proposed margin sampling approach outperforms the other methods on most tasks."},"ethics_flag":{"value":1},"reasons_to_reject":{"value":"- The supervised models trained for comparison seem rather weak. Only 500-100 examples are used for training which does not sound optimal. Moreover, the various models are trained on top of distilbert which is a very small model. Since the training cost is negligible, it makes sense to train the compared models on more data and with a much larger model. Running BERT or Roberta for classification would be much smaller than using the actual model. \n- The evaluation uses only short context tasks and it is not clear how the proposed approach would work on a longer context. One concern is that the margin sampling might not be super efficient in larger contexts."},"confidence":{"value":3},"reasons_to_accept":{"value":"- The proposed method does not require tuning a model to decide when to switch the prediction. \n- The results show that the method is efficient compared to the rest of the evaluated examples. \n- The method has been tested on several model families including Mistral, Llama, and OpenAI models through the API."},"rating":{"value":6},"questions_to_authors":{"value":"What is the maximum number of examples that can be  used for training the rankers?"}},"nonreaders":[],"tmdate":1724633715000,"tcdate":1715761356194,"writers":["colmweb.org/COLM/2024/Conference","colmweb.org/COLM/2024/Conference/Submission910/Reviewer_Q76R"],"signatures":["colmweb.org/COLM/2024/Conference/Submission910/Reviewer_Q76R"],"forum":"T9cOYH0wGF","number":4,"license":"CC BY 4.0","cdate":1715761356194,"readers":["everyone"],"invitations":["colmweb.org/COLM/2024/Conference/Submission910/-/Official_Review","colmweb.org/COLM/2024/Conference/-/Edit"],"mdate":1724633715000,"domain":"colmweb.org/COLM/2024/Conference","replyto":"T9cOYH0wGF","id":"6Wcje6Ulj4","forumContent":{"TLDR":{"value":"We optimise calls on a scheme of two LLMs of different size by looking at the uncertainty of the generations from the smaller one."},"venue":{"value":"COLM"},"code_of_ethics":{"value":["I acknowledge that I and all co-authors of this work have read and commit to adhering to the COLM Code of Ethics on https://colmweb.org/CoE.html"]},"keywords":{"value":"API calls optimisation, uncertainty, calibration, cascading"},"abstract":{"value":"Researchers and practitioners operating on a limited budget face the well-known cost-performance trade-off dilemma. The challenging decision often centers on whether to use a large LLM with better performance or a smaller one with reduced costs. This has motivated recent research in the optimisation of LLM calls. Either a cascading strategy is used, where a smaller LLM or both are called causally, or a routing strategy is used, where only one model is ever called. This is dependent on a decision criterion in both scenarios which is typically an auxiliary neural model. In this work, we propose a cost-effective solution; we use only the uncertainty of the generations of the small LLM as the decision criterion. We compare our approach with both cascading and routing strategies using three different pairs of pre-trained small and large LLMs, on nine different tasks and against approaches that require an additional neural model. Our experiments reveal this simple solution optimally balances cost and performance, outperforming existing methods on 25 out of 27 experimental setups."},"_bibtex":{"value":"@inproceedings{\nram{\\'\\i}rez2024optimising,\ntitle={Optimising Calls to Large Language Models with Uncertainty-Based Two-Tier Selection},\nauthor={Guillem Ram{\\'\\i}rez and Alexandra Birch and Ivan Titov},\nbooktitle={First Conference on Language Modeling},\nyear={2024},\nurl={https://openreview.net/forum?id=T9cOYH0wGF}\n}"},"title":{"value":"Optimising Calls to Large Language Models with Uncertainty-Based Two-Tier Selection"},"research_area":{"value":["Compute efficient LMs","Inference algorithms for LMs"]},"pdf":{"value":"/pdf/76d09606fff7aafa2e6c57d9bbf3dd9ec18804fe.pdf"},"venueid":{"value":"colmweb.org/COLM/2024/Conference"},"paperhash":{"value":"ramírez|optimising_calls_to_large_language_models_with_uncertaintybased_twotier_selection"},"authorids":{"value":["~Guillem_Ramírez1","~Alexandra_Birch1","~Ivan_Titov1"]},"author_guide":{"value":["I certify that this submission complies with the submission instructions as described on https://colmweb.org/AuthorGuide.html"]},"authors":{"value":["Guillem Ramírez","Alexandra Birch","Ivan Titov"]}},"version":2},{"content":{"summary":{"value":"This paper introduces Agora, a novel market-based framework for coordinating heterogeneous Vision-Language Models (VLMs) in multi-agent systems. The key idea is to treat epistemic uncertainty as a tradable asset across perceptual, semantic, and inferential dimensions. Agents trade uncertainty based on a profitability-driven protocol, guided by a market-aware Thompson Sampling broker, to minimize system-wide cost while maintaining task performance. The authors provide both theoretical and empirical evidence that Agora outperforms strong baselines (e.g., Mixture-of-Agents, KABB) in terms of accuracy (+8.5% on MMMU) and cost-efficiency (3× reduction)."},"soundness":{"value":4},"confidence":{"value":2},"questions":{"value":"- How would Agora perform in the presence of adversarial or strategic agents that intentionally misreport uncertainty or cost to exploit the market? For instance, how does the framework handle strategic manipulation or collusion that could distort trading dynamics or market equilibrium?\n- Could you clarify the computational complexity of the broker and trading protocol, particularly their overhead within a more realistic analysis?\n- Would Agora still work if agents have overlapping expertise or non-stationary cost profiles?"},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":4},"contribution":{"value":3},"strengths":{"value":"- The core concept of a decentralized market for uncertainty is a powerful and novel metaphor that drives the entire paper.\n- The work tackles a foundational challenge (economic scalability) with a comprehensive solution that includes theoretical formulation, algorithm design, and empirical validation.\n- The framework demonstrates consistent and sometimes substantial gains in accuracy and cost-efficiency across multiple benchmarks."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"-  The framework introduces significant complexity with its multi-dimensional uncertainty quantification, trading ledger, and sophisticated broker. While the paper claims scalability, the experiments are limited to ≤15 agents. It’s unclear how the market mechanism would behave with hundreds of agents or high-frequency trading.\n- The paper overlooks key transaction costs (coordination latency, broker computation, and communication overhead), which are excluded from its cost metrics. A system that is 3x cheaper in API calls but 10x slower due to coordination may not be viable for real-time applications.\n-  While numerous baselines are included, the adaptation of language-model-focused routers (like FrugalGPT, RouteLLM) to the multi-modal, multi-agent setting is only briefly mentioned in the appendix."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762915553066,"tcdate":1762281948121,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission582/Reviewer_oqqC"],"signatures":["ICLR.cc/2026/Conference/Submission582/Reviewer_oqqC"],"forum":"zeqCjGQB4U","number":4,"license":"CC BY 4.0","cdate":1762281948121,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission582/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762915553066,"domain":"ICLR.cc/2026/Conference","replyto":"zeqCjGQB4U","id":"aj5Eea9564","forumContent":{"TLDR":{"value":"We presents a framework that reframes coordination as a decentralized market for uncertainty."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["agent; Vision Language Model; Uncernity"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Vision-Language Models (VLMs) enable powerful multi-agent systems, but scaling them is economically unsustainable: coordinating heterogeneous agents under information asymmetry often spirals costs. Existing paradigms, such as Mixture-of-Agents and knowledge-based routers, rely on heuristic proxies that ignore costs and collapse uncertainty structure, leading to provably suboptimal coordination.\nWe introduce Agora, a framework that reframes coordination as a decentralized market for uncertainty. Agora formalizes epistemic uncertainty into a structured, tradable asset (perceptual, semantic, inferential), and enforces profitability-driven trading among agents based on rational economic rules. A market-aware broker, extending Thompson Sampling, initiates collaboration and guides the system toward cost-efficient equilibria. Experiments on five multimodal benchmarks (MMMU, MMBench, MathVision, InfoVQA, CC-OCR) show that Agora outperforms strong VLMs and heuristic multi-agent strategies, e.g., achieving +8.5% accuracy over the best baseline on MMMU while reducing cost by over 3×. These results establish market-based coordination as a principled and scalable paradigm for building economically viable multi-agent visual intelligence systems."},"_bibtex":{"value":"@inproceedings{\nzhang2026why,\ntitle={Why Keep Your Doubts to Yourself? Trading Visual Uncertainties among Vision-Language Models},\nauthor={Jusheng Zhang and Yijia Fan and Kaitong Cai and Jing Yang and Jiawei Yao and Jian Wang and Guanlong Qu and Ziliang Chen and Keze Wang},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=zeqCjGQB4U}\n}"},"title":{"value":"Why Keep Your Doubts to Yourself? Trading Visual Uncertainties among Vision-Language Models"},"pdf":{"value":"/pdf/03f4bead771e441ade14013c814de209cd0630ea.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"zhang|why_keep_your_doubts_to_yourself_trading_visual_uncertainties_among_visionlanguage_models"},"authorids":{"value":["~Jusheng_Zhang3","~Yijia_Fan1","~Kaitong_Cai1","~Jing_Yang39","~Jiawei_Yao3","~Jian_Wang10","~Guanlong_Qu2","~Ziliang_Chen1","~Keze_Wang1"]},"authors":{"value":["Jusheng Zhang","Yijia Fan","Kaitong Cai","Jing Yang","Jiawei Yao","Jian Wang","Guanlong Qu","Ziliang Chen","Keze Wang"]}},"version":2},{"content":{"summary":{"value":"The authors formalize the concept of \"chemistry\" between LLMs as the change in benefit when models are combined, proposing the CHEME algorithm to compute chemistry scores using Model Interaction Graphs (MIGs) and the RECOMMEND algorithm for optimal ensemble selection. Theoretical analysis demonstrates that chemistry emerges primarily under heterogeneous model performance profiles, with effects modulated by task type, complexity, and ensemble size. Empirical evaluation on the benchmarks provide evidence that chemistry-based selection can improve ensemble effectiveness in specific contexts."},"soundness":{"value":1},"confidence":{"value":4},"questions":{"value":"0. Authors are requested to respond to my comments in the weakness section above.\n1. In line 119, the quality and accuracy scores are both defined over the same index $i$, but the quality is defined on the outputs, and accuracy is defined over the LLMs. Can you clarify this?\n2. Why does equation 2 need to account for the benefit of a w.r.t. X? Shouldn't the benefit (or cost) of a be only dependent on b?\n3. How much historical performance data is required to reliably construct MIGs and estimate chemistry? For a new task domain without existing performance histories, what cold-start strategy would you recommend?\n4. How does the proposed RECOMMEND algorithm distinguish between beneficial (synergistic) and detrimental (antagonistic) high-chemistry pairs? Would a signed (non-absolute) metric that explicitly separates synergy from antagonism be more appropriate for a recommendation task?\n5. The paper reports a negative correlation between chemistry and ensemble complementarity for medium- and high-complexity tasks, attributing this to \"saturation effects.\" Could this finding also be interpreted as a failure of the chemistry metric itself, where it preferentially assigns high scores to redundant model pairs (whose individual benefits drop sharply when combined), thus actively selecting for lower complementarity?"},"rating":{"value":2},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":2},"strengths":{"value":"1. The paper makes a notable contribution by formalizing LLM interactions through a principled cost function framework with proven properties. Theorem 1 and Corollary 1 provide counterintuitive insights that chemistry emerges only under heterogeneous performance profiles, establishing clear theoretical boundaries for when the approach is applicable. \n2. The authors ground their novel concept in a solid mathematical framework, providing clear and formal definitions for cost, benefit, and chemistry.\n3. Rather than overclaiming effectiveness, the authors honestly report mixed results and provide thoughtful analysis of when chemistry helps versus hurts (e.g., saturation effects in high-performing ensembles, task complexity moderation)."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. My biggest concern lies in the computational complexity of the proposed CHEME algorithm. It requires finding the maximum interaction effect over all possible subsets X in S\\{a,b}. This complexity renders the method computationally intractable for any realistic scenario where the pool of candidate LLMs, $S$, is non-trivial (for example, |S| > 20)\n2. Despite claiming \"≈1,800 total experimental task executions,\" the actual dataset sizes are modest—only 10 records per trial across 10 trials means just 100 evaluation instances per configuration. For Liar benchmark (4,000 statements available), this represents 2.5% coverage. The high variance in results (e.g., Clinical Notes N=3 chemistry at 0.700 vs. baseline at 1.000) and the note about \"temporary service issues\" affecting claude-3-7 performance raise questions about result stability and whether observed differences reflect genuine chemistry effects or experimental noise.\n3. The core cost formulation (Equation 1) uses arbitrary weights (1/i) and a 75%/25% weighting between generation and review accuracy that lack principled justification. The paper doesn't explore sensitivity to these hyperparameter choices, leaving uncertainty about whether results depend critically on these specific formulations.\n4. The baseline selection strategies (Random, Performance, Local, Remote) are relatively simple. The paper doesn't compare against recent LLM routing methods (RouteLLM [1] , FrugalGPT [2] mentioned in related work), mixture-of-experts approaches, or learned selection (such as Smoothie [3] and RELM [4] which also uses a combination of model selection and evaluation) policies.\n\nRefs:\n1. Ong, Isaac, et al. \"Routellm: Learning to route llms with preference data.\" arXiv preprint arXiv:2406.18665 (2024).\n2. Chen, Lingjiao, Matei Zaharia, and James Zou. \"Frugalgpt: How to use large language models while reducing cost and improving performance.\" arXiv preprint arXiv:2305.05176 (2023).\n3. Guha, Neel, et al. \"Smoothie: Label free language model routing.\" Advances in Neural Information Processing Systems 37 (2024): 127645-127672.\n4. Kumar, Tarun, et al. \"Co-Optimizing Recommendation and Evaluation for LLM Selection\", ICLR 2025 Workshop on Foundation Models in the Wild."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762933004493,"tcdate":1761904281742,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission20110/Reviewer_zKce"],"signatures":["ICLR.cc/2026/Conference/Submission20110/Reviewer_zKce"],"forum":"T6IIvog2Bk","number":4,"license":"CC BY 4.0","cdate":1761904281742,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission20110/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762933004493,"domain":"ICLR.cc/2026/Conference","replyto":"T6IIvog2Bk","id":"D0CMlZiMfq","forumContent":{"TLDR":{"value":"We introduce the notion of LLM Chemistry resulting in a framework that measures where model combinations exhibit synergistic or antagonistic behaviors that shape collective performance beyond individual capabilities."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["LLM Chemistry","Multi-LLM Recommendation","Multi-LLM Collaboration"]},"primary_area":{"value":"generative models"},"abstract":{"value":"Multi-LLM collaboration promises accurate, robust, and context-aware solutions, yet existing approaches rely on implicit selection and output assessment without analyzing whether collaborating models truly complement or conflict. We introduce LLM Chemistry – a framework that measures when LLM combinations exhibit synergistic or antagonistic behaviors that shape collective performance beyond individual capabilities. We formalize the notion of chemistry among LLMs, propose algorithms that quantify it by analyzing interaction dependencies, and recommend optimal model ensembles accordingly. Our theoretical analysis shows that chemistry among collaborating LLMs is most evident under heterogeneous model profiles, with its outcome impact shaped by task type, group size, and complexity. Evaluation on classification, summarization, and program repair tasks provides initial evidence for these task-dependent effects, thereby reinforcing our theoretical results. This establishes LLM Chemistry as both a diagnostic factor in multi-LLM systems and a foundation for ensemble recommendation."},"_bibtex":{"value":"@misc{\nsanchez2026llm,\ntitle={{LLM} Chemistry Estimation for Multi-{LLM} Recommendation},\nauthor={Huascar Sanchez and Briland Hitaj},\nyear={2026},\nurl={https://openreview.net/forum?id=T6IIvog2Bk}\n}"},"title":{"value":"LLM Chemistry Estimation for Multi-LLM Recommendation"},"pdf":{"value":"/pdf/c0810b2144726d0d016332f3c8934851d4ebe084.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"sanchez|llm_chemistry_estimation_for_multillm_recommendation"},"authorids":{"value":["~Huascar_Sanchez1","~Briland_Hitaj1"]},"authors":{"value":["Huascar Sanchez","Briland Hitaj"]}},"version":2},{"content":{"comment":{"value":"## **Q2: Is the true \"routing overhead\" underestimated? There may be multiple levels of consecutive rejections on the cascading path (the experiment used a 5-model approach, and the worst case was only 4 rejections to reach 14 bytes). Could you please provide the end-to-end latency of the first token for comparison with other methods?**\n\nWe appreciate the reviewer's scrutiny regarding the **practical latency** of our cascading architecture. We share the concern that multiple consecutive rejections could theoretically accumulate delay. However, our new comprehensive analysis in **Appendix A.1** empirically demonstrates that this overhead is **statistically negligible**.\n\n### **1. Negligible Cumulative Overhead (<5%)**\n\nOur data confirms that the \"routing overhead\" (time spent on rejections) is marginal compared to the cost of generation.\n\n- **Worst-Case Analysis:** Even in the worst-case path—where a query is rejected by all four smaller agents before reaching the final 14B model—the cumulative time consumed by these rejections constitutes **less than 5%** of the time required for the final LLM to generate the answer (*see* **Table 7 in the paper**).\n- **Reasoning:** This efficiency stems from the **brevity of the rejection signal**. A rejection typically involves generating only a few tokens (e.g., \"I don't know!\"), whereas generating a valid solution involves a lengthy CoT sequence. Thus, the cost of “saying no\" is a fraction of the cost of solving.\n\n### **2. Time-to-First-Token Analysis**\n\nTo provide the requested comparison, we benchmarked the **System Latency (Time to First Token)** of DiSRouter against router baselines (detailed in **Table R4**).\n\n*R4: Comparison of Time to First Token (s) between routing systems.*\n\n|  | RouterLLM | FrugalGPT | Automix | FORC | GraphRouter | DiSRouter |\n| --- | --- | --- | --- | --- | --- | --- |\n| Average | 0.048 | 2.620 | 3.475 | 0.070 | 0.024 | 0.081 |\n| Worst Case | 0.063 | 9.565 | 11.34 | 0.076 | 0.028 | 0.241 |\n- **vs. Predictive Routers:** DiSRouter incurs slightly higher latency than predictive routers (e.g., *RouteLLM*, *GraphRouter*) that utilize lightweight external classifiers, as our method requires LLM inference.\n- **vs. Generative Baselines:** Crucially, DiSRouter is **significantly faster** than other generative or verification-based routers (e.g., *FrugalGPT*, *AutoMix*). These baselines often require generating **full-length responses** from LLMs before performing verification or cascading, whereas DiSRouter aborts generation immediately upon detecting incompetence.\n\n**Conclusion:** Based on this analysis, we conclude that the \"routing overhead\" introduced by the rejection mechanism is **computationally minor** and does not impede the system's practical utility or user experience."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764155406128,"tcdate":1764155406128,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7635/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission7635/Authors"],"forum":"KDcwXKr0NU","number":7,"license":"CC BY 4.0","cdate":1764155406128,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7635/-/Official_Comment"],"mdate":1764155406128,"domain":"ICLR.cc/2026/Conference","replyto":"6wZj9Kgstr","id":"l2iofs9f9X","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"TLDR":{"value":"We introduce DiSRouter, a distributed system where LLMs leverage \"self-awareness\" to route queries among themselves, outperforming conventional centralized routers."},"keywords":{"value":["query routing","model selection","distributed system","self-awareness of LLM"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"The proliferation of Large Language Models (LLMs) has created a diverse ecosystem of models with highly varying performance and costs, necessitating effective query routing to balance performance and expense. Current routing systems often rely on a centralized external router trained on a fixed set of LLMs, making them inflexible and prone to poor performance since the small router can not fully understand the knowledge boundaries of different LLMs. We introduce DiSRouter (Distributed Self-Router), a novel paradigm that shifts from centralized control to distributed routing. In DiSRouter, a query traverses a network of LLM agents, each independently deciding whether to answer or route to other agents based on its own self-awareness—its ability to judge its competence. This distributed design offers superior flexibility, scalability, and generalizability. To enable this, we propose a two-stage Self-Awareness Training pipeline that enhances each LLM's self-awareness. Extensive experiments demonstrate that DiSRouter significantly outperforms existing routing methods in utility across various scenarios, effectively distinguishes between easy and hard queries, and shows strong generalization to out-of-domain tasks. Our work validates that leveraging an LLM's intrinsic self-awareness is more effective than external assessment, paving the way for more modular and efficient multi-agent systems."},"_bibtex":{"value":"@inproceedings{\nzheng2026disrouter,\ntitle={Di{SR}outer: Distributed Self-Routing for {LLM} Selections},\nauthor={Hang Zheng and Hongshen Xu and Yongkai.lin and Shuai Fan and Lu Chen and Kai Yu},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=KDcwXKr0NU}\n}"},"title":{"value":"DiSRouter: Distributed Self-Routing for LLM Selections"},"pdf":{"value":"/pdf/0190f98d94890b700d03febdee678e22a2a5e380.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"zheng|disrouter_distributed_selfrouting_for_llm_selections"},"authorids":{"value":["~Hang_Zheng3","~Hongshen_Xu1","~Yongkai.lin1","~Shuai_Fan1","~Lu_Chen3","~Kai_Yu3"]},"authors":{"value":["Hang Zheng","Hongshen Xu","Yongkai.lin","Shuai Fan","Lu Chen","Kai Yu"]}},"version":2},{"content":{"decision":{"value":"Accept (poster)"},"comment":{"value":"This paper introduces EcoRouter, a recommendation-system-based framework for routing user instructions to the most cost-effective LLM in a candidate pool. By constructing multidimensional model and data representations (intrinsic properties, benchmark capability scores, and cross features) and training a Wide&Deep/DeepFM-style recommender on >1B synthetic interactions, EcoRouter learns to predict which (model, inference mode) pairing best satisfies a user-defined accuracy target under cost constraints. A Pareto frontier–based evaluation metric is also proposed. Empirically, EcoRouter reduces costs by >38% on six benchmarks across 50+ LLMs while maintaining accuracy.\n\nStrengths. Reviewers agreed the framing of routing as a recommendation problem is both novel and practically important (Reviewer k8wC, N3dk, MPGf). The introduction of a Pareto-frontier cost–accuracy metric was noted as a methodological contribution (Reviewer N3dk). The large-scale evaluation across diverse LLMs and reasoning modes demonstrated strong headline improvements (Reviewer 3Co2). Practical deployment advantages, such as avoiding inference on all candidate models and adaptability to new LLMs, were emphasized as real-world strengths (Reviewer MPGf).\n\nWeaknesses. Several reviewers highlighted limitations in evaluation: missing competitive baselines such as RouteLLM, GraphRouter, Hybrid-LLM, and FrugalGPT (Reviewer 3Co2, N3dk, k8wC); absence of RouterBench coding/multi-turn benchmarks (Reviewer N3dk); and limited ablation analyses (Reviewer N3dk, k8wC). Cost modeling was flagged as simplified and heuristic, not tied to real API pricing or latency (Reviewer 3Co2, MPGf). Concerns about router overhead and whether cost savings persist in real deployments were raised (Reviewer 3Co2, MPGf). Reviewer 3Co2 also questioned possible train–test leakage from the MMLU-based coreset. Finally, the approach requires a massive precomputed interaction dataset (>1B samples), raising barriers for replication (Reviewer MPGf).\n\nDiscussion and Rebuttal. The authors’ rebuttal was substantive and constructive. They added nine missing baselines (FrugalGPT, GraphRouter, Hybrid-LLM, RouteLLM, etc.) and seven new benchmarks (including coding and dialogue), as well as ablations on feature groups and sensitivity to training data size. They clarified that router overhead is ~0.3s vs. 5–7s for a 7B LLM, and committed to adding real API pricing experiments. They also confirmed no leakage between the bridge coreset and test sets. Reviewers largely accepted these clarifications: Reviewer k8wC explicitly raised their score; Reviewer N3dk stated all concerns were fully addressed; Reviewer 3Co2 maintained a borderline-accept rating but acknowledged improvements; Reviewer MPGf appreciated the responses but held their borderline-reject score, citing only partial resolution of concerns.\n\nDecision. While one reviewer remains unconvinced, the majority of reviewers shifted positively after rebuttal, recognizing the paper’s strong empirical contribution, novel framing, and practical significance for LLM deployment. The weaknesses (e.g., dataset construction cost, proxy cost model) are real but do not undermine the central technical contributions. Given its relevance to scalable and economical LLM usage—a timely and practically pressing problem—I recommend acceptance. However, due to residual concerns about evaluation completeness and cost realism, I do not recommend spotlight/oral; the work is valuable but not yet fully mature in evaluation rigor.\n\nFollow-up for Camera-Ready. For the final version, please:\n1. Incorporate the newly added baselines (FrugalGPT, GraphRouter, Hybrid-LLM, RouteLLM, etc.) and expanded benchmarks (e.g., coding, dialogue) into the main paper.\n2. Provide empirical results under real API pricing and latency measurements, not only heuristic proxies, to strengthen cost-effectiveness claims.\n3. Explicitly report routing overhead in ms/query and clarify the amortized cost relative to direct large-model inference.\n4. Clearly document how the bridge coreset is separated from evaluation datasets to avoid any impression of train–test leakage.\n5. Discuss the scalability and accessibility of the required 1B+ interaction dataset, and outline strategies for researchers with limited resources (e.g., smaller-scale training feasibility).\n6. Add a short discussion of broader limitations, such as reliance on single-turn prompts and the simplified cost definition, as flagged by reviewers.\n\nRecommendation: Accept."},"title":{"value":"Paper Decision"}},"parentInvitations":"NeurIPS.cc/2025/Conference/-/Decision","nonreaders":[],"tmdate":1761718706890,"tcdate":1758113604062,"writers":["NeurIPS.cc/2025/Conference","NeurIPS.cc/2025/Conference/Program_Chairs"],"signatures":["NeurIPS.cc/2025/Conference/Program_Chairs"],"forum":"uC3DI4YPRv","number":1,"license":"CC BY 4.0","cdate":1758113604062,"readers":["everyone"],"invitations":["NeurIPS.cc/2025/Conference/Submission24785/-/Decision","NeurIPS.cc/2025/Conference/-/Edit"],"mdate":1761718706890,"domain":"NeurIPS.cc/2025/Conference","replyto":"uC3DI4YPRv","id":"UQ5HKW4D1Z","forumContent":{"venue":{"value":"NeurIPS 2025 poster"},"keywords":{"value":["LLM Routing","Deep Learning"]},"primary_area":{"value":"deep_learning"},"abstract":{"value":"Recently, test-time scaling of Large Language Models (LLMs) has emerged as a practical alternative to parameter and data scaling. Reasoning tasks often require large-scale, RLVR-based LLMs, while more economical LLMs can handle simpler tasks. Routing an LLM tailored to *suitability* (*i.e.*, capability and cost) ensures usability and efficiency. We introduce LLMRec, which routes the most suitable LLM to the user query without pre-inference on the candidate LLM zoo. It pioneeringly reframes the LLM routing problem as a comprehensive recommendation system (RecSys) task. Our core insight is that an LLM's suitability for a query is a complex, latent signal equal to user-item preference. LLMRec systematically engineers features for candidate LLMs (intrinsic attributes and capability distributions), queries (general semantics and meta-dimensional info), and context (inference type, cost budgets). It also incorporates behavioral features to learn high-order interactions. LLMRec is designed to generalize to out-of-domain datasets and adapt to new LLMs as the model zoo evolves. We define the metric with the Pareto frontier under user-specified cost budgets. Across six datasets, LLMRec achieves an average cost reduction of over 38% while maintaining accuracy and consistently outperforming baselines in converging toward the Pareto frontier."},"_bibtex":{"value":"@inproceedings{\nzhang2025let,\ntitle={Let the {LLM} Stick to Its Strengths: Learning to Route Economical {LLM}},\nauthor={Yi-Kai Zhang and Shiyin Lu and Qing-Guo Chen and Weihua Luo and De-Chuan Zhan and Han-Jia Ye},\nbooktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},\nyear={2025},\nurl={https://openreview.net/forum?id=uC3DI4YPRv}\n}"},"title":{"value":"Let the LLM Stick to Its Strengths: Learning to Route Economical LLM"},"pdf":{"value":"/pdf/f8ce3f48c335fa784e0e6aef63e636d13c30d93d.pdf"},"venueid":{"value":"NeurIPS.cc/2025/Conference"},"paperhash":{"value":"zhang|let_the_llm_stick_to_its_strengths_learning_to_route_economical_llm"},"authorids":{"value":["~Yi-Kai_Zhang2","~Shiyin_Lu1","~Qing-Guo_Chen1","~Weihua_Luo2","~De-Chuan_Zhan1","~Han-Jia_Ye1"]},"authors":{"value":["Yi-Kai Zhang","Shiyin Lu","Qing-Guo Chen","Weihua Luo","De-Chuan Zhan","Han-Jia Ye"]}},"version":2},{"content":{"TLDR":{"value":"Step-level model routing across agent trajectories reduces frontier-model inference costs by 72% while maintaining task quality, by routing planning steps to large models and formatting steps to 7B models."},"venue":{"value":"LM4Plan@ICML-26 Poster"},"keywords":{"value":["model routing","agentic planning","cost-optimal LLM selection","multi-step workflows","trajectory-level optimization","LLM planning efficiency"]},"abstract":{"value":"Enterprise agentic systems that route every trajectory step to a frontier model waste 60-80% of their inference budget on subtasks that smaller models handle equally well. Existing routing solutions optimize single-turn query assignment but ignore a property unique to agentic workflows: subtask complexity varies widely within a single trajectory. A planning step may require frontier-class reasoning while a subsequent formatting step needs only a 7B model. We formalize step-level model routing as a sequential assignment problem over agent trajectories and propose AgentRouter, a lightweight classifier (12M parameters, <5ms overhead per step on an A100 GPU) that maps each trajectory step to one of four model tiers using five features extractable at routing time. Trained on 50,000 annotated agent trajectory steps across planning, coding, research, and data analysis, AgentRouter achieves 72% cost reduction with less than 3% quality degradation; per-step routing accuracy reaches 91% on minimal-complexity steps, 85% on efficient-tier steps, and 76-82% on the harder tiers. On the same benchmarks, RouteLLM and FrugalGPT (applied per-step) achieve only 31% and 44% cost reduction respectively, because their single-turn training signal misses trajectory-level quality dependencies."},"_bibtex":{"value":"@inproceedings{\nanonymous2026agentrouter,\ntitle={AgentRouter: Heterogeneous Model Routing for Cost-Optimal Multi-Step Agentic Workflows},\nauthor={Anonymous},\nbooktitle={ICML 2026 Workshop LM4Plan},\nyear={2026},\nurl={https://openreview.net/forum?id=05yAwXIImz}\n}"},"title":{"value":"AgentRouter: Heterogeneous Model Routing for Cost-Optimal Multi-Step Agentic Workflows"},"pdf":{"value":"/pdf/9a76c41c60b55a5f5d02fad74b22eb389f1d34d8.pdf"},"venueid":{"value":"ICML.cc/2026/Workshop/LM4Plan"}},"tmdate":1791933510110,"pdate":1779502869618,"tcdate":1775875700844,"writers":["ICML.cc/2026/Workshop/LM4Plan","ICML.cc/2026/Workshop/LM4Plan/Submission3/Authors"],"signatures":["ICML.cc/2026/Workshop/LM4Plan/Submission3/Authors"],"forum":"05yAwXIImz","license":"CC BY 4.0","number":3,"cdate":1775875700844,"readers":["everyone"],"invitations":["ICML.cc/2026/Workshop/LM4Plan/-/Submission","ICML.cc/2026/Workshop/LM4Plan/-/Post_Submission","ICML.cc/2026/Workshop/LM4Plan/-/Edit","ICML.cc/2026/Workshop/LM4Plan/Submission3/-/Revision"],"mdate":1791933510110,"odate":1791933510088,"domain":"ICML.cc/2026/Workshop/LM4Plan","id":"05yAwXIImz","version":2},{"content":{"venue":{"value":"SCALE@ICML2026 Poster"},"LLM_usage":{"value":"No or minor, the authors took negligible or no help from any kind of LLM in the whole process."},"keywords":{"value":["agentic systems","multi-agent orchestration","LLM agents","bandits","optimal transport","uncertainty"]},"Funding":{"value":"Yes, the presenting author of this submission falls under ICML’s funding aims, and funding would significantly impact their ability to attend the workshop in person."},"abstract":{"value":"Adaptive orchestration of heterogeneous agents requires making sequential delegation decisions under uncertain and evolving agent behaviour, e.g., coordinating specialised AI models with varying reliability, cost, and response quality. While prior work on agent orchestration focuses on performance or cost, uncertainty in agent reliability and output distributions is typically not modelled explicitly at the orchestration level. In this work, we study the problem of adaptive orchestration of heterogeneous agents under uncertainty, where a meta-controller must decide when to delegate to an agent, accounting for reliability, cost, and uncertainty. We propose BOT-Orch, a lightweight framework that recasts orchestration as a bandit problem over agents, regularized by OT distances between agent output distributions and task-specific reference distributions. We show that the regularised orchestration enjoys $\\mathcal{O}(\\sqrt{T})$ regret under standard assumptions, and provably induces preference ordering among agents with identical mean rewards but differing distributional alignment. Empirically, we demonstrate that BOT-Orch outperforms standard and contextual bandits (UCB1, Thompson, LinUCB/LinTS) as well as deployed cost-aware LLM-routing baselines (RouteLLM, FrugalGPT, Auto) across synthetic and semi-synthetic (MNIST) settings with heterogeneous, non-i.i.d. agent behaviour, and we quantify the computational cost of the optimal-transport step."},"_bibtex":{"value":"@inproceedings{\noliver2026learning,\ntitle={Learning to Orchestrate Heterogeneous Agents under Uncertainty},\nauthor={Mary Chriselda Antony Oliver and Lan Jiang and Elaf Almahmoud and Francesco Quinzan and Umang Bhatt},\nbooktitle={ICML 2026 Workshop on Multimodal AI Agents},\nyear={2026},\nurl={https://openreview.net/forum?id=l4TnpuiC2j}\n}"},"title":{"value":"Learning to Orchestrate Heterogeneous Agents under Uncertainty"},"Anonymization":{"value":"This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs."},"pdf":{"value":"/pdf/9f2bc29d7be081027fdb43dc355ba8455be4bc3f.pdf"},"Format":{"value":"Yes, the presenting author will attend in person if this work is accepted to the workshop."},"Presenter":{"value":"~Lan_Jiang7"},"venueid":{"value":"ICML.cc/2026/Workshop/SCALE"},"paperhash":{"value":"oliver|learning_to_orchestrate_heterogeneous_agents_under_uncertainty"},"authorids":{"value":["~Mary_Chriselda_Antony_Oliver1","~Lan_Jiang7","~Elaf_Almahmoud1","~Francesco_Quinzan1","~Umang_Bhatt1"]},"Track":{"value":"Main track (up to 7 pages excluding references and appendix)"},"authors":{"value":["Mary Chriselda Antony Oliver","Lan Jiang","Elaf Almahmoud","Francesco Quinzan","Umang Bhatt"]}},"tmdate":1783264827043,"pdate":1778875849566,"tcdate":1777395512767,"writers":["ICML.cc/2026/Workshop/SCALE","ICML.cc/2026/Workshop/SCALE/Submission59/Authors"],"signatures":["ICML.cc/2026/Workshop/SCALE/Submission59/Authors"],"forum":"l4TnpuiC2j","license":"CC BY 4.0","number":59,"cdate":1777395512767,"readers":["everyone"],"invitations":["ICML.cc/2026/Workshop/SCALE/-/Submission","ICML.cc/2026/Workshop/SCALE/-/Post_Submission","ICML.cc/2026/Workshop/SCALE/-/Edit","ICML.cc/2026/Workshop/SCALE/Submission59/-/Camera-Ready_Revision"],"mdate":1783264827043,"odate":1783264826742,"domain":"ICML.cc/2026/Workshop/SCALE","id":"l4TnpuiC2j","version":2},{"content":{"data_release":{"value":"We authorize the release of our submission and author names to the public in the event of acceptance."},"TLDR":{"value":"AgentRouter demonstrates that step-level routing of agent trajectory steps to heterogeneous model tiers achieves 72% cost reduction at scale, outperforming adapted single-turn routers by 28–41 percentage points."},"venue":{"value":"ICML 2026 Workshop AgenticUQ Poster"},"email_sharing":{"value":"We authorize the sharing of all author emails with Program Chairs."},"keywords":{"value":["scalable AI inference","model routing","cost optimization","heterogeneous model tiers","agentic workflows","efficient foundation models","step-level routing"]},"abstract":{"value":"Enterprise agentic systems that route every trajectory step to a frontier model waste 60–80% of their inference budget on subtasks that smaller models handle equally well. Existing routing solutions optimize single-turn query assignment but ignore a property unique to agentic workflows: subtask complexity varies widely within a single trajectory. A planning step may require frontier-class reasoning while a subsequent formatting step needs only a 7B model. We formalize step-level model routing as a sequential assignment problem over agent trajectories and propose AgentRouter, a lightweight classifier (12M parameters, <5ms overhead per step on an A100 GPU) that maps each trajectory step to one of four model tiers using five features extractable at routing time. Trained on 50,000 annotated agent trajectory steps across planning, coding, research, and data analysis, AgentRouter achieves 72% cost reduction with less than 3% quality degradation; per-step routing accuracy reaches 91% on minimal-complexity steps, 85% on efficient-tier steps, and 76–82% on the harder tiers. On the same benchmarks, RouteLLM and FrugalGPT (applied per-step) achieve only 31% and 44% cost reduction respectively, because their single-turn training signal misses trajectory-level quality dependencies."},"_bibtex":{"value":"@inproceedings{\npaul2026agentrouter,\ntitle={AgentRouter: Heterogeneous Model Routing for Cost-Optimal Multi-Step Agentic Workflows},\nauthor={Rudrendu Kumar Paul and Sourav Nandy},\nbooktitle={ICML 2026 Statistical Frameworks for Uncertainty in Agentic Systems},\nyear={2026},\nurl={https://openreview.net/forum?id=XWc0Yw8cig}\n}"},"title":{"value":"AgentRouter: Heterogeneous Model Routing for Cost-Optimal Multi-Step Agentic Workflows"},"pdf":{"value":"/pdf/6ee2c8daa3b7a23707d1c064b1d383576465333b.pdf"},"venueid":{"value":"ICML.cc/2026/Workshop/AgenticUQ"},"paperhash":{"value":"paul|agentrouter_heterogeneous_model_routing_for_costoptimal_multistep_agentic_workflows"},"authorids":{"value":["~Rudrendu_Kumar_Paul1","~Sourav_Nandy2"]},"authors":{"value":["Rudrendu Kumar Paul","Sourav Nandy"]}},"tmdate":1782482402708,"pdate":1780246182349,"tcdate":1776403676339,"writers":["ICML.cc/2026/Workshop/AgenticUQ","ICML.cc/2026/Workshop/AgenticUQ/Submission8/Authors"],"signatures":["ICML.cc/2026/Workshop/AgenticUQ/Submission8/Authors"],"forum":"XWc0Yw8cig","license":"CC BY 4.0","number":8,"cdate":1776403676339,"readers":["everyone"],"invitations":["ICML.cc/2026/Workshop/AgenticUQ/-/Submission","ICML.cc/2026/Workshop/AgenticUQ/-/Submission_Change_Before_Reviewing","ICML.cc/2026/Workshop/AgenticUQ/-/Edit","ICML.cc/2026/Workshop/AgenticUQ/Submission8/-/Camera_Ready_Revision","ICML.cc/2026/Workshop/AgenticUQ/-/Submission_Release"],"mdate":1782482402708,"odate":1782482401292,"domain":"ICML.cc/2026/Workshop/AgenticUQ","id":"XWc0Yw8cig","version":2},{"content":{"summary":{"value":"This paper introduces GMTRouter, a personalized LLM router for LLMs that extracts structured preference information from multi-turn user-LLM interactions. The methodology is to model user-LLM interactions as a heterogeneous graph with different node types (users, queries, responses, LLMs, and virtual \"turn\" nodes) to capture the relational structure among entities. Then, a heterogeneous graph transformer (HGT) is used to predict which LLM best suits a given user-query pair. Experiments on Chatbot Arena, MT-Bench, GSM8K, and MMLU show consistent improvements over baselines such as GraphRouter and FrugalGPT. The authors also show that GMTRouter generalizes well to new users and remains computationally efficient."},"soundness":{"value":2},"confidence":{"value":2},"questions":{"value":"1. As the heterogeneous graph is constructed directly from the interaction history, it should not contain more information than the history itself. Could the authors elaborate on why such a graph-based representation improves preference elicitation compared to directly learning from the raw interaction table?\n2. The graph construction includes multiple users within the same graph (e.g., Figure 3(b)). However, user preferences are typically independent of one another. Would constructing individual user-specific graphs (and learning over them) be more intuitive? How does the shared multi-user graph influence personalization performance?"},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"1. Modeling multi-turn user-LLM interactions as a heterogeneous graph with explicit node types and virtual \"turn\" nodes is a novel approach to personalized routing.\n2. The inductive training strategy allows adaptation to new users with minimal interaction data, addressing the common cold-start problem.\n3. The framework is lightweight and practical, requiring modest computational resources, which enhances its potential for real-world deployment in LLM routing systems."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. The design of GMTRouter is largely empirical and heuristic. There is no theoretical justification for why the proposed graph structure and the HGT-based model can better extract and generalize user preferences compared to other models.\n2. The main novelty lies in the data modeling design (heterogeneous graph construction), while the learning components (e.g., HGT backbone, inductive training) largely follow existing graph learning literature.\n3. There are many duplicated references, e.g., (Chen et al., 2023a) and (Chen et al., 2023b), (Chen et al., 2023a) and (Chen et al., 2023b), (Feng et al., 2024a) and (Feng et al., 2024b), (Hu et al., 2020a) and (Hu et al., 2020b), (Ong et al., 2024a) and (Ong et al., 2024b).\n\nMinor issues:\n1. The abbreviation \"AUC\" is not defined in the paper."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762925027588,"tcdate":1761748206118,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission14657/Reviewer_8ann"],"signatures":["ICLR.cc/2026/Conference/Submission14657/Reviewer_8ann"],"forum":"oRe9tHpSpn","number":1,"license":"CC BY 4.0","cdate":1761748206118,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission14657/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762925027588,"domain":"ICLR.cc/2026/Conference","replyto":"oRe9tHpSpn","id":"ryB9mqPWFS","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Heterogeneous GNN","LLM routing","Preference Learning"]},"primary_area":{"value":"learning on graphs and other geometries & topologies"},"abstract":{"value":"Large Language Model (LLM) routing has demonstrated strong capability in balancing response quality with computational cost. As users exhibit diverse preferences, personalization has attracted increasing attention in LLM routing, since even identical queries may require different models to generate responses tailored to individual needs. However, existing approaches are not fully personalized and often fail to faithfully capture the complex interactions between specific users and LLMs. Moreover, user preference data is typically scarce, noisy, and inconsistent in format, which limits the effectiveness of methods that rely solely on user-specific data. To address these challenges, we propose GMTRouter, which represents multi-turn user–LLM interactions as a heterogeneous graph with four node types: user, LLM, query, and response, thereby maximally preserving the rich relational structure of the interaction. Through a tailored message-passing mechanism, GMTRouter learns to capture user preferences from few-shot data within a lightweight inductive graph learning framework, enabling effective personalization. Extensive experiments demonstrate that GMTRouter consistently outperforms the strongest baselines, achieving 0.9%–21.6% higher accuracy and 0.006–0.309 higher AUC across multiple datasets. More importantly, we further demonstrate that GMTRouter can adapt to new users and evolving preferences using only few-shot data, without extensive fine-tuning."},"_bibtex":{"value":"@misc{\nxie2026gmtrouter,\ntitle={{GMTR}outer: Personalized {LLM} Router over Multi-turn User Interactions},\nauthor={Encheng Xie and Yihang Sun and Tao Feng and Jiaxuan You},\nyear={2026},\nurl={https://openreview.net/forum?id=oRe9tHpSpn}\n}"},"title":{"value":"GMTRouter: Personalized LLM Router over Multi-turn User Interactions"},"pdf":{"value":"/pdf/b4f227b8ee59ab4f2a14d8fc028c0e06e9455c44.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"xie|gmtrouter_personalized_llm_router_over_multiturn_user_interactions"},"authorids":{"value":["~Encheng_Xie1","~Yihang_Sun2","~Tao_Feng5","~Jiaxuan_You2"]},"authors":{"value":["Encheng Xie","Yihang Sun","Tao Feng","Jiaxuan You"]}},"version":2},{"content":{"summary":{"value":"This paper analyzes the technical and structural problems of the mainstream monolithic LLM paradigm and proposes an alternative framework called Expert Orchestration. The authors argue that competition centered on large corporations not only limits innovation but also risks concentrating AI power. They further note the inherent weaknesses of single models. The proposed framework consists of two core components. First, independent Judge models evaluate LLM outputs using multi-dimensional criteria such as legality and ethics. Second, Router integrates these evaluations with user requirements to direct queries to the most suitable expert model. The authors claim this approach enhances performance, cost-efficiency, safety, and transparency."},"agreement":{"value":4},"code_of_conduct_acknowledgement":{"value":"Yes"},"confidence":{"value":3},"questions":{"value":"- How can the objectivity and reliability of Judge models be secured and validated?\n- How can \"expertise\" be formally defined and measured to ensure that the Router makes optimal decisions when selecting among specialist models?"},"rating":{"value":6},"author_identification":{"value":"No."},"discussion":{"value":4},"ethics":{"value":["NO or VERY MINOR ethics concerns only"]},"significance":{"value":4},"presentation":{"value":4},"thoroughness":{"value":4},"alternative_position":{"value":"Yes, and alternative positions are well-considered and addressed by the argument"},"strengths":{"value":"- This is a very well-written paper and highly suitable for a position paper. It has a clear problem statement, justification, and future direction.\n\n- Directly challenges the dominant monolithic LLM paradigm and proposes the concrete alternative of Expert Orchestration.\n\n- It moves beyond simple critique by proposing specific components, such as 'Judge models' and 'Routers,' and provides a clear roadmap for the research community through its discussion of open research questions (Section 6).\n\n- Grounded in reality by referencing existing technologies (e.g., Mixture-of-Experts, FrugalGPT), which supports feasibility."},"weaknesses":{"value":"- While the vision of a system where numerous models are fairly evaluated and seamlessly connected by objective 'Judge' models is compelling, it is also highly idealistic. The success of the entire framework could become critically dependent on the performance and objectivity of these Judge models.\n\n- Implementation raises major engineering challenges, including API standardization, version control, and maintaining large-scale reliability."},"context":{"value":4},"position":{"value":"Yes, the paper argues for or against a position related to machine learning."},"support":{"value":3}},"parentInvitations":"NeurIPS.cc/2025/Position_Paper_Track/-/Official_Review","nonreaders":[],"tmdate":1761706975480,"tcdate":1755394083118,"writers":["NeurIPS.cc/2025/Position_Paper_Track","NeurIPS.cc/2025/Position_Paper_Track/Submission63/Reviewer_m7e4"],"signatures":["NeurIPS.cc/2025/Position_Paper_Track/Submission63/Reviewer_m7e4"],"forum":"g8Fo6qtnMR","number":3,"license":"CC BY 4.0","cdate":1755394083118,"readers":["everyone"],"invitations":["NeurIPS.cc/2025/Position_Paper_Track/Submission63/-/Official_Review","NeurIPS.cc/2025/Position_Paper_Track/-/Edit"],"mdate":1761706975480,"domain":"NeurIPS.cc/2025/Position_Paper_Track","replyto":"g8Fo6qtnMR","id":"Mhitmf6md9","forumContent":{"TLDR":{"value":"Replace the current reliance on monolithic general-purpose LLMs with a \"judges and routers\" architecture that coordinates specialist and generalist models to deliver more accurate, cost-effective, and user-aligned AI responses."},"venue":{"value":"Submitted to NeurIPS 2025 Position Paper Track"},"keywords":{"value":["LLM","judge","router","ensemble","meta-model","transparency","trust","experts"]},"abstract":{"value":"This position paper argues that the prevailing trajectory toward ever larger, more expensive generalist foundation models controlled by a handful of big companies limits innovation and constrains progress. \nWe challenge this approach by advocating for an “Expert Orchestration\" architecture as a superior alternative that democratizes LLM advancement. \nOur proposed architecture intelligently selects from thousands of existing models based on query requirements and decomposition, focusing on identifying what models do well rather than how they work internally. \nIndependent “judge\" models assess various models' capabilities across dimensions that matter to users, while “router\" systems direct queries to the most appropriate specialists within an approved set. \nThis approach delivers superior performance by leveraging targeted expertize rather than forcing costly generalist models to address all user requirements. \nThe expert orchestration paradigm represents a significant advancement in LLM capability by enhancing transparency, control, alignment, and safety through model selection while fostering a more democratic ecosystem."},"_bibtex":{"value":"@misc{\nanonymous2025beyond,\ntitle={Beyond Monoliths: Expert Orchestration for More Capable, Democratic, and Safe Large Language Models},\nauthor={Anonymous},\nyear={2025},\nurl={https://openreview.net/forum?id=g8Fo6qtnMR}\n}"},"title":{"value":"Beyond Monoliths: Expert Orchestration for More Capable, Democratic, and Safe Large Language Models"},"pdf":{"value":"/pdf/f7718725748238ae9cc25f1b32d29c4751f3174b.pdf"},"venueid":{"value":"NeurIPS.cc/2025/Position_Paper_Track/Rejected_Submission"},"paperhash":{"value":"quirke|beyond_monoliths_expert_orchestration_for_more_capable_democratic_and_safe_large_language_models"},"authorids":{"value":["~Philip_Quirke1","~Narmeen_Fatimah_Oozeer1","~Amir_Abdullah1","~Jason_Hoelscher-Obermaier1","~Clement_Neo1","~Jeff_M._Phillips1","~Joshua_Greaves1","~Chaithanya_Bandi2","~Fazl_Barez1","~Shriyash_Kaustubh_Upadhyay1"]},"authors":{"value":["Philip Quirke","Narmeen Fatimah Oozeer","Amir Abdullah","Jason Hoelscher-Obermaier","Clement Neo","Jeff M. Phillips","Joshua Greaves","Chaithanya Bandi","Fazl Barez","Shriyash Kaustubh Upadhyay"]}},"version":2},{"content":{"comment":{"value":"Thanks for the very detailed response!\n\n> We believe beyond just the exact cost values, Automix provides a novel method to balance cost performance trade off which works across a wide variety of cost ranges and is robust across a very wide range of cost ratios.\n\nIt's worth highlighting this in the paper as the first-time reader may fail to realize this when looking at the main results.\n\n> Finally, we note that implementing the POMDP model is straightforward and fast, requiring only a few lines of code (~5) for the end-user to incorporate our library. Additionally, as mentioned in line 285, it takes less than 1 ms to run the POMDP model for an input query.\n\nThis removes my doubt.\n\n> However, note that this cost is incurred only during training and not evaluation. Further, even the training cost is similar to baselines since both FrugalGPT and HybridLLM require running the inference on all samples.\n\nPlease include this in the paper.\n\nGiven major points are addressed, I updated my score to indicate a weak support of the acceptance."}},"tmdate":1730890001474,"tcdate":1723265823001,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission20326/Reviewer_RAHf"],"signatures":["NeurIPS.cc/2024/Conference/Submission20326/Reviewer_RAHf"],"forum":"e6WrwIvgzX","number":1,"license":"CC BY 4.0","cdate":1723265823001,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission20326/-/Official_Comment","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730890001474,"domain":"NeurIPS.cc/2024/Conference","replyto":"lnLqvXvVGL","id":"2b6RxZhpYj","forumContent":{"TLDR":{"value":"AutoMix robustly routes queries among language models of varying sizes, efficiently balancing computational cost and solution accuracy"},"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["Few-shot learning","Zero-shot learning","Self-Verification","cost-quality optimization","Decision making","Prompting","LLMs"]},"primary_area":{"value":"natural_language_processing"},"abstract":{"value":"Large language models (LLMs) are now available from cloud API providers in various sizes and configurations. While this diversity offers a broad spectrum of choices, effectively leveraging the options to optimize computational cost and performance remains challenging. In this work, we present AutoMix, an approach that strategically routes queries to larger LMs, based on the approximate correctness of outputs from a smaller LM. Central to AutoMix are two key technical contributions. First, it has a few-shot self-verification mechanism, which estimates the reliability of its own outputs without requiring extensive training. Second, given that self-verification can be noisy, it employs a POMDP based router that can effectively select an appropriately sized model, based on answer confidence. Experiments across five language models and five challenging datasets show that Automix consistently surpasses strong baselines, reducing computational cost by over 50\\% for comparable performance."},"_bibtex":{"value":"@inproceedings{\naggarwal2024automix,\ntitle={AutoMix: Automatically Mixing Language Models},\nauthor={Pranjal Aggarwal and Aman Madaan and Ankit Anand and Srividya Pranavi Potharaju and Swaroop Mishra and Pei Zhou and Aditya Gupta and Dheeraj Rajagopal and Karthik Kappaganthu and Yiming Yang and Shyam Upadhyay and Manaal Faruqui and Mausam .},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=e6WrwIvgzX}\n}"},"title":{"value":"AutoMix: Automatically Mixing Language Models"},"pdf":{"value":"/pdf/f99d5e58580a4ddb7a4d60eef66db492cfafb4d9.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"aggarwal|automix_automatically_mixing_language_models"},"authorids":{"value":["~Pranjal_Aggarwal1","~Aman_Madaan1","~Ankit_Anand4","~Srividya_Pranavi_Potharaju1","~Swaroop_Mishra1","~Pei_Zhou1","~Aditya_Gupta2","~Dheeraj_Rajagopal1","~Karthik_Kappaganthu1","~Yiming_Yang1","~Shyam_Upadhyay1","~Manaal_Faruqui2","~Mausam_.1"]},"authors":{"value":["Pranjal Aggarwal","Aman Madaan","Ankit Anand","Srividya Pranavi Potharaju","Swaroop Mishra","Pei Zhou","Aditya Gupta","Dheeraj Rajagopal","Karthik Kappaganthu","Yiming Yang","Shyam Upadhyay","Manaal Faruqui","Mausam ."]}},"version":2},{"content":{"comment":{"value":"Dear reviewers, thanks for your suggestions, we include several additional small-scale experiments in this rebuttal:\n\n- **Correctness experiment on MATH EM:**  \nThis provides an objective correctness comparison between the original model and SOMA, complementing the qualitative and similarity-based metrics.\n\n| Model Family | Original (EM %) | Surrogate       | History-Prefix     | History-FT         | RouteLLM          | SOMA              |\n|--------------|------------------|-----------------|--------------------|--------------------|--------------------|-------------------|\n| **LLaMA**    | 68.0 ± 0.3       | 14.6 ± 0.8      | 28.9 ± 0.9         | 43.7 ± 1.0         | 49.8 ± 0.7         | 55.1 ± 0.6        |\n| **Qwen**     | 62.0 ± 0.4       | 14.0 ± 0.7      | 27.1 ± 0.9         | 39.9 ± 1.0         | 45.3 ± 0.7         | 50.4 ± 0.6        |\n- **Within-family experiment (e.g., LLaMA-3.1-70B → LLaMA-3.1-8B)**  \n\n| Method         | ShareGPT           | ReMeDi             | Craigslist         | Multi-Char         | MATH               | MT-Bench          | Avg                |\n|----------------|--------------------|---------------------|---------------------|---------------------|---------------------|--------------------|--------------------|\n| Surrogate      | 88.7 ± 2.13        | 86.2 ± 2.27         | 82.1 ± 2.41         | 83.4 ± 2.18         | 79.1 ± 2.63         | 85.3 ± 2.12        | 84.1 ± 2.29        |\n| History-Prefix | 91.8 ± 1.77        | 89.4 ± 1.92         | 86.0 ± 1.81         | 87.9 ± 1.73         | 83.5 ± 1.97         | 89.1 ± 1.74        | 87.9 ± 1.82        |\n| History-FT     | 94.2 ± 1.53        | 92.1 ± 1.58         | 89.3 ± 1.49         | 90.8 ± 1.38         | 87.9 ± 1.62         | 92.4 ± 1.41        | 91.1 ± 1.52        |\n| LLMLingua-2    | 90.6 ± 1.88        | 88.0 ± 1.83         | 85.1 ± 1.91         | 86.1 ± 1.76         | 82.2 ± 2.04         | 88.0 ± 1.87        | 86.7 ± 1.88        |\n| RouteLLM       | 95.8 ± 1.22        | 94.0 ± 1.17         | 91.9 ± 1.29         | 92.0 ± 1.09         | 89.9 ± 1.33         | 93.8 ± 1.14        | 92.9 ± 1.20        |\n| SOMA           | **97.1 ± 0.94**    | **95.3 ± 0.89**     | **93.4 ± 1.07**     | **93.7 ± 0.83**     | **91.8 ± 0.97**     | **95.1 ± 0.91**    | **94.4 ± 0.93**    |\n\n\nWe present the results on three datasets due to time constraints. Because LLaMA-3.1-8B is MUCH stronger than LLaMA-2-7B, the surrogate similarity rises substantially, and all baselines shift upward accordingly. However, SOMA still remain the strongest method.\n\n- **More baselines results**  \nWe additionally compare SOMA with speculative decoding and FrugalGPT cascades. Speculative decoding only verifies token-level consistency, leaving higher-level semantic errors uncorrected, while cascades may prematurely trust the small model and fail to escalate to the large model when needed. These mechanisms expose the inherent behavioral mismatch between small and large models, leading to lower similarity across all datasets. In contrast, SOMA explicitly reduces this mismatch by adapting the surrogate to the large model’s local conversational manifold, yielding consistently higher alignment with comparable efficiency.\n\n| Method          | ShareGPT (%)       | ReMeDi (%)         | Craigslist (%)     | Multi-Char (%)     | MATH (%)           | MT-Bench (%)       | Avg (%)             |\n|-----------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|----------------------|\n| Spec-Decoding| 89.7 ± 1.60       | 88.3 ± 1.75         | 82.6 ± 2.10         | 83.1 ± 1.80         | 78.8 ± 2.50         | 87.9 ± 1.46         | 85.1 ± 2.20          |\n| FrugalGPT | 92.0 ± 1.55       | 89.4 ± 1.68         | 84.1 ± 2.00         | 85.5 ± 1.75         | 80.2  ± 2.40         | 89.0 ± 1.33         | 86.7 ± 2.12          |\n| **SOMA**        | **96.4 ± 1.91**     | **93.2 ± 0.98**     | **91.9 ± 2.49**     | **92.3 ± 1.05**     | **90.7 ± 1.12**     | **94.1 ± 0.91**     | **93.1 ± 1.99**      |\n\n- **Empirical variance concentration and theoretical upper bound**  \nWe also conduct new experiments in Appendix E.6 to empirically confirming that early turns provide the most information and that SOMA’s switching rule operates within the regime assumed by Lemma 1 and Theorem 2."},"title":{"value":"Global Responses with Additional Experiment Results"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763925037683,"tcdate":1763925037683,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission22291/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission22291/Authors"],"forum":"B821Pc5huD","number":1,"license":"CC BY 4.0","cdate":1763925037683,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission22291/-/Official_Comment"],"mdate":1763925037683,"domain":"ICLR.cc/2026/Conference","replyto":"B821Pc5huD","id":"ij7oj4ybOi","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Small Language Model","multi-turn conversations","local manifold approximation"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Large Language Models (LLMs) are increasingly deployed in multi-turn dialogue settings where preserving conversational context across turns is essential. A standard serving practice concatenates the full dialogue history at every turn, which reliably maintains coherence but incurs substantial cost in latency, memory, and API expenditure, especially when queries are routed to large proprietary models. Existing approaches often struggle to balance the trade-off between response quality and efficiency. We propose a framework that exploits the early turns of a session to estimate a local response manifold and then adapt a smaller surrogate model to this local region for the remainder of the conversation. Concretely, we learn soft prompts that maximize semantic divergence between the large and surrogate small language models' responses to surface least-aligned local directions, stabilize training with anti-degeneration control, and distill the mined cases into localized LoRA fine-tuning so the surrogate runs without prompts at inference. A simple gate enables a one-time switch with rollback on drift. We further provide a theoretical analysis for key components in SOMA. Extensive experiments show the effectiveness of SOMA. The source code is provided at: https://anonymous.4open.science/r/SOMA-D377."},"_bibtex":{"value":"@misc{\ncheng2026soma,\ntitle={{SOMA}: Efficient Multi-turn {LLM} Serving via Small Language Model},\nauthor={Xueqi Cheng and Qiong Wu and Zhengyi Zhou and Xugui Zhou and Tyler Derr and Yushun Dong},\nyear={2026},\nurl={https://openreview.net/forum?id=B821Pc5huD}\n}"},"title":{"value":"SOMA: Efficient Multi-turn LLM Serving via Small Language Model"},"pdf":{"value":"/pdf/b48d6bf40b465bca46fe81bea8253ea534b19776.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"cheng|soma_efficient_multiturn_llm_serving_via_small_language_model"},"authorids":{"value":["~Xueqi_Cheng2","~Qiong_Wu3","~Zhengyi_Zhou1","~Xugui_Zhou1","~Tyler_Derr1","~Yushun_Dong1"]},"authors":{"value":["Xueqi Cheng","Qiong Wu","Zhengyi Zhou","Xugui Zhou","Tyler Derr","Yushun Dong"]}},"version":2},{"content":{"comment":{"value":"We thank the reviewer for their time and for recognizing that ABC is `intuitive, broadly applicable, practically valuable` with a `thorough analysis of performance and usage scenarios`. We appreciate these positive assessments. We address the weaknesses the reviewer brought up below:\n\n>> The method is not truly novel. A similar approach was already explored in [1], where early exit is determined by checking the consistency between the previous cascading expert and the current classification head.\n\nThere are significant differences between our work and the work of Li et al. 2023 work (IRENE). First, the motivations differ: IRENE considers ensemble learning optimization, i.e., how to train ensembles more efficiently by learning when to stop adding models. In contrast, ABC addresses inference routing, i.e., how to efficiently route requests across heterogeneous pre-trained models. The methods themselves are also quite different:\n\n- **IRENE**: Requires training specialized neural selectors with complex multi-objective losses for models you control and train jointly\n- **ABC**: Zero training required, as “agreement” is used as deferral metric; works with any collection of off-the-shelf models from different vendors.\n\nWhile you could potentially use IRENE-trained models as one tier in an ABC cascade, these two approaches are orthogonal and solve fundamentally different problems: IRENE optimizes ensemble training, while ABC optimizes inference routing. \n\n\n>> WoC is from 2021, while FrugalGPT and related baselines are from 2022–2023 and are relatively weak by today’s standards.\n\nWhile we appreciate the reviewer’s concern about baseline recency, we clarify the publication dates of the baselines we compared to:\n\n\n- FrugalGPT: TMLR December 2024\n- AutoMix: NeurIPS, December 2024\n- MoT LLM Cascade: ICLR, May 2024\n\nWe selected WoC (2021) as a simple, representative method for the classic confidence-based cascading approach. To the best of our knowledge, these baselines depict a strong representation of the current state-of-the-art. Please let us know if there is a more recent method we have missed.\n\n>> The paper includes virtually no ablation study to justify the design choices or quantify the contribution of each component, leaving the actual benefit of cascading and voting unclear.\n\nWe disagree that our work lacks ablation studies. We conducted extensive ablations throughout Section 5 and in the appendices (see, e.g., Figure 3 on component necessity, Figure 8 on parallelization, Figure 6 on threshold robustness, Tables 4-5 on cost breakdowns). Many of these were included in the appendix and thus may have been missed by the reviewer. Following the reviewer’s feedback, we have instead created a dedicated section in our revision, Section 5.3, to consolidate these comprehensive ablations for easier navigation.\n\n>> The experiments focus solely on classification tasks. Many real-world applications cannot be framed as classification problems, which calls the generality of the approach into question.\n\n\n\nWhile we acknowledged this limitation in our paper draft, it is important to note that this limitation is shared across much of the cascading field, including recent works at top venues such as NeurIPS [1, 6], ICLR [2, 3], ICML [4], and TMLR [5]. We agree and welcome the increased focus on LLM open-generation workflows given the recent advances in LLMs; however, we also note that more classical ML tasks such as classification, closed generation and sentiment analysis, especially for edge deployments are still very much relevant. The simplicity of our approach is appealing in such scenarios as we do not require any platform-specific optimizations.\n\n\n---\n[1] Aggarwal, P., Madaan, A., Anand, A., Potharaju, S. P., Mishra, S., Zhou, P., ... & Mausam, M. *Automix: Automatically mixing language models.* NeurIPS, 2024.\n\n[2] Ding, D., Mallick, A., Wang, C., Sim, R., Mukherjee, S., Rühle, V., ... & Awadallah, A. H. *Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing.* ICLR, 2024.\n\n[3] Yue, M., Zhao, J., Zhang, M., Du, L., & Yao, Z.  *Large Language Model Cascades with Mixture of Thought Representations for Cost-Efficient Reasoning.* ICLR, 2024.\n\n[4] Nie, L., Ding, Z., Hu, E., Jermaine, C., & Chaudhuri, S. *Online Cascade Learning for Efficient Inference over Streams.* ICML, 2024.\n\n[5] Chen, L., Zaharia, M., & Zou, J. *FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance.* TMLR, 2024.\n\n[6] Jitkrittum, W., Gupta, N., Menon, A. K., Narasimhan, H., Rawat, A., & Kumar, S. *When does confidence-based cascade deferral suffice?.* NeurIPS, 2023."},"title":{"value":"Response to Reviewr 91ij"}},"parentInvitations":"TMLR/-/Official_Comment","tmdate":1749591036488,"tcdate":1749579170302,"writers":["TMLR","TMLR/Paper4338/Authors"],"signatures":["TMLR/Paper4338/Authors"],"forum":"jn9B7LMlzk","number":3,"license":"CC BY 4.0","cdate":1749579170302,"readers":["everyone"],"invitations":["TMLR/Paper4338/-/Official_Comment"],"mdate":1749591036488,"domain":"TMLR","replyto":"TImZlVY7cn","id":"ZAyAnzQbQQ","forumContent":{"submission_length":{"value":"Long submission (more than 12 pages of main content)"},"venue":{"value":"Accepted by TMLR"},"code":{"value":"https://github.com/stevenkolawole/Agreement-Based-Cascading"},"abstract":{"value":"Adaptive inference schemes reduce the cost of machine learning inference by assigning smaller models to easier examples, attempting to avoid invocation of larger models when possible. In this work we explore a simple, effective adaptive inference technique we term Agreement-Based Cascading (ABC). ABC builds a cascade of models of increasing size/complexity and uses agreement between ensembles of models at each level of the cascade as a basis for data-dependent routing. Although ensemble execution introduces additional expense, we show that these costs can be easily offset in practice due to large expected differences in model sizes, parallel inference execution capabilities, and accuracy benefits of ensembling. We examine ABC theoretically and empirically in terms of these parameters, showing that the approach can reliably act as a drop-in replacement for existing models and surpass the best single model it aims to replace in terms of both efficiency and accuracy. Additionally, we explore the performance of ABC relative to existing cascading methods in three common scenarios: (1) edge-to-cloud inference, where ABC reduces communication costs by up to 14x; (2) cloud-based model serving, where it achieves a 3x reduction in rental costs; and (3) inference via model API services, where ABC achieves a 2-25x reduction in average price per token/request relative to state-of-the-art LLM cascades."},"_bibtex":{"value":"@article{\nkolawole2025agreementbased,\ntitle={Agreement-Based Cascading for Efficient Inference},\nauthor={Steven Kolawole and Don Dennis and Ameet Talwalkar and Virginia Smith},\njournal={Transactions on Machine Learning Research},\nissn={2835-8856},\nyear={2025},\nurl={https://openreview.net/forum?id=jn9B7LMlzk},\nnote={}\n}"},"title":{"value":"Agreement-Based Cascading for Efficient Inference"},"pdf":{"value":"/pdf/9c0fb8f6822c99d87fbb5b43ea8fa0be630e8670.pdf"},"venueid":{"value":"TMLR"},"paperhash":{"value":"kolawole|agreementbased_cascading_for_efficient_inference"},"authorids":{"value":["~Steven_Kolawole1","~Don_Dennis2","~Ameet_Talwalkar1","~Virginia_Smith1"]},"assigned_action_editor":{"value":"~Yaoliang_Yu1"},"authors":{"value":["Steven Kolawole","Don Dennis","Ameet Talwalkar","Virginia Smith"]}},"version":2},{"content":{"data_release":{"value":"We authorize the release of our submission and author names to the public in the event of acceptance."},"TLDR":{"value":"AgentRouter, a 12M-parameter step-level classifier, routes each agent trajectory step to one of four model tiers and reduces enterprise inference costs by 72% with less than 3% quality degradation."},"venue":{"value":"AdaptFM Poster"},"email_sharing":{"value":"We authorize the sharing of all author emails with Program Chairs."},"keywords":{"value":["model routing","resource-adaptive inference","agentic workflows","cost optimization","heterogeneous models","step-level routing","foundation models"]},"abstract":{"value":"Enterprise agentic systems that route every trajectory step to a frontier model waste 60-80% of their inference budget on subtasks that smaller models handle equally well. Existing routing solutions optimize single-turn query assignment but ignore a property unique to agentic workflows: subtask complexity varies widely within a single trajectory. A planning step may require frontier-class reasoning while a subsequent formatting step needs only a 7B model. We formalize step-level model routing as a sequential assignment problem over agent trajectories and propose AgentRouter, a lightweight classifier (12M parameters, <5ms overhead per step on an A100 GPU) that maps each trajectory step to one of four model tiers using five features extractable at routing time. Trained on 50,000 annotated agent trajectory steps spanning planning, coding, research, and data analysis tasks, AgentRouter achieves 72% cost reduction relative to frontier-only baselines, retaining 97.3% of frontier-only quality (less than 3% degradation in end-to-end task completion); per-step routing accuracy reaches 91% on minimal-complexity steps and 85% on efficient-tier steps, with 76-82% on the harder mid-range and frontier tiers. On the same benchmarks, RouteLLM and FrugalGPT (applied per-step) achieve only 31% and 44% cost reduction respectively, because their single-turn training signal misses trajectory-level quality dependencies."},"_bibtex":{"value":"@inproceedings{\npaul2026agentrouter,\ntitle={AgentRouter: Heterogeneous Model Routing for Cost-Optimal Multi-Step Agentic Workflows},\nauthor={Rudrendu Kumar Paul and Sourav Nandy},\nbooktitle={AdaptFM: Resource-Adaptive Foundation Model Inference},\nyear={2026},\nurl={https://openreview.net/forum?id=nu3GPfkyJV}\n}"},"title":{"value":"AgentRouter: Heterogeneous Model Routing for Cost-Optimal Multi-Step Agentic Workflows"},"pdf":{"value":"/pdf/6b9ccf623ac62f0a2476da7161d0aa044457b4aa.pdf"},"venueid":{"value":"ICML.cc/2026/Workshop/AdaptFM"},"paperhash":{"value":"paul|agentrouter_heterogeneous_model_routing_for_costoptimal_multistep_agentic_workflows"},"authorids":{"value":["~Rudrendu_Kumar_Paul1","~Sourav_Nandy2"]},"authors":{"value":["Rudrendu Kumar Paul","Sourav Nandy"]}},"tmdate":1780570443231,"pdate":1780353776095,"tcdate":1775875533252,"writers":["ICML.cc/2026/Workshop/AdaptFM","ICML.cc/2026/Workshop/AdaptFM/Submission2/Authors"],"signatures":["ICML.cc/2026/Workshop/AdaptFM/Submission2/Authors"],"forum":"nu3GPfkyJV","license":"CC BY 4.0","number":2,"cdate":1775875533252,"readers":["everyone"],"invitations":["ICML.cc/2026/Workshop/AdaptFM/-/Submission","ICML.cc/2026/Workshop/AdaptFM/-/Submission_Change_Before_Reviewing","ICML.cc/2026/Workshop/AdaptFM/-/Submission_Release","ICML.cc/2026/Workshop/AdaptFM/Submission2/-/Camera_Ready_Revision","ICML.cc/2026/Workshop/AdaptFM/-/Edit"],"mdate":1780570443231,"odate":1780353776095,"domain":"ICML.cc/2026/Workshop/AdaptFM","id":"nu3GPfkyJV","version":2},{"content":{"summary":{"value":"This paper introduces **DiSRouter (Distributed Self-Router)**, a framework that replaces centralized query routing in LLM systems with a distributed paradigm where each agent autonomously decides to answer or route queries based on self-awareness. Key contributions include:\n\n- A **distributed routing architecture** that enhances modularity and scalability compared to centralized routers.\n- A **Self-Awareness Training pipeline** with supervised fine-tuning (SFT) and reinforcement learning (RL) using a localized reward function, enabling independent agent training.\n- **Scenario adaptability** via a preference factor $\\alpha$ to balance performance and cost dynamically. Experiments on datasets like GSM8K and MMLU show DiSRouter outperforms baselines (e.g., RouteLLM, FrugalGPT) in utility. The core idea is that intrinsic self-assessment is more effective than external evaluation."},"soundness":{"value":3},"confidence":{"value":3},"questions":{"value":"1. I am still confused that how the the local routing policy $\\pi_i$ of agent $m_i$ is trained, it seems that $m_i$ only output \"I don't know!\" if choosing reject, how does $m_i$ select the next agent?\n2. Can DiSRouter's self-awareness training be applied to LLMs with different architectures (e.g., encoder-decoder models)? Please provide results with at least one non-Qwen model family to validate broad applicability.\n3. How does DiSRouter perform with large-scale agent pools (e.g., >10 agents)? \n4. What are the practical limitations (e.g., latency, fault tolerance) in deploying DiSRouter? Case studies or simulations with real-world constraints would be valuable."},"rating":{"value":6},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":3},"strengths":{"value":"- **Originality**: The shift from centralized to distributed routing is novel, creatively integrating self-assessment concepts (e.g., LLM uncertainty) into a routing framework. The scenario-adaptive reward function is an innovative touch.\n- **Quality**: Experimental design is rigorous within its scope, with utility metrics and modularity tests. The pipeline is methodically described.\n- **Clarity**: The problem formulation in Section 2 is precise. Writing is concise.\n- **Significance**: The \"plug-and-play\" modularity addresses a key pain point in LLM systems, though real-world impact depends on scalability beyond the tested settings."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- **Limited Generalizability**: Experiments use only the Qwen2.5-Instruct series, neglecting other LLM families (e.g. Llama). This casts doubt on DiSRouter's applicability to diverse models. The authors should include cross-architecture validation.\n- **Training Efficiency Concerns**: The Self-Awareness Training (SFT + RL) is computationally heavy, but the paper dismisses routing cost as \"negligible\" without quantifying training overhead. A cost-benefit analysis is missing.\n- **Limited Robustness Testing**: There is no evaluation under adversarial conditions (e.g., ambiguous queries) to test the robustness of self-assessment. This is critical for real-world reliability."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762919711527,"tcdate":1761903816475,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7635/Reviewer_xhvr"],"signatures":["ICLR.cc/2026/Conference/Submission7635/Reviewer_xhvr"],"forum":"KDcwXKr0NU","number":4,"license":"CC BY 4.0","cdate":1761903816475,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7635/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762919711527,"domain":"ICLR.cc/2026/Conference","replyto":"KDcwXKr0NU","id":"XKc4HTHIu8","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"TLDR":{"value":"We introduce DiSRouter, a distributed system where LLMs leverage \"self-awareness\" to route queries among themselves, outperforming conventional centralized routers."},"keywords":{"value":["query routing","model selection","distributed system","self-awareness of LLM"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"The proliferation of Large Language Models (LLMs) has created a diverse ecosystem of models with highly varying performance and costs, necessitating effective query routing to balance performance and expense. Current routing systems often rely on a centralized external router trained on a fixed set of LLMs, making them inflexible and prone to poor performance since the small router can not fully understand the knowledge boundaries of different LLMs. We introduce DiSRouter (Distributed Self-Router), a novel paradigm that shifts from centralized control to distributed routing. In DiSRouter, a query traverses a network of LLM agents, each independently deciding whether to answer or route to other agents based on its own self-awareness—its ability to judge its competence. This distributed design offers superior flexibility, scalability, and generalizability. To enable this, we propose a two-stage Self-Awareness Training pipeline that enhances each LLM's self-awareness. Extensive experiments demonstrate that DiSRouter significantly outperforms existing routing methods in utility across various scenarios, effectively distinguishes between easy and hard queries, and shows strong generalization to out-of-domain tasks. Our work validates that leveraging an LLM's intrinsic self-awareness is more effective than external assessment, paving the way for more modular and efficient multi-agent systems."},"_bibtex":{"value":"@inproceedings{\nzheng2026disrouter,\ntitle={Di{SR}outer: Distributed Self-Routing for {LLM} Selections},\nauthor={Hang Zheng and Hongshen Xu and Yongkai.lin and Shuai Fan and Lu Chen and Kai Yu},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=KDcwXKr0NU}\n}"},"title":{"value":"DiSRouter: Distributed Self-Routing for LLM Selections"},"pdf":{"value":"/pdf/0190f98d94890b700d03febdee678e22a2a5e380.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"zheng|disrouter_distributed_selfrouting_for_llm_selections"},"authorids":{"value":["~Hang_Zheng3","~Hongshen_Xu1","~Yongkai.lin1","~Shuai_Fan1","~Lu_Chen3","~Kai_Yu3"]},"authors":{"value":["Hang Zheng","Hongshen Xu","Yongkai.lin","Shuai Fan","Lu Chen","Kai Yu"]}},"version":2},{"content":{"comment":{"value":"In addition to correcting typos and formatting issues, we have revised the paper to rigorously address all reviewers' comments. The major updates are listed below:\n\n* **Optimality Gap Analysis (New Appendix D):** We added an empirical analysis comparing Agora against a Brute-Force Global Oracle, demonstrating that our decentralized equilibrium is within 0.8% of the theoretical global optimum.\n* **Cross-Dataset Comparison (New Appendix E.3):** We included head-to-head comparisons against MoA and KABB-VLM on MathVision, InfoVQA, and CC-OCR, demonstrating consistent superiority beyond MMBench.\n* **Computational Tractability (Revised Section 3.3 & New Appendix M.3):** We explicitly clarified the low computational complexity of the Broker and trading protocol, and detailed the efficient closed-form calculation of the Strategic Uncertainty Index.\n* **Fairness in Baselines (Revised Section 4.1):** We detailed the \"Unified Multimodal Wrapper\" implementation to ensure fair comparison with text-centric routers like FrugalGPT.\n* **Uncertainty Calibration (New Appendix M.2):** We added concrete calculation examples for uncertainty vectors and explained how structural normalization ensures cross-model comparability without expensive calibration.\n* **Visual Corrections:** We corrected the color descriptions in （Figure 1，Figure 5） and updated visualizations for better readability.\n* **Reference Cleanup:** We removed duplicate citations and corrected citation formats (e.g., Gale & Shapley)."},"title":{"value":"General Response (2)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763785179273,"tcdate":1763785179273,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission582/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission582/Authors"],"forum":"zeqCjGQB4U","number":10,"license":"CC BY 4.0","cdate":1763785179273,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission582/-/Official_Comment"],"mdate":1763785179273,"domain":"ICLR.cc/2026/Conference","replyto":"0ODm8GM5Ws","id":"gfFDl0EF6i","forumContent":{"TLDR":{"value":"We presents a framework that reframes coordination as a decentralized market for uncertainty."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["agent; Vision Language Model; Uncernity"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Vision-Language Models (VLMs) enable powerful multi-agent systems, but scaling them is economically unsustainable: coordinating heterogeneous agents under information asymmetry often spirals costs. Existing paradigms, such as Mixture-of-Agents and knowledge-based routers, rely on heuristic proxies that ignore costs and collapse uncertainty structure, leading to provably suboptimal coordination.\nWe introduce Agora, a framework that reframes coordination as a decentralized market for uncertainty. Agora formalizes epistemic uncertainty into a structured, tradable asset (perceptual, semantic, inferential), and enforces profitability-driven trading among agents based on rational economic rules. A market-aware broker, extending Thompson Sampling, initiates collaboration and guides the system toward cost-efficient equilibria. Experiments on five multimodal benchmarks (MMMU, MMBench, MathVision, InfoVQA, CC-OCR) show that Agora outperforms strong VLMs and heuristic multi-agent strategies, e.g., achieving +8.5% accuracy over the best baseline on MMMU while reducing cost by over 3×. These results establish market-based coordination as a principled and scalable paradigm for building economically viable multi-agent visual intelligence systems."},"_bibtex":{"value":"@inproceedings{\nzhang2026why,\ntitle={Why Keep Your Doubts to Yourself? Trading Visual Uncertainties among Vision-Language Models},\nauthor={Jusheng Zhang and Yijia Fan and Kaitong Cai and Jing Yang and Jiawei Yao and Jian Wang and Guanlong Qu and Ziliang Chen and Keze Wang},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=zeqCjGQB4U}\n}"},"title":{"value":"Why Keep Your Doubts to Yourself? Trading Visual Uncertainties among Vision-Language Models"},"pdf":{"value":"/pdf/03f4bead771e441ade14013c814de209cd0630ea.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"zhang|why_keep_your_doubts_to_yourself_trading_visual_uncertainties_among_visionlanguage_models"},"authorids":{"value":["~Jusheng_Zhang3","~Yijia_Fan1","~Kaitong_Cai1","~Jing_Yang39","~Jiawei_Yao3","~Jian_Wang10","~Guanlong_Qu2","~Ziliang_Chen1","~Keze_Wang1"]},"authors":{"value":["Jusheng Zhang","Yijia Fan","Kaitong Cai","Jing Yang","Jiawei Yao","Jian Wang","Guanlong Qu","Ziliang Chen","Keze Wang"]}},"version":2},{"content":{"comment":{"value":"## **W2: Another issue is the computational cost of the method. The method does seem more costly than certain router baselines, but performs slightly better. Have the authors considered potentially skipping an LLM when say the first model's confidence is very low, or outputting more nuanced reject answers, such as \"I don't now.\", \"I really don't know.\", and depending on such output, skip an LLM or two in the chain?**\n\nWe appreciate this practical perspective, which was echoed by other reviewers. To address concerns regarding latency and overhead, we have added a detailed statistical analysis of actual inference and routing costs in **Appendix A.1** of the revised paper.\n\n### **Latency Analysis**\n\n1. **Negligible Routing Overhead:** Our empirical data (**Table 6**) shows that although a query may be rejected by multiple agents before being solved, the **cumulative routing cost** (time spent generating rejections) remains negligible. Even in the worst-case scenario (routing to the final agent), this overhead constitutes less than **5%** of the actual inference cost of generating the final answer (**Table 7**). \n2. **Competitive Latency:** We also benchmarked **System Latency** (Time to First Token, **Table R4 and Table 8 in paper**). Results indicate that while DiSRouter incurs slightly higher latency than predictive routers (e.g., RouteLLM) which use lightweight classifiers, it is significantly faster than generative baselines that rely on full model outputs or verification steps (e.g., FrugalGPT, AutoMix).\n\n*Table R4: Time to First Token (s) Results Comparison for Different Routing Systems.*\n\n|  | RouterLLM | FrugalGPT | Automix | FORC | GraphRouter | DiSRouter |\n| --- | --- | --- | --- | --- | --- | --- |\n| Average | 0.048 | 2.620 | 3.475 | 0.070 | 0.024 | 0.081 |\n| Worst Case | 0.063 | 9.565 | 11.34 | 0.076 | 0.028 | 0.241 |\n\n### **Design of Cross-Level Routing**\n\nWe appreciate this forward-looking suggestion. We have actually initiated preliminary research into it, where agents are conditioned on system-level information to potentially route difficult queries directly to a much larger model, skipping intermediate steps. However, our initial experimental results showed no significant improvement in **system Utility**. We attribute this to two primary factors:\n\n1. **Complexity of Routing Precision:** Accurately determining which specific future agent (e.g., skipping from 1.5B to 14B vs. 7B) is required is a significantly harder problem than the binary decision of determining one's own incompetence. Our current algorithm for this fine-grained skipping is not yet precise enough to outweigh the risks of over-routing (using too large a model) or under-routing.\n2. **Marginal Cost Benefits:** As discussed in our response regarding routing overhead (and in **Appendix A.1**), the cost of traversing the cascade is already very low (<5% of inference cost). Therefore, the latency savings gained by \"skipping a level\" are minimal. The primary theoretical value of skipping would be to bypass intermediate agents that might confidently answer incorrectly. However, since our Self-Awareness Training already ensures agents reject when uncertain, this \"error skipping\" value is also diminished.\n\nWe consider cross-level routing a promising direction for optimizing extremely deep cascades or heterogeneous agent pools and plan to explore it further in future work."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764157291610,"tcdate":1764157291610,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7635/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission7635/Authors"],"forum":"KDcwXKr0NU","number":13,"license":"CC BY 4.0","cdate":1764157291610,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7635/-/Official_Comment"],"mdate":1764157291610,"domain":"ICLR.cc/2026/Conference","replyto":"2E7rUSRc1n","id":"nkycQnT7To","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"TLDR":{"value":"We introduce DiSRouter, a distributed system where LLMs leverage \"self-awareness\" to route queries among themselves, outperforming conventional centralized routers."},"keywords":{"value":["query routing","model selection","distributed system","self-awareness of LLM"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"The proliferation of Large Language Models (LLMs) has created a diverse ecosystem of models with highly varying performance and costs, necessitating effective query routing to balance performance and expense. Current routing systems often rely on a centralized external router trained on a fixed set of LLMs, making them inflexible and prone to poor performance since the small router can not fully understand the knowledge boundaries of different LLMs. We introduce DiSRouter (Distributed Self-Router), a novel paradigm that shifts from centralized control to distributed routing. In DiSRouter, a query traverses a network of LLM agents, each independently deciding whether to answer or route to other agents based on its own self-awareness—its ability to judge its competence. This distributed design offers superior flexibility, scalability, and generalizability. To enable this, we propose a two-stage Self-Awareness Training pipeline that enhances each LLM's self-awareness. Extensive experiments demonstrate that DiSRouter significantly outperforms existing routing methods in utility across various scenarios, effectively distinguishes between easy and hard queries, and shows strong generalization to out-of-domain tasks. Our work validates that leveraging an LLM's intrinsic self-awareness is more effective than external assessment, paving the way for more modular and efficient multi-agent systems."},"_bibtex":{"value":"@inproceedings{\nzheng2026disrouter,\ntitle={Di{SR}outer: Distributed Self-Routing for {LLM} Selections},\nauthor={Hang Zheng and Hongshen Xu and Yongkai.lin and Shuai Fan and Lu Chen and Kai Yu},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=KDcwXKr0NU}\n}"},"title":{"value":"DiSRouter: Distributed Self-Routing for LLM Selections"},"pdf":{"value":"/pdf/0190f98d94890b700d03febdee678e22a2a5e380.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"zheng|disrouter_distributed_selfrouting_for_llm_selections"},"authorids":{"value":["~Hang_Zheng3","~Hongshen_Xu1","~Yongkai.lin1","~Shuai_Fan1","~Lu_Chen3","~Kai_Yu3"]},"authors":{"value":["Hang Zheng","Hongshen Xu","Yongkai.lin","Shuai Fan","Lu Chen","Kai Yu"]}},"version":2},{"content":{"comment":{"value":"**Q1. Add a small human user study to validate simulated preferences.** \n\n**Response:**  Thanks for your constructive feedback. Following your constructive suggestion, we additionally collected a small-scale real-user dataset consisting of 10 LLMs, 40 users, and 80 queries, and conducted an additional LLM-as-a-Judge experiment. The detailed results are presented in Table 1. These results demonstrate that PersonalizedRouter generalizes beyond simulated settings and is also effective in real-user scenarios.\n\n**Table1:Comparison between different methods under small-scale human interaction dataset. The Improvement is measured relative to the best baseline.** \n\n| Scenario    |       LLM-as-a-Judge strategy      |\n|-------------|------------------------------------|\n| Method      |       Accuracy &nbsp;  Improvement (%)   |\n| HybridLLM   |       0.315  &emsp;&emsp;&ensp;   -23.73             |\n| FrugalGPT   |       0.350  &emsp;&emsp;&ensp;   -15.25             |\n| GraphRouter |       0.357  &emsp;&emsp;&ensp;   -13.56             |\n| RouterDC    |       0.413  &emsp;&emsp;&ensp;&nbsp;    0.00              |\n| PersonalizedRouter| 0.438  &emsp;&emsp;&ensp;&nbsp;    6.05              |\n| Oracle      |       1.000  &emsp;&emsp;&ensp;   142.13             |\n\n**Q2. Extend the model to handle dynamic, evolving user preferences.** \n\n**Response:** Thanks for your constructive advice. Real users may make different choices for the same query due to dynamic psychological states. To better simulate this realistic setting, we conducted additional LLM-as-a-Judge experiments. In the new experimental setting, we employed three LLMs, each evaluated with two types of instruction prompts. Consequently, each user query was evaluated under six different LLM-judge settings, thereby simulating the variability of real users’ psychological states when responding to the same query. This design allows us to assess whether our current model can effectively handle dynamic user preferences. The results of these new LLM-as-a-Judge experiments are shown in Table2 , demonstrating that the current model performs well in the dynamic user preference experimental setting.\n\n**Table2: Comparison between different methods under two simulation strategies in the large-scale experimental setting. The Deltas represent the improvement of the Reward over the best baseline. The Reduction indicates the relative reduction in time cost compared to the baseline with the highest time cost.** \n\n| Scenario    | multi-cost-efficiency simulation             | LLM-as-a-Judge strategy                |\n|-------------|----------------------------------------------|----------------------------------------|\n| Method      | Reward &nbsp;  Delta (%) &nbsp;  Time   &nbsp;  Reduction (%)   | Accuracy &nbsp;  Delta (%) &nbsp;  Time   &nbsp; Reduction (%) |\n| HybridLLM   | 0.119  &emsp;  -43.33    &ensp;&emsp;  4:32   &emsp;&emsp;  98.50  &nbsp;         | 0.196    &emsp;&emsp;&emsp;  0.00      &emsp;&emsp;  4:56   &emsp;&nbsp; 98.40         |\n| FrugalGPT   | 0.141  &emsp;  -32.86    &ensp;&emsp;  47:44  &emsp;&emsp;  84.29  &nbsp;         | 0.108    &emsp;&emsp; -44.90     &emsp;&emsp;  49:31  &ensp;&nbsp; 83.92         |\n| GraphRouter | 0.204  &emsp;  -2.86     &ensp;&emsp;&ensp;  8:26   &emsp;&emsp;  97.22  &nbsp;         | 0.137    &emsp;&emsp; -30.10     &emsp;&emsp;  9:28   &emsp;&ensp; 96.93         |\n|RouterDC | 0.210  &nbsp;&emsp; 0.00     &ensp;&emsp;  303:47 &emsp;&emsp;  0.00   &nbsp;         | 0.153    &emsp;&emsp; -21.94     &emsp;&emsp;  308:02 &ensp;&nbsp; 0.00          |\n| Ours        | 0.244  &emsp;   16.19    &ensp;&emsp; 10:15   &emsp;&emsp; 96.63   &nbsp;         | 0.313    &emsp;&emsp;&nbsp;  59.69     &emsp;&emsp;  11:37  &emsp;&nbsp; 96.23         |\n| Oracle      | 0.310  &emsp;   47.62    &ensp;&emsp;  /      &emsp;&emsp;  /      &nbsp;         | 1.000    &emsp;&emsp; 410.20     &emsp;&emsp;  /      &ensp;&nbsp; /             |"},"title":{"value":"Response to Reviewer iphw (Part 1)"}},"parentInvitations":"TMLR/-/Official_Comment","tmdate":1758311248740,"tcdate":1758311248740,"writers":["TMLR","TMLR/Paper5500/Authors"],"signatures":["TMLR/Paper5500/Authors"],"forum":"W80eE3ArAl","number":5,"license":"CC BY 4.0","cdate":1758311248740,"readers":["everyone"],"invitations":["TMLR/Paper5500/-/Official_Comment"],"mdate":1758311248740,"domain":"TMLR","replyto":"eoIk9QVWx4","id":"gNdXoRJ5En","forumContent":{"submission_length":{"value":"Regular submission (no more than 12 pages of main content)"},"venue":{"value":"Accepted by TMLR"},"code":{"value":"https://github.com/ulab-uiuc/PersonalizedRouter"},"supplementary_material":{"value":"/attachment/e4bbeecf2accf23bec60f721de3508b2cf488096.zip"},"abstract":{"value":"The growing number of Large Language Models (LLMs) with diverse capabilities and response styles provides users with a wider range of choices, which presents challenges in selecting appropriate LLMs, as user preferences vary in terms of performance, cost, and response style. Current LLM selection methods typically optimize for a single fixed objective, such as performance, cost, or a trade-off between them, and fail to learn individual user preferences from interaction data. To address these limitations, we propose PersonalizedRouter, a graph-based framework that models diverse user profiles and performs personalized LLM selection by leveraging interaction data that includes task context, queries, candidate LLMs, and user decisions. To capture contextual information between user queries and optimal LLMs, PersonalizedRouter converts the interaction data into a heterogeneous graph, where the relationships between different types of nodes are represented by edges. To evaluate adaptability across users, we design two strategies: the multi-cost-efficiency simulation strategy and the LLM-as-a-Judge strategy. In addition, we construct PersonaRoute-Bench, a large-scale benchmark with 1,000 simulated users and 10 LLMs. Experimental results show that PersonalizedRouter significantly outperforms existing LLM selection methods and surpasses the strongest methods by a large margin of 15.38% and 9.83% under two simulation strategies. On the PersonaRoute-Bench with 1,000 users, it further surpasses the best methods by 16.19% and 59.69% while maintaining higher efficiency. Moreover, PersonalizedRouter demonstrates strong few-shot generalization, achieving 64.81% and 85.80% of the fully trained model’s performance when adapting to new users and new LLMs."},"_bibtex":{"value":"@article{\ndai2025personalizedrouter,\ntitle={PersonalizedRouter: Personalized {LLM} Routing via Graph-based User Preference Modeling},\nauthor={Zhongjie Dai and Tao Feng and Jiaxuan You},\njournal={Transactions on Machine Learning Research},\nissn={2835-8856},\nyear={2025},\nurl={https://openreview.net/forum?id=W80eE3ArAl},\nnote={}\n}"},"title":{"value":"PersonalizedRouter: Personalized LLM Routing via Graph-based User Preference Modeling"},"pdf":{"value":"/pdf/1c7d7c4a7a37c3cd71672b102988c3788f67c8e7.pdf"},"venueid":{"value":"TMLR"},"paperhash":{"value":"dai|personalizedrouter_personalized_llm_routing_via_graphbased_user_preference_modeling"},"authorids":{"value":["~Zhongjie_Dai1","~Tao_Feng5","~Jiaxuan_You2"]},"assigned_action_editor":{"value":"~Yen-Chang_Hsu1"},"authors":{"value":["Zhongjie Dai","Tao Feng","Jiaxuan You"]}},"version":2},{"content":{"comment":{"value":"> We acknowledge that the referenced work [1] attempted to address the routing problem, but their results clearly fall short of the performance of their underlying models (see Table 3 of [1]). \n\nIndeed, Table 3 of [1] summarizes the accuracy of their proposed meta-model, a light-weight model performance predictor, on different datasets and for different underlying models. Not sure how the authors drew the conclusion that “their results clearly fall short of the performance of their underlying models” from Table 3 of [1]. Moreover, it also suggests that light-weight performance predictors have been proposed and studied in prior literature [1], which nullifies the claimed contributions from the performance predictor perspective of Routoo.\n\n> However, the superior experimental results of our approach show that we have tackled this problem far more effectively.\n\nWithout an apple-to-apple comparison, it remains unclear that if the proposed Routoo “tackled this problem far more effectively” than prior literature [1,2]. Specifically, in [2], FrugalGPT can “match the performance of GPT-4 with up to 98% cost reduction or improve the accuracy over GPT-4 by 4% with the same cost”, which seems even better than Routoo, which “nearly matches GPT4’s performance at half the cost and exceeds it with a 25% cost reduction”.\n\n> Finally, I refer to the ICLR guidelines, which state that the novelty of a work is determined by both the proposed formulation and the experimental results.\n\nI fully agree with this point, which makes it critical to understand if the proposed formulation and experimental results in this work significantly improve on prior literature. However, as discussed in my previous comments, it remains unclear according to the current manuscript.\n
\n> Thank you for your comment. Lines 288-290 in Section 3 present an approximation proposal intended for general use cases of the Routoo model, not exclusively for the MMLU benchmark, as we did not specify that our greedy algorithm is designed solely for MMLU. \n\nIf Routoo is designed for the “general use cases”, please conduct evaluation on the “general use cases”. Otherwise, only considering benchmarks of single output tokens seems not comprehensive to evaluate the effectiveness of Routoo from the cost perspective.\n\n\n> Lastly, we appreciate the reviewer’s feedback, but we believe that the insistence on including this specific reference [1] may not be entirely impartial, and we respectfully suggest that the value of our work should be assessed based on its broader scientific merits and relevance.\n\nPlease be assured that all my previous comments are impartial. The only reason why [1] is referenced several times in my comments is that it shares notable similarity with the proposed Routoo which makes a thorough discussion necessary to better appreciate the value of this work.\n\n[1] Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modeling. https://arxiv.org/abs/2308.06077.  \n[2] FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance, https://arxiv.org/pdf/2305.05176"}},"tmdate":1733131773443,"tcdate":1733131773443,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission11514/Reviewer_hRVj"],"signatures":["ICLR.cc/2025/Conference/Submission11514/Reviewer_hRVj"],"forum":"RQ9fQLEajC","number":23,"license":"CC BY 4.0","cdate":1733131773443,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission11514/-/Official_Comment"],"mdate":1733131773443,"domain":"ICLR.cc/2025/Conference","replyto":"vqVx0y12ok","id":"LVr5U7LZgC","forumContent":{"TLDR":{"value":"An LLM-based Router trained to route to the most effective LLM optimizing performance, cost, and efficiency, creating a new state-of-the-art model by integrating knowledge of underlying models."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["LLM","efficiency","knowledge integration","generation"]},"supplementary_material":{"value":"/attachment/2e84b1f1c381960311d11bdd23f4d081c4e4d14a.zip"},"primary_area":{"value":"generative models"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"LLMs with superior response quality—particularly larger or closed-source models—often come with higher inference costs, making their deployment inefficient and costly. Meanwhile, developing foundational LLMs from scratch is becoming increasingly resource-intensive and impractical for many applications. To address the challenge of balancing quality and cost, we introduce Routoo, an architecture designed to optimize the selection of LLMs for specific prompts based on performance, cost, and efficiency. Routoo provides controllability over the trade-off between inference cost and quality, enabling significant reductions in inference costs for a given quality requirement.\nRoutoo comprises two key components: a performance predictor and cost-aware selector. The performance predictor is a lightweight LLM that estimates the expected performance of various underlying LLMs on a given prompt without executing them. The cost-aware selector module then selects the most suitable model based on these predictions and constraints such as cost and latency, significantly reducing inference costs for the same quality. \nWe evaluated Routoo using the MMLU benchmark across 57 domains employing open-source models. Our results show that Routoo matches the performance of the Mixtral 8x7b model while reducing inference costs by one-third. Additionally, by allowing increased costs, Routoo surpasses Mixtral's accuracy by over 5\\% at equivalent costs, achieving an accuracy of 75.9\\%. When integrating GPT4 into our model pool, Routoo nearly matches GPT4's performance at half the cost and exceeds it with a 25\\% cost reduction. \nThese outcomes highlight Routoo's potential to significantly reduce inference costs without compromising quality, and even to establish new state-of-the-art results by leveraging the collective capabilities of multiple LLMs."},"_bibtex":{"value":"@misc{\nmohammadshahi2025routoo,\ntitle={Routoo: Learning to Route to Large Language Models Effectively},\nauthor={Alireza Mohammadshahi and Arshad Rafiq Shaikh and Majid Yazdani},\nyear={2025},\nurl={https://openreview.net/forum?id=RQ9fQLEajC}\n}"},"title":{"value":"Routoo: Learning to Route to Large Language Models Effectively"},"pdf":{"value":"/pdf/454cdce2a12d959ded0bc50a6c5b05c494a2d4dd.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"mohammadshahi|routoo_learning_to_route_to_large_language_models_effectively"},"authorids":{"value":["~Alireza_Mohammadshahi1","~Arshad_Rafiq_Shaikh1","~Majid_Yazdani1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Alireza Mohammadshahi","Arshad Rafiq Shaikh","Majid Yazdani"]}},"version":2},{"content":{"venue":{"value":"Crossref"},"pdf":{"value":"https://www.techrxiv.org/doi/pdf/10.36227/techrxiv.177220388.80392106"},"venueid":{"value":"OpenReview.net/Public_Article"},"paperhash":{"value":"sharma|phiguard_complianceaware_llm_routing_for_healthcare_with_distributionfree_safety_guarantees"},"html":{"value":"https://doi.org/10.36227/techrxiv.177220388.80392106/v1"},"abstract":{"value":"Healthcare organizations deploying Large Language Models face a regulatory tradeoff: route queries through cloud APIs for cost efficiency and risk exposing protected health information (PHI), or keep everything on-premises. Existing LLM routers (RouteLLM, FrugalGPT, PILOT, BaRP) optimize cost against quality-none enforce data-residency constraints. We propose PHI-GUARD, a framework that treats data sensitivity as a hard compliance constraint. We develop SafeTS (Safetyconstrained Thompson Sampling), which builds safe action sets from classifier posteriors, and CARES (Compliance-Aware Residual Envelope Scoring), which provides distribution-free safety guarantees via asymmetric conformal residuals-requiring only exchangeability, not calibration. On 30,000 MIMIC-IV clinical queries with 10,000-query MIMIC-III cross-validation, our Dis-tilBERT classifier achieves 98.4% accuracy with 97.4% T3 (PHI) recall, but 8 misclassifications exceed confidence 0.80 (max 0.997). Static routing leaks 0.92% of PHI queries. SafeTS reduces violations from 83 to 1-8 through safe action sets but cannot eliminate them. CARES is the only strategy achieving zero violations with a verified distribution-free bound (δ + 1/(n + 1) = 0.005), routing 35.6% to cloud. Classifier ablation confirms CARES maintains zero violations across all tested architectures-from DistilBERT (98.4% accuracy) to a rule-based heuristic (56.2% accuracy)-demonstrating that the guarantee adapts to classifier quality automatically. Cross-dataset evaluation confirms CARES maintains zero violations despite 11-point T3 recall degradation on MIMIC-III, while StaticILP violations increase 5.8×. End-toend validation with real API endpoints (GPT-4o-mini, Llama-3.1) confirms safety guarantees transfer to live execution."},"title":{"value":"PHI-GUARD: Compliance-Aware LLM Routing for Healthcare with Distribution-Free Safety Guarantees"},"authors":{"value":[{"fullname":"Aman Sharma","username":"~Aman_Sharma8"}]}},"tmdate":1789090137990,"pdate":1772150400000,"externalIds":["doi:10.36227/techrxiv.177220388.80392106/v1"],"tcdate":1774328534056,"writers":["~","OpenReview.net/Public_Article/ORCID.org","OpenReview.net/Support"],"signatures":["~Aman_Sharma8"],"forum":"s4IdSmvYM3","license":"CC BY-SA 4.0","number":58966,"cdate":1772255937381,"readers":["everyone"],"invitations":["OpenReview.net/Public_Article/ORCID.org/-/Record","OpenReview.net/Public_Article/-/Edit","OpenReview.net/-/Edit"],"mdate":1789090137990,"domain":"OpenReview.net/Public_Article","id":"s4IdSmvYM3","version":2},{"content":{"comment":{"value":"Thank you for the supportive and constructive review. In response, we have added a dedicated related-work paragraph situating the method among uncertainty-based routing approaches (§2) and a new analysis quantifying how retrieval shifts the judge's uncertainty (§5). We respond to each point below.\n\n---\n\n## 1. Positioning relative to uncertainty-based routing literature\n\nThe connection to uncertainty-based routing for more capable models and tool-calling is a valuable framing we had underweighted. The revised §2 includes a new paragraph covering this line of work explicitly.\n\nFrugalGPT (Chen et al. 2023) and AutoMix (Madaan et al. 2023) route queries between models of different capacity based on estimated answer correctness, treating model invocation as a cost-quality tradeoff. FLARE (Jiang et al. 2023) triggers retrieval mid-generation when the generator's own next-token probabilities fall below a threshold. Our framework sits in the same broader paradigm of uncertainty-driven escalation, with two distinctions worth surfacing.\n\nFirst, the statistical object differs. The cited works optimize average accuracy or cost without a formal risk guarantee. We bound the false discovery rate among non-abstained verdicts with a high-probability finite-sample guarantee, which is what enables downstream deployments that need a contractual error ceiling.\n\nSecond, the escalation target differs. FrugalGPT and AutoMix escalate to a stronger LLM; FLARE escalates to retrieval during candidate generation. We escalate from a parametric judge to a retrieval-augmented judge for meta-evaluation, bounding the rate at which incorrect judge verdicts are accepted rather than incorrect candidate answers. The judge-side formulation is what propagates trustworthiness into model rankings, reward modeling, and data curation.\n\nThe revised §2 includes these three references and the framing above.\n\n---\n\n## 2. Effect of the search call on the judge's uncertainty\n\nThe revised Section 5 includes a paragraph quantifying this effect. Using the existing experimental data (Qwen3-8B candidate × Qwen3-8B judge × $k{=}3$, llm-evaluator admission, PE), we compute the per-item change $\\Delta U = U_2 - U_1$ between Mode 1 (parametric, pre-search) and Mode 2 (with retrieved snippets) uncertainty scores. Across the four datasets, the distribution of $\\Delta U$ is not monotonically negative. Retrieval typically reduces uncertainty when the snippets are relevant, but can increase it when retrieved evidence introduces ambiguity or contradicts the model's parametric belief.\n\nThe main configuration produces the following distribution.\n\n| Dataset | % items $U_2 > U_1$ | mean $\\Delta U$ (when increased) | % items $U_2 < U_1$ | mean $\\Delta U$ (when decreased) |\n|---|---:|---:|---:|---:|\n| TriviaQA | 21.1% | $+0.091$ | 78.8% | $-0.057$ |\n| HotpotQA | 29.5% | $+0.122$ | 70.5% | $-0.121$ |\n| NQ-Open | 28.0% | $+0.109$ | 72.0% | $-0.086$ |\n| PopQA | 14.5% | $+0.095$ | 85.5% | $-0.101$ |\n\nThe fraction of items where retrieval makes the judge more uncertain ranges from 14.5% (PopQA, where most answers are single named entities cleanly retrievable from Wikipedia) to 29.5% (HotpotQA, where multi-hop questions surface partially relevant snippets and competing sources). These higher-$U_2$ items are concentrated where retrieved snippets are partially relevant or where multiple sources disagree, exactly the items the framework routes to abstention via the third branch of the two-mode policy.\n\nThis asymmetry also reinforces the two-threshold design: the second threshold $t_2$ routes items to abstention when retrieval fails to resolve the judge's uncertainty. \n\n---\nWe thank you again for the thoughtful review. The points you raised have helped us improve the paper's positioning and analysis, which we have incorporated into the revised version."},"title":{"value":"Response to 3NmK: Related-work positioning and search-uncertainty analysis"}},"parentInvitations":"colmweb.org/COLM/2026/Conference/-/Official_Comment","tmdate":1787538115431,"tcdate":1780376676982,"writers":["colmweb.org/COLM/2026/Conference","colmweb.org/COLM/2026/Conference/Submission2359/Authors"],"signatures":["colmweb.org/COLM/2026/Conference/Submission2359/Authors"],"forum":"PWv1RIS6QK","number":10,"license":"CC BY 4.0","cdate":1780376676982,"readers":["everyone"],"invitations":["colmweb.org/COLM/2026/Conference/Submission2359/-/Official_Comment","colmweb.org/COLM/2026/Conference/-/Edit"],"mdate":1787538115431,"domain":"colmweb.org/COLM/2026/Conference","replyto":"t0vBKXtIkH","id":"4mGqQ3yT89","forumContent":{"TLDR":{"value":"We equip LLM-as-a-judge with calibrated uncertainty thresholds and adaptive web retrieval that enable provable error-rate guarantees while maximizing the fraction of instances it evaluates."},"venue":{"value":"COLM 2026"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the COLM Code of Ethics on https://colmweb.org/CoE.html"},"LLM_usage":{"value":"In accordance with the Policy on the use of Large Language Models in the Call for Papers https://colmweb.org/cfp.html, I certify that my submission discloses any substantive use of LLMs in the research and paper-writing process."},"keywords":{"value":["LLM as a judge","LLM as evaluator","verification","factuality","Conformal Prediction","Risk Control"]},"abstract":{"value":"Using LLMs as judges has become standard practice for evaluating model outputs at scale. This is particularly common for subjective, open-ended tasks such as assessing helpfulness or alignment, where no single reference answer exists. However, objective tasks introduce a distinct reliability challenge for reference-free LLM judging. In the absence of a reference answer, the judge evaluates factual correctness either through its parametric knowledge or through tool augmentation. Although the former enables efficient evaluation, the judge may hallucinate or lack sufficient evidence for its verdict. Conversely, tool augmentation can provide additional evidence but introduces extra computational cost and requires an appropriate mechanism to determine when and how that evidence should be used reliably. More importantly, neither approach alone provides formal control over the risk of accepted verdicts or guarantees their reliability at a specified level. We propose a risk-controlled framework that calibrates uncertainty thresholds on a held-out set so that the false discovery rate among accepted verdicts remains below a user-specified level $\\alpha$ with high probability, using finite-sample Clopper-Pearson intervals. When the parametric mode is not sufficiently confident, the instance is routed to a retrieval-augmented mode, where the judge gathers web evidence and re-evaluates the instance under a second calibrated threshold. The finite-sample guarantee carries over to this two-threshold routing without additional assumptions. Across open-domain QA benchmarks and judges of varying scales, the framework maintains the target error rate while achieving substantially higher coverage than single-mode baselines."},"_bibtex":{"value":"@inproceedings{\nbadshah2026judge,\ntitle={Judge, Retrieve, or Abstain: Uncertainty-Guarded {LLM} Judging with Provable Risk Guarantees},\nauthor={Sher Badshah and Ali Emami and Hassan Sajjad},\nbooktitle={Third Conference on Language Modeling},\nyear={2026},\nurl={https://openreview.net/forum?id=PWv1RIS6QK}\n}"},"title":{"value":"Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guarantees"},"pdf":{"value":"/pdf/b902e2e17faa4119ce3e7dda520d5e78b160365e.pdf"},"venueid":{"value":"colmweb.org/COLM/2026/Conference"},"paperhash":{"value":"badshah|judge_retrieve_or_abstain_uncertaintyguarded_llm_judging_with_provable_risk_guarantees"},"authorids":{"value":["~Sher_Badshah1","~Ali_Emami3","~Hassan_Sajjad1"]},"author_guide":{"value":"I certify that this submission complies with the submission instructions as described on https://colmweb.org/AuthorGuide.html"},"authors":{"value":["Sher Badshah","Ali Emami","Hassan Sajjad"]}},"version":2},{"content":{"comment":{"value":"We are pleased that our rebuttal addressed the reviewer’s concerns and greatly appreciate the reviewer raising the score.\n\n**1. Hint help/hurt breakdown:** We thank the reviewer for this suggestion. We provide a breakdown across both math benchmarks. For each query, we compare the SLM's correctness without a hint against correctness with the best hint size from [10%,20%,...,90%] of LLM response:\n\n| Benchmark | Total | Helped | No Diff. | Hurt |\n|---|---|---|---|---|\n| GSM8K | 776 | 198 (25.5%) | 541 (69.7%) | 37 (4.8%) |\n| CNK12 | 2,147 | 1,039 (48.4%) | 862 (40.1%) | 246 (11.5%) |\n\nThe hurt-to-helped ratio – 4.8% vs. 25.5% on GSM8K and 11.5% vs. 48.4% on CNK12 – confirms that hints are far more likely to improve SLM accuracy than degrade it.\n\n*Does the predictor avoid requesting hints when they would hurt?* We checked how Reactive Shepherding handles the hurt queries specifically:\n\n| Benchmark | Hurt Queries | Predictor Correctly Avoided | Still Hurt |\n|---|---|---|---|\n| GSM8K | 37 | 34 | 3 |\n| CNK12 | 246 | 197 | 49 |\n\nWhile the hurt rate with the best hint size on CNK12 is 11.5%, our predictor reduces this to an effective hurt rate of only 49/2,147 (2.3%) by correctly withholding hints for 80.1% of those cases. On GSM8K, the predictor reduces the hurt rate from 4.8% to just 0.4%. This confirms that the two-stage predictor effectively learns to withhold hints when they would be counterproductive – and that the 11.5% hurt rate on CNK12 is precisely why the predictor exists.\n\n**2. Pareto curves**: We have carefully examined the Pareto curves and identified two regions where Reactive Shepherding underperforms:\n\nMBPP: FrugalGPT achieves slightly higher accuracy at very low budgets (cost ≤ \\\\$0.0075).\n\nCNK12: FrugalGPT similarly outperforms Reactive Shepherding at low budgets (cost ≤ \\\\$0.22).\n\nIn both cases, the gap is confined to the lowest-cost region of the curve. As budget increases even marginally, shepherding's accuracy rises sharply – once sufficient budget is available, the system can allocate varying hint sizes across queries, and this flexibility quickly overtakes routing and cascading strategies. On GSM8K and HumanEval, Reactive Shepherding dominates across the entire cost range. We will add this analysis to the revised paper.\n\n**3.Example of failure:** We thank the reviewer for this question — it motivated us to closely examine cases where the LLM hint led the SLM astray, revealing insights that may help our algorithm design. We identified two distinct patterns:\n\n*Error propagation*: The LLM makes a subtle reasoning error: a double-subtraction (beehive problem), an off-by-one miscount (pizza slices), or a misreading of the problem setup (gift bags) — and the SLM anchors to the flawed hint, abandoning its own correct reasoning. For instance, in the pizza-slices problem, the SLM correctly identifies 5 people without a hint but switches to the LLM's incorrect count of 6 when the hint is provided.\n\n*Error amplification*: The LLM introduces a spurious step (multiplying by 24 hours in an electricity bill problem), and the SLM not only follows this error but over-extends it — the LLM answer is \\\\$504 (wrong), while the SLM under the bad hint produces \\\\$864, far worse than both the LLM's error and the correct answer of \\\\$21. This shows that bad hints can be worse than no hint at all.\n\nThese cases are rare (affecting, for example,  0.4% queries in case of GSM8K after our predictor filtering) but they point to concrete directions for strengthening the shepherding policy. For example, incorporating a post-hint verification step that compares the SLM's hinted and unhinted answers, or detecting when the SLM's confidence drops after receiving a hint. We will include this analysis and discuss these design extensions in the revised paper"}},"parentInvitations":"ICML.cc/2026/Conference/-/Reply_Rebuttal_Comment","tmdate":1782369957377,"tcdate":1775258485505,"writers":["ICML.cc/2026/Conference","ICML.cc/2026/Conference/Submission30175/Authors"],"signatures":["ICML.cc/2026/Conference/Submission30175/Authors"],"forum":"H8rywbFXuM","number":1,"license":"CC BY 4.0","cdate":1775258485505,"readers":["everyone"],"invitations":["ICML.cc/2026/Conference/Submission30175/Official_Review3/Rebuttal1/Rebuttal_Acknowledgement1/-/Reply_Rebuttal_Comment","ICML.cc/2026/Conference/-/Edit"],"mdate":1782369957377,"domain":"ICML.cc/2026/Conference","replyto":"KeG7FTLDci","id":"GCgUdW2B0s","forumContent":{"venue":{"value":"Submitted to ICML 2026"},"keywords":{"value":["LLM routing; LLM cascading; SLM-LLM collaboration; Cost-efficient inference"]},"_bibtex":{"value":"@misc{\nanonymous2026pay,\ntitle={Pay for Hints, Not Answers: {LLM} Shepherding for Cost-Efficient Inference},\nauthor={Anonymous},\nyear={2026},\nurl={https://openreview.net/forum?id=H8rywbFXuM}\n}"},"title":{"value":"Pay for Hints, Not Answers: LLM Shepherding for Cost-Efficient Inference"},"paperhash":{"value":"dong|pay_for_hints_not_answers_llm_shepherding_for_costefficient_inference"},"primary_area":{"value":"deep_learning->large_language_models"},"abstract":{"value":"Large Language Models (LLMs) deliver state-of-the-art performance on complex reasoning tasks, but their inference costs limit deployment at scale. Small Language Models (SLMs) offer dramatic cost savings yet lag substantially in accuracy. Existing approaches—routing and cascading—treat the LLM as an all-or-nothing resource: either the query bypasses the LLM entirely, or the LLM generates a complete response at full cost. We introduce **LLM Shepherding**, a framework that requests only a short prefix (a hint) from the LLM and provides it to SLM. This simple mechanism is surprisingly effective for math and coding tasks: even hints comprising 10--30\\% of the full LLM response improve SLM accuracy significantly. Shepherding generalizes both routing and cascading, and it achieves lower cost under oracle decision-making. We develop a two-stage predictor that jointly determines whether a hint is needed and how many tokens to request. On the widely-used mathematical reasoning (GSM8K, CNK12) and code generation (HumanEval, MBPP) benchmarks, Shepherding reduces costs by 42--94\\% relative to LLM-only inference. Compared to state-of-the-art routing and cascading baselines, shepherding delivers up to 2.8$\\times$ cost reduction while matching accuracy. To our knowledge, this is the first work to exploit token-level budget control for SLM-LLM collaboration. Code is available at https://anonymous.4open.science/r/LLM_Sheperding-77C3/."},"pdf":{"value":"/pdf/b5f1805f8847279424d1a89d1e99c69e9d93c924.pdf"},"venueid":{"value":"ICML.cc/2026/Conference/Rejected_Submission"},"authorids":{"value":["~Ziming_Dong1","~Hardik_Sharma3","~Evan_O'Toole1","~Jaya_Prakash_Champati1","~Kui_Wu1"]},"authors":{"value":["Ziming Dong","Hardik Sharma","Evan O'Toole","Jaya Prakash Champati","Kui Wu"]}},"version":2},{"content":{"comment":{"value":"## **W1: The core utility metric (Utility = Performance - α * Cost) is flawed. It completely ignores the significant cumulative latency incurred by the cascade structure. A query rejected multiple times will have a real-world response time far exceeding that of a single-shot centralized router, a critical cost factor this paper overlooks.**\n\nWe appreciate the reviewer’s rigorous scrutiny regarding the formulation of our Utility metric. We wish to clarify that the omission of routing overhead in the original formula was a **deliberate simplification for mathematical conciseness**, predicated on our analysis that this term is **statistically negligible** compared to inference costs, as we’ve declared in **Section 2.1** and discussed in **Appendix A.1**.\n\nWhile the omission was a statistically justified simplification, we agree with the reviewer that explicitly formalizing this term enhances theoretical rigor. We will **revise the formal definition** of the cost function in the later version of manuscript to explicitly include the routing overhead term, ensuring the methodology is mathematically flawless.\n\nTo validate this design choice and further address your concern, we provide a detailed breakdown proving that the \"cumulative latency\" does not materially affect the cost-benefit landscape.\n\n### **1. Empirical Analysis of Routing Overhead**\n\nAs detailed in **Appendix A.1**, we conducted a comprehensive statistical analysis of inference and routing costs.\n\n- **Marginal Cost Ratio:** Our empirical data confirms that the \"routing cost\" (time spent on rejections) is minimal compared to the \"inference cost\" (time spent generating the solution) (**Table 7**). Even in the worst-case path (traversing the entire chain to the final agent), the cumulative time spent on rejections constitutes **less than 5%** of the time required for the final answer generation. This is structurally inherent: a rejection is a short sequence (e.g., \"I don't know!\"), whereas a valid answer is typically a long CoT sequence.\n- **Latency Analysis:** We further benchmarked System Latency (Time to First Token) in **Table R1**. While DiSRouter is slightly slower than lightweight predictive routers (e.g., RouteLLM), it is **significantly faster** than generative verification baselines (e.g., FrugalGPT, AutoMix), confirming its practical efficiency.\n    \n    *Table R1: Comparison of Time to First Token (s) between routing systems.*\n    \n    |  | RouterLLM | FrugalGPT | Automix | FORC | GraphRouter | DiSRouter |\n    | --- | --- | --- | --- | --- | --- | --- |\n    | Average | 0.048 | 2.620 | 3.475 | 0.070 | 0.024 | 0.081 |\n    | Worst Case | 0.063 | 9.565 | 11.34 | 0.076 | 0.028 | 0.241 |\n\n### **2. Sensitivity Analysis: Impact on System Utility**\n\nTo directly address the reviewer’s concern, we re-evaluated our system using an Adjusted Utility Metric that explicitly subtracts the routing overhead cost:\n\n$$Utility_{adjusted} = Performance - \\alpha \\cdot (Cost_{inference} + Cost_{routing})$$\n\nThe comparison between the original and adjusted metrics is presented in **Table R2**.\n\nTable R2: Sensitivity Analysis of System Utility: A comparison between original metrics (inference cost only) and adjusted metrics (incorporating cumulative routing overhead, **marked with a \\***).\n\n| Scenario | Alpha | Accuracy | Cost | Cost* | Utility | Utility* |\n| --- | --- | --- | --- | --- | --- | --- |\n| Performance First | 0.2 | 0.81 | 0.396 | 0.413 | 0.731 | 0.727 |\n| Balance | 0.5 | 0.77 | 0.321 | 0.333 | 0.610 | 0.604 |\n| Cost First | 0.8 | 0.75 | 0.289 | 0.299 | 0.519 | 0.511 |\n- **Minimal Deviation:** The results demonstrate that including the routing cost has a minimal impact on the overall System Utility. Even in the sensitive \"Cost First\" scenario, the overall utility drops by only approximately **1.5%**.\n- **Conclusion:** This confirms that the cumulative latency, while physically present, does not significantly alter the cost-benefit landscape or invalidate the comparative advantages of DiSRouter."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764224779917,"tcdate":1764155954249,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7635/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission7635/Authors"],"forum":"KDcwXKr0NU","number":8,"license":"CC BY 4.0","cdate":1764155954249,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7635/-/Official_Comment"],"mdate":1764224779917,"domain":"ICLR.cc/2026/Conference","replyto":"6yw99h60N0","id":"lcrSfTaBex","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"TLDR":{"value":"We introduce DiSRouter, a distributed system where LLMs leverage \"self-awareness\" to route queries among themselves, outperforming conventional centralized routers."},"keywords":{"value":["query routing","model selection","distributed system","self-awareness of LLM"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"The proliferation of Large Language Models (LLMs) has created a diverse ecosystem of models with highly varying performance and costs, necessitating effective query routing to balance performance and expense. Current routing systems often rely on a centralized external router trained on a fixed set of LLMs, making them inflexible and prone to poor performance since the small router can not fully understand the knowledge boundaries of different LLMs. We introduce DiSRouter (Distributed Self-Router), a novel paradigm that shifts from centralized control to distributed routing. In DiSRouter, a query traverses a network of LLM agents, each independently deciding whether to answer or route to other agents based on its own self-awareness—its ability to judge its competence. This distributed design offers superior flexibility, scalability, and generalizability. To enable this, we propose a two-stage Self-Awareness Training pipeline that enhances each LLM's self-awareness. Extensive experiments demonstrate that DiSRouter significantly outperforms existing routing methods in utility across various scenarios, effectively distinguishes between easy and hard queries, and shows strong generalization to out-of-domain tasks. Our work validates that leveraging an LLM's intrinsic self-awareness is more effective than external assessment, paving the way for more modular and efficient multi-agent systems."},"_bibtex":{"value":"@inproceedings{\nzheng2026disrouter,\ntitle={Di{SR}outer: Distributed Self-Routing for {LLM} Selections},\nauthor={Hang Zheng and Hongshen Xu and Yongkai.lin and Shuai Fan and Lu Chen and Kai Yu},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=KDcwXKr0NU}\n}"},"title":{"value":"DiSRouter: Distributed Self-Routing for LLM Selections"},"pdf":{"value":"/pdf/0190f98d94890b700d03febdee678e22a2a5e380.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"zheng|disrouter_distributed_selfrouting_for_llm_selections"},"authorids":{"value":["~Hang_Zheng3","~Hongshen_Xu1","~Yongkai.lin1","~Shuai_Fan1","~Lu_Chen3","~Kai_Yu3"]},"authors":{"value":["Hang Zheng","Hongshen Xu","Yongkai.lin","Shuai Fan","Lu Chen","Kai Yu"]}},"version":2},{"content":{"comment":{"value":"## **Q2: Can DiSRouter's self-awareness training be applied to LLMs with different architectures (e.g., encoder-decoder models)? Please provide results with at least one non-Qwen model family to validate broad applicability.**\n\nAs detailed in our Response to `W1`, our method has proven effective across distinct model families (Gemma3, Phi4). We prioritized decoder-only architectures in our experiments simply because they currently dominate the high-performance LLM landscape, leaving few competitive encoder-decoder options for complex reasoning.\n\nCrucially, our Self-Awareness Training is a **data-driven alignment strategy** (SFT + RL) rather than an architecture-specific modification. Consequently, the methodology is architecture-agnostic and should be applicable to any model architecture—including encoder-decoder frameworks—provided the model possesses sufficient capacity to learn self-reflection from data.\n\n## **Q3: How does DiSRouter perform with large-scale agent pools (e.g., >10 agents)?**\n\nIn our current experimental setup, we utilized a pool of **general-purpose LLMs** (`Qwen2.5-Instruct` series). Since these models are homogeneous and possess broad capabilities across various domains, a concise cascade of 5 agents proved sufficient to effectively cover the difficulty spectrum of the test tasks without requiring a large-scale pool (>10 agents).\n\nHowever, we are actively researching **heterogeneous routing networks** for future work. In such scenarios, the pool would contain diverse, domain-specific expert models (e.g., **specialized math solvers or coding experts**). This setting would naturally benefit from a larger scale DiSRouter network, likely organized in more complex topologies such as **tree structures** rather than a simple linear cascade. As this paper serves as the foundational work validating the efficacy of the distributed self-routing paradigm, we have focused on the cascade structure to establish the core concept. We intend to explore large-scale, heterogeneous agent pools in subsequent research.\n\n## **Q4: What are the practical limitations (e.g., latency, fault tolerance) in deploying DiSRouter? Case studies or simulations with real-world constraints would be valuable.**\n\nWe appreciate this practical perspective, which was echoed by other reviewers. To address concerns regarding latency and overhead, we have added a detailed statistical analysis of actual inference and routing costs in **Appendix A.1** of the revised paper.\n\n### **Latency Analysis**\n\n1. **Negligible Routing Overhead:** Our empirical data (**Table 6**) shows that although a query may be rejected by multiple agents before being solved, the cumulative **routing cost** (time spent generating rejections) remains negligible. Even in the worst-case scenario (routing to the final agent), this overhead constitutes less than **5%** of the actual inference cost of generating the final answer (**Table 7**). \n2. **Competitive Latency:** We also benchmarked **System Latency** (Time to First Token, **Table R6 and Table 8 in paper**). Results indicate that while DiSRouter incurs slightly higher latency than predictive routers (e.g., RouteLLM) which use lightweight classifiers, it is significantly faster than generative baselines that rely on full model outputs or verification steps (e.g., FrugalGPT, AutoMix).\n\n    *Table R6: Time to First Token (s) Results Comparison for Different Routing Systems.*\n\n    |  | RouterLLM | FrugalGPT | Automix | FORC | GraphRouter | DiSRouter |\n    | --- | --- | --- | --- | --- | --- | --- |\n    | Average | 0.048 | 2.620 | 3.475 | 0.070 | 0.024 | 0.081 |\n    | Worst Case | 0.063 | 9.565 | 11.34 | 0.076 | 0.028 | 0.241 |\n\n3. **Deployment Constraints:** We acknowledge that our current evaluation assumes efficient inter-agent communication. In **physically distributed deployments**, **network latency** becomes a non-negligible factor. Real-world implementation would require incorporating network transmission overhead into the global cost function to optimize for total system latency."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764154327090,"tcdate":1764154327090,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7635/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission7635/Authors"],"forum":"KDcwXKr0NU","number":4,"license":"CC BY 4.0","cdate":1764154327090,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7635/-/Official_Comment"],"mdate":1764154327090,"domain":"ICLR.cc/2026/Conference","replyto":"hD3voPCEXO","id":"VfNBXqGk56","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"TLDR":{"value":"We introduce DiSRouter, a distributed system where LLMs leverage \"self-awareness\" to route queries among themselves, outperforming conventional centralized routers."},"keywords":{"value":["query routing","model selection","distributed system","self-awareness of LLM"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"The proliferation of Large Language Models (LLMs) has created a diverse ecosystem of models with highly varying performance and costs, necessitating effective query routing to balance performance and expense. Current routing systems often rely on a centralized external router trained on a fixed set of LLMs, making them inflexible and prone to poor performance since the small router can not fully understand the knowledge boundaries of different LLMs. We introduce DiSRouter (Distributed Self-Router), a novel paradigm that shifts from centralized control to distributed routing. In DiSRouter, a query traverses a network of LLM agents, each independently deciding whether to answer or route to other agents based on its own self-awareness—its ability to judge its competence. This distributed design offers superior flexibility, scalability, and generalizability. To enable this, we propose a two-stage Self-Awareness Training pipeline that enhances each LLM's self-awareness. Extensive experiments demonstrate that DiSRouter significantly outperforms existing routing methods in utility across various scenarios, effectively distinguishes between easy and hard queries, and shows strong generalization to out-of-domain tasks. Our work validates that leveraging an LLM's intrinsic self-awareness is more effective than external assessment, paving the way for more modular and efficient multi-agent systems."},"_bibtex":{"value":"@inproceedings{\nzheng2026disrouter,\ntitle={Di{SR}outer: Distributed Self-Routing for {LLM} Selections},\nauthor={Hang Zheng and Hongshen Xu and Yongkai.lin and Shuai Fan and Lu Chen and Kai Yu},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=KDcwXKr0NU}\n}"},"title":{"value":"DiSRouter: Distributed Self-Routing for LLM Selections"},"pdf":{"value":"/pdf/0190f98d94890b700d03febdee678e22a2a5e380.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"zheng|disrouter_distributed_selfrouting_for_llm_selections"},"authorids":{"value":["~Hang_Zheng3","~Hongshen_Xu1","~Yongkai.lin1","~Shuai_Fan1","~Lu_Chen3","~Kai_Yu3"]},"authors":{"value":["Hang Zheng","Hongshen Xu","Yongkai.lin","Shuai Fan","Lu Chen","Kai Yu"]}},"version":2},{"content":{"summary":{"value":"This paper addresses the problem of selecting appropriate Large Language Models for multi-stage complex tasks to jointly optimize result quality while minimizing inference cost. The authors formalize this as a constraint-based optimization problem with the intention to guarantee lower bounds on accuracy and minimize LLM inference cost. They propose MixLLM, a training-free tree-based search framework that leverages exploration-exploitation principles with a two-phase greedy strategy to adaptively balance result quality and cost. The method includes automatic early termination through search-space pruning based on cost estimation and accuracy prediction."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"Q0: Answer the concerns raised in the Weaknesses section above.\nQ1: The candidate LLM pool in your experiments is limited to 5 models, yet you mention production environments may have 60+ candidate LLMs. Can you provide empirical analysis of: (a) wall-clock time for the search process across different pool sizes, (b) average number of examined nodes before termination, (c) how performance degrades with LLM pools of 10, 20, and 40+ models, and (d) what percentage of inference cost savings is consumed by exploration overhead?\nQ2: Can you provide: (a) theoretical guarantees on solution quality (e.g., approximation bounds showing the found solution is within X% of optimal), (b) convergence analysis or proof that the method terminates in finite time under all conditions, (c) worst-case complexity analysis for the number of examined nodes, and (d) conditions under which the method is guaranteed to find the global optimum?\nQ3: For practitioners deploying MixLLM, how should they determine: (a) the baseline accuracy $a$ when there is no clear \"best same-LLM\" combination or when different same-LLM combinations perform similarly, (b) the relaxation parameter $\\theta$, (c) thresholds for pruning nodes in the cost estimation and accuracy prediction mechanisms."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":2},"contribution":{"value":3},"strengths":{"value":"1. The problem formulation is technically sound and practical. The constraint-based single-objective optimization approach with the relaxation parameter $\\theta$ elegantly handles LLM stochasticity, which is empirically validated through multi-round experiments showing stable fluctuation ranges.\n2. The technical contributions are substantial. The two-phase greedy selection strategy that switches from accuracy prioritization to cost minimization once the accuracy constraint is satisfied is intuitive yet effective.\n3. Unlike competing methods (FrugalGPT, GraphRouter, HybridLLM, RouteLLM) that require training routers on task-specific data, MixLLM is training-free and automatically terminates when no promising nodes remain. This is a significant practical advantage, reducing deployment barriers and making the approach more generalizable across different domains."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"1. My major concern is, while the paper claims the method is efficient, there is insufficient analysis of computational overhead and scalability. The candidate LLM pool is limited to 5 models, and for a k-stage task, the combinatorial search space is |L|^k. It is not clear how the proposed approach can scale to thousands of LLMs. \n2. The accuracy prediction mechanism shows concerning limitations. For MAC-SQL(Spider), the prediction success rate is only 40%, attributed to marginal accuracy differences (≤2%) and limited empirical runtime data. This suggests the prediction mechanism may fail precisely when fine-grained distinctions matter most. The paper does not provide: (a) detailed description of the prediction algorithm beyond high-level concepts about \"runtime experience knowledge\" and \"node lineage,\" (b) ablation studies isolating the accuracy prediction component's contribution, (c) analysis of failure modes and when the predictor can be trusted, or (d) comparison with standard supervised learning baselines for accuracy prediction.\n3. The related work section omits several relevant areas that diminish the paper's positioning. Missing categories include  (a) Mixture-of-Experts (MoE) literature for routing between specialized models, which addresses similar selection problems, (b) LLM selection [1] and routing works [2], and more. \n4.  The paper lacks systematic analysis of when and why the method works. Key missing elements include: (a) characterization of task properties (e.g., stage dependencies, error propagation patterns, intermediate result quality distributions) that influence LLM selection effectiveness, (b) theoretical analysis or convergence guarantees for the search procedure. \n\n\n[1] RELM: https://openreview.net/pdf?id=gWi4ZcPQRl\n[2] Smoothie: https://openreview.net/pdf?id=pPSWHsgqRp"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762922269322,"tcdate":1760688046584,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission11091/Reviewer_Wf1K"],"signatures":["ICLR.cc/2026/Conference/Submission11091/Reviewer_Wf1K"],"forum":"jVjlpauPD5","number":1,"license":"CC BY 4.0","cdate":1760688046584,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission11091/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762922269322,"domain":"ICLR.cc/2026/Conference","replyto":"jVjlpauPD5","id":"cjzdRZjaaJ","forumContent":{"TLDR":{"value":"We study the problem of selecting LLMs for multi-stage complex tasks to preserve high-quality results with minimum LLM inference cost."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["LLM selection","multi-stage tasks","high-quality results","minimum inference cost","quality-cost trade-off"]},"primary_area":{"value":"other topics in machine learning (i.e., none of the above)"},"abstract":{"value":"We study the problem of selecting LLMs for multi-stage complex tasks to jointly\noptimize final result quality and minimize LLM inference cost. Existing approaches primarily target simple tasks or optimize for improving result quality\nor reducing cost only, overlooking the trade-off between them. We address this\ngap by systematically investigating LLM performance in multi-stage workflows.\nInspired by our findings from real-world applications, we formalize the LLM\nselection problem as a constraint-based optimization task with good properties:\nguarantee lower bounds on accuracy, minimize LLM inference cost and tolerate\nperformance fluctuations caused by LLM stochasticity, making it more practical\nfor users. We further introduce MixLLM, a search framework that leverages the\nexploration–exploitation principle to adaptively balance result quality and LLM\ninference cost. MixLLM is carefully designed to efficiently identify a (near-)optimal solution with minimal exploration and to terminate automatically and\nearly via search-space pruning. Experimental results demonstrate that, compared\nto using a single powerful commercial or open-source LLM, or selecting LLMs\nvia existing state-of-the-art methods, our approach not only improves result quality (by 1% – 16%) but also significantly reduces inference cost (by 18% – 92%). In addition, our approach efficiently adapts to different tasks, methods, and datasets,\ndemonstrating its practicality and robustness for multi-stage complex tasks."},"_bibtex":{"value":"@misc{\nding2026mixllm,\ntitle={Mix{LLM}: Selecting Large Language Models with High-Quality Results and Minimum Inference Cost for Multi-Stage Complex Tasks},\nauthor={Mengsu Ding and Rong Zhu and Bolin Ding and Jingren Zhou},\nyear={2026},\nurl={https://openreview.net/forum?id=jVjlpauPD5}\n}"},"title":{"value":"MixLLM: Selecting Large Language Models with High-Quality Results and Minimum Inference Cost for Multi-Stage Complex Tasks"},"pdf":{"value":"/pdf/0ae5042abe7f52c2dcdc6ec0992d7eece6849a12.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"ding|mixllm_selecting_large_language_models_with_highquality_results_and_minimum_inference_cost_for_multistage_complex_tasks"},"authorids":{"value":["~Mengsu_Ding1","~Rong_Zhu2","~Bolin_Ding3","~Jingren_Zhou1"]},"authors":{"value":["Mengsu Ding","Rong Zhu","Bolin Ding","Jingren Zhou"]}},"version":2},{"content":{"venue":{"value":"ICLR 2027 Conference Submission"},"keywords":{"value":["Local language models","LLM cascades","hidden-state probing","selective prediction","adaptive computation"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Running a language model locally offers advantages in privacy, latency, and cost, but local hardware fits only small models, which are less capable than frontier models. The usual remedy for a hard query, escalating it to a cloud model, gives up the privacy and cost advantages of running locally. A deployment that stays local faces two decisions for hard queries instead. First, it can spend more computation on a query, e.g., reasoning before answering, which raises accuracy at a cost in latency, so it must decide which queries are worth the extra computation (efficiency). Second, some queries are beyond the local model, and delivering a wrong answer is worse than deferring the query to a human in the loop, so it must decide which answers are safe to deliver (safety). We show that both decisions can be made from the model's own hidden states. The prefill state, computed before any token is generated, predicts whether the model will answer correctly, and the answer state, at the end of the generated answer, predicts whether that answer is correct. HARISSA fine-tunes the model so that both states predict correctness, then makes both decisions with one policy that cascades through the ways of answering from cheapest to most expensive, skipping a way the prefill state predicts will fail and deferring the query when the answer it stops with is predicted wrong. On a device running a single model, HARISSA is within one accuracy point of chain-of-thought at 2.7 times lower latency. On a server holding four sizes of one model, HARISSA is more accurate than the FrugalGPT and Self-REF cascades at the same latency, and at the same deferral rate the answer state leaves fewer wrong answers than the standard confidence signals in five of six task and setting pairs."},"_bibtex":{"value":"@inproceedings{\nanonymous2026harissa,\ntitle={{HARISSA}: Inference-Time Self-Checks for Efficient and Safe Local Language Model Deployment},\nauthor={Anonymous},\nbooktitle={Submitted to The Fifteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=0MfTMeelT0},\nnote={under review}\n}"},"title":{"value":"HARISSA: Inference-Time Self-Checks for Efficient and Safe Local Language Model Deployment"},"pdf":{"value":"/pdf/605516af16de856175f1a1b765016bb21d4fd542.pdf"},"venueid":{"value":"ICLR.cc/2027/Conference/Submission"}},"tmdate":1791231656006,"tcdate":1789663053103,"writers":["ICLR.cc/2027/Conference","ICLR.cc/2027/Conference/Submission33191/Authors"],"signatures":["ICLR.cc/2027/Conference/Submission33191/Authors"],"forum":"0MfTMeelT0","license":"CC BY 4.0","number":33191,"cdate":1789663053103,"readers":["everyone"],"invitations":["ICLR.cc/2027/Conference/-/Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Bidding","ICLR.cc/2027/Conference/Submission33191/-/Full_Submission","ICLR.cc/2027/Conference/-/Submission_Change_Before_Reviewing","ICLR.cc/2027/Conference/-/Edit"],"mdate":1791231656006,"odate":1791031328286,"domain":"ICLR.cc/2027/Conference","id":"0MfTMeelT0","version":2},{"content":{"comment":{"value":">**W1:** The framework introduces significant complexity. The experiments are limited to ≤15 agents. \n\n**A1:** Although Agora introduces an expressive economic structure, its runtime complexity remains modest because the system enforces **strict sparsity** rather than activating all agents.\n\n* **Theoretical Convergence (No Oscillation):** As proven in **Proposition 3.3.1** (Appendix C), the trading protocol performs a deterministic greedy descent on the global cost. Since the total cost is bounded, the trading sequence is **mathematically guaranteed to terminate** in finite steps, preventing infinite loops or high-frequency trading risks.\n\n* **Sparsity via Economic Rationality:** The profitability constraint (Eq. 5, $\\Delta \\mathcal{C} < 0$) acts as a strict gatekeeper. In a hypothetical pool of 100 agents, the vast majority will lack a \"comparative advantage\" for a specific query and will be automatically filtered out. The system maintains a small **active set** of experts regardless of the total pool size.\n\n* **Empirical Evidence:** Our experiments in **Appendix D** confirm this self-regulation: tripling the pool size ($N=5 \\to 15$) only marginally increased trade frequency ($0.8 \\to 1.2$ trades/task), demonstrating that communication overhead hits a saturation point and does not scale linearly with $N$.\n\n* **Negligible Overhead:** The Broker's selection complexity is linear $O(N)$ and computationally trivial compared to the VLM inference costs (TFLOPs) it saves.\n\n\n---\n\n\n>**W2:** The paper overlooks key transaction costs and may not be viable for real-time applications.\n\n**A2:** The coordination cost in Agora is intentionally lightweight and negligible relative to VLM inference. The Broker evaluation and the ( $\\Delta\\mathcal{C}$ ) profitability checks consist of lightweight arithmetic operations that run in ($O(N)$) time over the naturally sparse active set enforced by Eq. 5. Communication is similarly minimal, as only small uncertainty vectors are exchanged, and the system performs on average just **0.8–1.2 trades per task** (Table 4).\n\nImportantly, the **“Average Time ($s$)”** in Fig. 4 and the runtime measurements in Appendix I (8.7–22.8s) already include all coordination steps end-to-end. These results show that Agora matches or outperforms router baselines in wall-clock time while achieving higher accuracy, confirming that coordination overhead does not hinder real-time viability.\n\n---\n\n\n>**W3:** While numerous baselines are included, the adaptation of language-model-focused routers (like FrugalGPT, RouteLLM) to the multi-modal, multi-agent setting is only briefly mentioned in the appendix.\n\n**A3:** We appreciate this opportunity to clarify. In the revised paper, we have explicitly detailed the multimodal adaptation in Section 4.1. Specifically:\n\n1.  **Unified Multimodal Wrapper:** As text-oriented routers (FrugalGPT, RouteLLM, etc.) cannot natively process images, we implemented a unified wrapper (detailed in Appendix K) that encodes visual inputs using the visual encoder of the strongest available VLM in the pool. This visual embedding is concatenated with the textual query, enabling these routers to apply their original selection logic to multimodal contexts without structural modification.\n2.  **Rigorous Fairness:** All baselines—including these adapted routers—operate on the **identical heterogeneous agent pool ($N=6$)** as Agora, using consistent prompts and decoding settings. This ensures that any performance gap arises purely from the coordination strategy, not from the underlying model capabilities.\n\nTo address the concern regarding cross-dataset validation, we have extended our head-to-head comparison beyond MMBench. **Table R2** below demonstrates that Agora consistently outperforms the strongest aggregation (MoA) and routing (KABB-VLM) baselines across reasoning-heavy (MathVision) and perception-heavy (InfoVQA, CC-OCR) benchmarks. Notably, while baselines struggle to significantly surpass the best single expert in perception tasks due to hallucination risks, Agora’s uncertainty trading mechanism effectively mitigates this, securing clear gains.\n\n---\n\n\n**Table R2: Head-to-Head Strategy Comparison on Additional Benchmarks (Accuracy, %)**\n*Comparison using the exact same heterogeneous agent pool ($N=6$) and protocol as Table 1.*\n\n| Dataset | MoA | KABB-VLM | **Agora (Ours)** |\n| :--- | :---: | :---: | :---: |\n| **MathVision** | 42.1 | 42.5 | **44.3** |\n| **InfoVQA** | 87.6 | 87.8 | **88.9** |\n| **CC-OCR** | 80.2 | 80.4 | **81.2** |"},"title":{"value":"Part I"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763926818347,"tcdate":1763568083435,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission582/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission582/Authors"],"forum":"zeqCjGQB4U","number":1,"license":"CC BY 4.0","cdate":1763568083435,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission582/-/Official_Comment"],"mdate":1763926818347,"domain":"ICLR.cc/2026/Conference","replyto":"aj5Eea9564","id":"6DxdiTQcro","forumContent":{"TLDR":{"value":"We presents a framework that reframes coordination as a decentralized market for uncertainty."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["agent; Vision Language Model; Uncernity"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Vision-Language Models (VLMs) enable powerful multi-agent systems, but scaling them is economically unsustainable: coordinating heterogeneous agents under information asymmetry often spirals costs. Existing paradigms, such as Mixture-of-Agents and knowledge-based routers, rely on heuristic proxies that ignore costs and collapse uncertainty structure, leading to provably suboptimal coordination.\nWe introduce Agora, a framework that reframes coordination as a decentralized market for uncertainty. Agora formalizes epistemic uncertainty into a structured, tradable asset (perceptual, semantic, inferential), and enforces profitability-driven trading among agents based on rational economic rules. A market-aware broker, extending Thompson Sampling, initiates collaboration and guides the system toward cost-efficient equilibria. Experiments on five multimodal benchmarks (MMMU, MMBench, MathVision, InfoVQA, CC-OCR) show that Agora outperforms strong VLMs and heuristic multi-agent strategies, e.g., achieving +8.5% accuracy over the best baseline on MMMU while reducing cost by over 3×. These results establish market-based coordination as a principled and scalable paradigm for building economically viable multi-agent visual intelligence systems."},"_bibtex":{"value":"@inproceedings{\nzhang2026why,\ntitle={Why Keep Your Doubts to Yourself? Trading Visual Uncertainties among Vision-Language Models},\nauthor={Jusheng Zhang and Yijia Fan and Kaitong Cai and Jing Yang and Jiawei Yao and Jian Wang and Guanlong Qu and Ziliang Chen and Keze Wang},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=zeqCjGQB4U}\n}"},"title":{"value":"Why Keep Your Doubts to Yourself? Trading Visual Uncertainties among Vision-Language Models"},"pdf":{"value":"/pdf/03f4bead771e441ade14013c814de209cd0630ea.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"zhang|why_keep_your_doubts_to_yourself_trading_visual_uncertainties_among_visionlanguage_models"},"authorids":{"value":["~Jusheng_Zhang3","~Yijia_Fan1","~Kaitong_Cai1","~Jing_Yang39","~Jiawei_Yao3","~Jian_Wang10","~Guanlong_Qu2","~Ziliang_Chen1","~Keze_Wang1"]},"authors":{"value":["Jusheng Zhang","Yijia Fan","Kaitong Cai","Jing Yang","Jiawei Yao","Jian Wang","Guanlong Qu","Ziliang Chen","Keze Wang"]}},"version":2},{"content":{"comment":{"value":"**Table1: Comparison between different methods under two simulation strategies in the large-scale experimental setting. The Deltas represent the improvement of the Reward over the best baseline. The Reduction indicates the relative reduction in time cost compared to the baseline with the highest time cost.** \n\n\n| Scenario    | multi-cost-efficiency simulation             | LLM-as-a-Judge strategy                |\n|-------------|----------------------------------------------|----------------------------------------|\n| Method      | Reward &nbsp;  Delta (%) &nbsp;  Time   &nbsp;  Reduction (%)   | Accuracy &nbsp;  Delta (%) &nbsp;  Time   &nbsp; Reduction (%) |\n| HybridLLM   | 0.119  &emsp;  -43.33    &ensp;&emsp;  4:32   &emsp;&emsp;  98.50  &nbsp;         | 0.196    &emsp;&emsp;&emsp;  0.00      &emsp;&emsp;  4:56   &emsp;&nbsp; 98.40         |\n| FrugalGPT   | 0.141  &emsp;  -32.86    &ensp;&emsp;  47:44  &emsp;&emsp;  84.29  &nbsp;         | 0.108    &emsp;&emsp; -44.90     &emsp;&emsp;  49:31  &ensp;&nbsp; 83.92         |\n| GraphRouter | 0.204  &emsp;  -2.86     &ensp;&emsp;&ensp;  8:26   &emsp;&emsp;  97.22  &nbsp;         | 0.137    &emsp;&emsp; -30.10     &emsp;&emsp;  9:28   &emsp;&ensp; 96.93         |\n|RouterDC | 0.210  &nbsp;&emsp; 0.00     &ensp;&emsp;  303:47 &emsp;&emsp;  0.00   &nbsp;         | 0.153    &emsp;&emsp; -21.94     &emsp;&emsp;  308:02 &ensp;&nbsp; 0.00          |\n| Ours        | 0.244  &emsp;   16.19    &ensp;&emsp; 10:15   &emsp;&emsp; 96.63   &nbsp;         | 0.313    &emsp;&emsp;&nbsp;  59.69     &emsp;&emsp;  11:37  &emsp;&nbsp; 96.23         |\n| Oracle      | 0.310  &emsp;   47.62    &ensp;&emsp;  /      &emsp;&emsp;  /      &nbsp;         | 1.000    &emsp;&emsp; 410.20     &emsp;&emsp;  /      &ensp;&nbsp; /             |\n\nMoreover, following the constructive suggestion, we additionally collected a small-scale real-user dataset consisting of 10 LLMs, 40 users, and 80 queries. The experimental results are shown in Table 2. These experiments confirm that PersonalizedRouter generalizes beyond simulated settings and is also effective in real-user scenarios.\n\n**Table2:Comparison between different methods under small-scale human interaction dataset. The Improvement is measured relative to the best baseline.** \n\n| Scenario    |       LLM-as-a-Judge strategy      |\n|-------------|------------------------------------|\n| Method      |       Accuracy &nbsp;  Improvement (%)   |\n| HybridLLM   |       0.315  &emsp;&emsp;&ensp;   -23.73             |\n| FrugalGPT   |       0.350  &emsp;&emsp;&ensp;   -15.25             |\n| GraphRouter |       0.357  &emsp;&emsp;&ensp;   -13.56             |\n| RouterDC    |       0.413  &emsp;&emsp;&ensp;&nbsp;    0.00              |\n| PersonalizedRouter| 0.438  &emsp;&emsp;&ensp;&nbsp;    6.05              |\n| Oracle      |       1.000  &emsp;&emsp;&ensp;   142.13             |\n\n**[why the accuracy of a math dataset can be improved by adding user preference]:**  We would like to clarify that our results do not reflect an improvement in the accuracy of math datasets through the addition of user preferences. Instead, under the LLM-as-a-Judge strategy, the judge selects the response that best aligns with its stylistic preference as the best answer (Section 4.2). Accordingly, our contribution lies in improving the router’s ability to select the best answer judged by the LLM judge, rather than increasing task-level accuracy on the math dataset.     \n\n**[And the author didn't provide comparison with GraphRouter or even SOTA baselines on those benchmarks.]:** In the latest revision, we have incorporated GraphRouter into all experiments and conducted comparisons with other baselines. The experimental results are reported in Tables 1, 2, 3, 4, and 5. Our PersonalizedRouter still outperforms all other baselines.\n\n**Q4.Fairness & bias: By simulating “user personas” with LLM prompts, the method risks encoding and amplifying stereotypes (e.g., engineer vs. literature enthusiast).** \n\n**Response:**  We thank the reviewer for raising the above concerns. Our work primarily focuses on demonstrating that PersonalizedRouter can effectively learn user preferences from interaction data and achieve personalized LLM selection. The issue of whether simulating users with LLMs may introduce stereotypes is indeed an important open question, but it is not the central focus of our work. As clarified in response to Q3, LLM simulation has been adopted in prior work and provides meaningful value for validating whether a router can capture latent user preferences. Furthermore, to alleviate concerns about potential bias from relying on a single LLM to simulate users, we conducted additional LLM-judge experiments using three LLMs, each with two distinct instruction prompts. The experimental results are presented in Table 1."},"title":{"value":"Response to Reviewer 2sp7 (Part 2)"}},"parentInvitations":"TMLR/-/Official_Comment","tmdate":1758313481898,"tcdate":1758310752017,"writers":["TMLR","TMLR/Paper5500/Authors"],"signatures":["TMLR/Paper5500/Authors"],"forum":"W80eE3ArAl","number":3,"license":"CC BY 4.0","cdate":1758310752017,"readers":["everyone"],"invitations":["TMLR/Paper5500/-/Official_Comment"],"mdate":1758313481898,"domain":"TMLR","replyto":"fIBO7UiOvL","id":"Ki0b9S4Qmn","forumContent":{"submission_length":{"value":"Regular submission (no more than 12 pages of main content)"},"venue":{"value":"Accepted by TMLR"},"code":{"value":"https://github.com/ulab-uiuc/PersonalizedRouter"},"supplementary_material":{"value":"/attachment/e4bbeecf2accf23bec60f721de3508b2cf488096.zip"},"abstract":{"value":"The growing number of Large Language Models (LLMs) with diverse capabilities and response styles provides users with a wider range of choices, which presents challenges in selecting appropriate LLMs, as user preferences vary in terms of performance, cost, and response style. Current LLM selection methods typically optimize for a single fixed objective, such as performance, cost, or a trade-off between them, and fail to learn individual user preferences from interaction data. To address these limitations, we propose PersonalizedRouter, a graph-based framework that models diverse user profiles and performs personalized LLM selection by leveraging interaction data that includes task context, queries, candidate LLMs, and user decisions. To capture contextual information between user queries and optimal LLMs, PersonalizedRouter converts the interaction data into a heterogeneous graph, where the relationships between different types of nodes are represented by edges. To evaluate adaptability across users, we design two strategies: the multi-cost-efficiency simulation strategy and the LLM-as-a-Judge strategy. In addition, we construct PersonaRoute-Bench, a large-scale benchmark with 1,000 simulated users and 10 LLMs. Experimental results show that PersonalizedRouter significantly outperforms existing LLM selection methods and surpasses the strongest methods by a large margin of 15.38% and 9.83% under two simulation strategies. On the PersonaRoute-Bench with 1,000 users, it further surpasses the best methods by 16.19% and 59.69% while maintaining higher efficiency. Moreover, PersonalizedRouter demonstrates strong few-shot generalization, achieving 64.81% and 85.80% of the fully trained model’s performance when adapting to new users and new LLMs."},"_bibtex":{"value":"@article{\ndai2025personalizedrouter,\ntitle={PersonalizedRouter: Personalized {LLM} Routing via Graph-based User Preference Modeling},\nauthor={Zhongjie Dai and Tao Feng and Jiaxuan You},\njournal={Transactions on Machine Learning Research},\nissn={2835-8856},\nyear={2025},\nurl={https://openreview.net/forum?id=W80eE3ArAl},\nnote={}\n}"},"title":{"value":"PersonalizedRouter: Personalized LLM Routing via Graph-based User Preference Modeling"},"pdf":{"value":"/pdf/1c7d7c4a7a37c3cd71672b102988c3788f67c8e7.pdf"},"venueid":{"value":"TMLR"},"paperhash":{"value":"dai|personalizedrouter_personalized_llm_routing_via_graphbased_user_preference_modeling"},"authorids":{"value":["~Zhongjie_Dai1","~Tao_Feng5","~Jiaxuan_You2"]},"assigned_action_editor":{"value":"~Yen-Chang_Hsu1"},"authors":{"value":["Zhongjie Dai","Tao Feng","Jiaxuan You"]}},"version":2},{"content":{"comment":{"value":"**Table1: Comparison between different methods under two simulation strategies in the large-scale experimental setting. The Deltas represent the improvement of the Reward over the best baseline. The Reduction indicates the relative reduction in time cost compared to the baseline with the highest time cost.**  \n\n| Scenario    | multi-cost-efficiency simulation             | LLM-as-a-Judge strategy                |\n|-------------|----------------------------------------------|----------------------------------------|\n| Method      | Reward &nbsp;  Delta (%) &nbsp;  Time   &nbsp;  Reduction (%)   | Accuracy &nbsp;  Delta (%) &nbsp;  Time   &nbsp; Reduction (%) |\n| HybridLLM   | 0.119  &emsp;  -43.33    &ensp;&emsp;  4:32   &emsp;&emsp;  98.50  &nbsp;         | 0.196    &emsp;&emsp;&emsp;  0.00      &emsp;&emsp;  4:56   &emsp;&nbsp; 98.40         |\n| FrugalGPT   | 0.141  &emsp;  -32.86    &ensp;&emsp;  47:44  &emsp;&emsp;  84.29  &nbsp;         | 0.108    &emsp;&emsp; -44.90     &emsp;&emsp;  49:31  &ensp;&nbsp; 83.92         |\n| GraphRouter | 0.204  &emsp;  -2.86     &ensp;&emsp;&ensp;  8:26   &emsp;&emsp;  97.22  &nbsp;         | 0.137    &emsp;&emsp; -30.10     &emsp;&emsp;  9:28   &emsp;&ensp; 96.93         |\n|RouterDC | 0.210  &nbsp;&emsp; 0.00     &ensp;&emsp;  303:47 &emsp;&emsp;  0.00   &nbsp;         | 0.153    &emsp;&emsp; -21.94     &emsp;&emsp;  308:02 &ensp;&nbsp; 0.00          |\n| Ours        | 0.244  &emsp;   16.19    &ensp;&emsp; 10:15   &emsp;&emsp; 96.63   &nbsp;         | 0.313    &emsp;&emsp;&nbsp;  59.69     &emsp;&emsp;  11:37  &emsp;&nbsp; 96.23         |\n| Oracle      | 0.310  &emsp;   47.62    &ensp;&emsp;  /      &emsp;&emsp;  /      &nbsp;         | 1.000    &emsp;&emsp; 410.20     &emsp;&emsp;  /      &ensp;&nbsp; /             |\n\n**Q4. a.The models are all \"mid tier\" models without any frontier LLMs from e.g. anthropic, openai, qwen or deepseek. This means that the range of performance is not as wide as it could be making assignment easier than what it should be. The LLMs are also all generalist models, rather than having e.g. experts in different domains (e.g. via low-rank adaptation) that might give more depth to the routing (e.g. use a small, cheap domain specific model rather than a generalist, but expensive frontier LLM)** \n\n**b. Increase the heterogeneity of LLMs (include frontier models, domain-expert models, etc...)**\n\n**Response:**  Thank you for your careful review. In the new large-scale experiments, we updated the set of candidate models. For frontier models, we introduced Gemini 2.5 Flash, GPT-4o-mini, and Qwen3 Coder 480B A35B. For domain-specific expert models, we added Qwen3 Coder 480B A35B, palmyra-fin, and palmyra-med. As a result, our new experiment covers 10 LLMs: Mixtral-8x7B, Gemini 2.5 Flash, LLaMA-3.1 (8B), Qwen3 Coder 480B A35B, Mistral Small 3.2 24B, palmyra-fin, palmyra-med, Qwen-2.5 7B, Gemma-3 12B, and GPT-4o-mini. Information about the candidate LLMs is provided in Appendix A.6 of the latest revision.\n\n**Q5. Make connections with GraphRouter clear, also include in benchmarks.** \n\n**Response:**  Thanks for your constructive feedback. As clarified in response to Q1, PersonalizedRouter learns latent user preferences from interaction data, enabling personalized LLM selection across multiple users. In contrast, GraphRouter is restricted to fixed-user routing. Furthermore, PersonalizedRouter augments the graph with user nodes, and edges encode different types of preferences (e.g., performance-oriented, style-oriented).\n\nIn our large-scale experiments under both the multi-cost-efficiency simulation strategy and the LLM-as-a-Judge strategy, we have included GraphRouter as part of the benchmark for comparison. The experimental results of large-scale experiments are shown in Table 1. We also added GraphRouter experiments under other settings, with detailed results reported in Tables 2,3,4, and 5 below. Our PersonalizedRouter still outperforms all other baselines.\n\n**Table2:Comparison between different methods under small-scale human interaction dataset. The Improvement is measured relative to the best baseline.** \n\n| Scenario    |       LLM-as-a-Judge strategy      |\n|-------------|------------------------------------|\n| Method      |       Accuracy &nbsp;  Improvement (%)   |\n| HybridLLM   |       0.315  &emsp;&emsp;&ensp;   -23.73             |\n| FrugalGPT   |       0.350  &emsp;&emsp;&ensp;   -15.25             |\n| GraphRouter |       0.357  &emsp;&emsp;&ensp;   -13.56             |\n| RouterDC    |       0.413  &emsp;&emsp;&ensp;&nbsp;    0.00              |\n| PersonalizedRouter| 0.438  &emsp;&emsp;&ensp;&nbsp;    6.05              |\n| Oracle      |       1.000  &emsp;&emsp;&ensp;   142.13             |"},"title":{"value":"Response to Reviewer h7mj (Part 2)"}},"parentInvitations":"TMLR/-/Official_Comment","tmdate":1758313312013,"tcdate":1758312529336,"writers":["TMLR","TMLR/Paper5500/Authors"],"signatures":["TMLR/Paper5500/Authors"],"forum":"W80eE3ArAl","number":8,"license":"CC BY 4.0","cdate":1758312529336,"readers":["everyone"],"invitations":["TMLR/Paper5500/-/Official_Comment"],"mdate":1758313312013,"domain":"TMLR","replyto":"WfA7D74GQL","id":"TJXP0lJyw7","forumContent":{"submission_length":{"value":"Regular submission (no more than 12 pages of main content)"},"venue":{"value":"Accepted by TMLR"},"code":{"value":"https://github.com/ulab-uiuc/PersonalizedRouter"},"supplementary_material":{"value":"/attachment/e4bbeecf2accf23bec60f721de3508b2cf488096.zip"},"abstract":{"value":"The growing number of Large Language Models (LLMs) with diverse capabilities and response styles provides users with a wider range of choices, which presents challenges in selecting appropriate LLMs, as user preferences vary in terms of performance, cost, and response style. Current LLM selection methods typically optimize for a single fixed objective, such as performance, cost, or a trade-off between them, and fail to learn individual user preferences from interaction data. To address these limitations, we propose PersonalizedRouter, a graph-based framework that models diverse user profiles and performs personalized LLM selection by leveraging interaction data that includes task context, queries, candidate LLMs, and user decisions. To capture contextual information between user queries and optimal LLMs, PersonalizedRouter converts the interaction data into a heterogeneous graph, where the relationships between different types of nodes are represented by edges. To evaluate adaptability across users, we design two strategies: the multi-cost-efficiency simulation strategy and the LLM-as-a-Judge strategy. In addition, we construct PersonaRoute-Bench, a large-scale benchmark with 1,000 simulated users and 10 LLMs. Experimental results show that PersonalizedRouter significantly outperforms existing LLM selection methods and surpasses the strongest methods by a large margin of 15.38% and 9.83% under two simulation strategies. On the PersonaRoute-Bench with 1,000 users, it further surpasses the best methods by 16.19% and 59.69% while maintaining higher efficiency. Moreover, PersonalizedRouter demonstrates strong few-shot generalization, achieving 64.81% and 85.80% of the fully trained model’s performance when adapting to new users and new LLMs."},"_bibtex":{"value":"@article{\ndai2025personalizedrouter,\ntitle={PersonalizedRouter: Personalized {LLM} Routing via Graph-based User Preference Modeling},\nauthor={Zhongjie Dai and Tao Feng and Jiaxuan You},\njournal={Transactions on Machine Learning Research},\nissn={2835-8856},\nyear={2025},\nurl={https://openreview.net/forum?id=W80eE3ArAl},\nnote={}\n}"},"title":{"value":"PersonalizedRouter: Personalized LLM Routing via Graph-based User Preference Modeling"},"pdf":{"value":"/pdf/1c7d7c4a7a37c3cd71672b102988c3788f67c8e7.pdf"},"venueid":{"value":"TMLR"},"paperhash":{"value":"dai|personalizedrouter_personalized_llm_routing_via_graphbased_user_preference_modeling"},"authorids":{"value":["~Zhongjie_Dai1","~Tao_Feng5","~Jiaxuan_You2"]},"assigned_action_editor":{"value":"~Yen-Chang_Hsu1"},"authors":{"value":["Zhongjie Dai","Tao Feng","Jiaxuan You"]}},"version":2},{"content":{"comment":{"value":"Thank you for your valuable feedback. With the cost-penalized formulation, tuning $\\lambda$ can further adjust the cost-accuracy tradeoff. For example, with regards to the reviewer's example, for a different parameter setting ($\\lambda=100, \\text{budget}= 3$; previous figures were for fixed $\\lambda=10$), the accuracy is 0.918 and the cost is 1.454. This is comparable to TREACLE without penalty and \\$1.5 budget, which has the same accuracy of 0.918 and cost 1.471. In other words, with a larger budget and by adding the cost penalty, we can more finely control the actual cost.\n\nOverall, many papers use the budget-constrained setting [1,2,3,4]. We explored the cost penalty based on reviewer feedback and found that the TREACLE framework flexibly extended to such a formulation. The results suggest that the cost penalty formulation can perform well with some parameter tuning, which is a general disadvantage of cost-penalty formulations. The advantage of our original formulation is that it does not require parameter tuning, only a simple total budget setting, which easier for practitioners. We would like to add the cost-penalty results as a subsection to the paper further exploring this alternative formulation and different settings of $\\lambda$.\n\n[1] Bai, Fan, Alan Ritter, and Wei Xu. \"Pre-train or Annotate? Domain Adaptation with a Constrained Budget.\" Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.\n\n[2] Hoffmann, Jordan, et al. \"Training compute-optimal large language models.\" Proceedings of the 36th International Conference on Neural Information Processing Systems. 2022.\n\n[3] Chen, Lingjiao, Matei Zaharia, and James Zou. \"Frugalgpt: How to use large language models while reducing cost and improving performance.\" arXiv preprint arXiv:2305.05176 (2023).\n\n[4] Shi, Chengshuai, et al. \"Best arm identification for prompt learning under a limited budget.\" arXiv preprint arXiv:2402.09723 (2024)."}},"tmdate":1730890145106,"tcdate":1723569443294,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission13319/Authors"],"signatures":["NeurIPS.cc/2024/Conference/Submission13319/Authors"],"forum":"aDQlAz09dS","number":10,"license":"CC BY 4.0","cdate":1723569443294,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission13319/-/Official_Comment","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730890145106,"domain":"NeurIPS.cc/2024/Conference","replyto":"X6I7V7T8e3","id":"VbD6LmZcKH","forumContent":{"TLDR":{"value":"Under a long-term budget constraint, answer reasoning questions by picking the best combinations of LLMs and prompts."},"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["LLMs; Budget constraint; Reinforcement learning"]},"supplementary_material":{"value":"/attachment/6e095dbe7ca2a33f1877c0c69b3ddface0d25759.zip"},"primary_area":{"value":"infrastructure"},"abstract":{"value":"Recent successes in natural language processing have led to the proliferation of large language models (LLMs) by multiple providers. Each LLM offering has different inference accuracy, monetary cost, and latency, and their accuracy further depends on the exact wording of the question (i.e., the specific prompt). At the same time, users often have a limit on monetary budget and latency to answer all their questions, and they do not know which LLMs to choose for each question to meet their accuracy and long term budget requirements. To navigate this rich design space, we propose TREACLE (Thrifty Reasoning via Context-Aware LLM and Prompt Selection), a reinforcement learning policy that jointly selects the model and prompting scheme while respecting the user's monetary cost and latency constraints. TREACLE uses the problem context, including question text embeddings (reflecting the type or difficulty of a query) and the response history (reflecting the consistency of previous responses) to make smart decisions. Our evaluations on standard reasoning datasets (GSM8K, CSQA, and LLC) with various LLMs and prompts show that TREACLE enables cost savings of up to 85% compared to baselines, while maintaining high accuracy. Importantly, it provides the user with the ability to gracefully trade off accuracy for cost."},"_bibtex":{"value":"@inproceedings{\nzhang2024efficient,\ntitle={Efficient Contextual {LLM} Cascades through Budget-Constrained Policy Learning},\nauthor={Xuechen Zhang and Zijian Huang and Ege Onur Taga and Carlee Joe-Wong and Samet Oymak and Jiasi Chen},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=aDQlAz09dS}\n}"},"title":{"value":"Efficient Contextual LLM Cascades through Budget-Constrained Policy Learning"},"pdf":{"value":"/pdf/6edc8a474ffa7f439968f38dd2ced40f203ae8db.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"zhang|efficient_contextual_llm_cascades_through_budgetconstrained_policy_learning"},"authorids":{"value":["~Xuechen_Zhang2","~Zijian_Huang2","~Ege_Onur_Taga1","~Carlee_Joe-Wong1","~Samet_Oymak2","~Jiasi_Chen1"]},"authors":{"value":["Xuechen Zhang","Zijian Huang","Ege Onur Taga","Carlee Joe-Wong","Samet Oymak","Jiasi Chen"]}},"version":2},{"content":{"comment":{"value":"### Response to Point 6\nOur primary contribution in this work is the interpretability analysis framework and the characterization of orchestration dynamics, rather than proposing a new state-of-the-art orchestration algorithm optimized for benchmark performance. Our baseline choices were designed to contrast interpretable learned orchestration against representative coordination paradigms, particularly rigid workflow-based systems such as MetaGPT and FrugalGPT. Many of the suggested approaches (e.g., self-consistency, verifier aggregation, static top-k routing, cascade routing, or LLM-Blender-style aggregation) primarily focus on improving predictive performance or inference efficiency rather than on exposing analyzable interaction structure, sequencing behavior, or attribution dynamics, which are the core focus of this work. We agree that these are valuable directions for future empirical expansion. However, incorporating the full spectrum of orchestration and ensemble baselines would substantially broaden the scope beyond the interpretability-centered objectives of the current paper.\n### Response to Point 7\nINFORM is fundamentally an analysis setup, not a routing architecture. To demonstrate its utility, we applied it to a highly regularized, representative orchestrator. However, INFORM's interpretability principles are expected to generalize to fundamentally different regimes. As demonstrated in Appendix I, we successfully applied INFORM to a confidence-based, fixed-order cascade orchestrator (similar to FrugalGPT), revealing causal dependencies and routing misalignments without relying on our proposed training regularizers, oracle alignment, or interaction matrices. The attribution analysis itself does not improve performance. Rather, it acts as a diagnostic lens to expose hidden dependencies and brittle coordination that accuracy metrics obscure, as detailed in Section 4.5.\n\nThe performance gains achieved by our representative orchestrator are driven by distinct architectural and training components, which we isolate in our existing ablation studies. The adaptive interaction matrix and dynamic sequencing are the primary drivers of routing efficiency. Appendix F and Figure 14 demonstrate severe performance degradation when these are replaced with static graphs or fixed sequences. Figure 8 further disentangles this by comparing Relational-Only and Intrinsic-Only variants, confirming that both structural history and semantic scoring are required for optimal performance. The training regularizers, specifically the oracle alignment and adaptive top-k schedule detailed in Appendix C, do not dictate the final routing logic but act as optimization stabilizers. The oracle prevents long cold-start phases by bootstrapping initial protocols, while the adaptive top-k schedule structurally forces the transition from exploratory routing to focused execution. \n### Response to Point 8\nIn the revised manuscript, we extend the perturbation analysis beyond routing and sequence divergence by additionally reporting downstream behavioral effects in terms of task accuracy and token generation cost under perturbations. Figures 4 and 5 now show how structural perturbations affect both MMLU accuracy and token usage across training epochs. These complement the KL-divergence analyses by connecting orchestration instability to concrete inference outcomes. We agree that qualitative failure-case analysis would further enrich the discussion and consider this a valuable direction for future extensions\n### Response to Point 9\nOur intent is not to claim universal applicability across all orchestration paradigms, but rather to demonstrate that the proposed interpretability methodology can analyze differentiable orchestration policies that expose intermediate routing representations and selection dynamics. The discussion of cascade-style routing in the appendix is intended only as an illustration of potential extensibility. \n### Response to Point 10\nOur intention in this experiment is not to establish strict architectural parity between fundamentally different orchestration paradigms, but rather to contrast learned adaptive orchestration against rigid role-based coordination under a controlled expert pool. To improve clarity, we now explicitly document the prompts, role definitions, and coordination setup used for MetaGPT in Appendix N. The same underlying expert models are used across both systems to maintain consistency in expert capability.\n### Response to Suggestions\n- In the revised manuscript, we explicitly report the standalone performance of each individual expert in the ten-expert homogeneous consortium in Appendix L (Table 11). \n- We use the term “homogeneous” to denote parity in capacity. Although the consortium includes experts from different model families, they operate within a similar parameter scale (~8B parameters), which we treat as providing comparable representational capacity."},"title":{"value":"Response to Reviewer jtDm Comments -- Part II"}},"parentInvitations":"TMLR/-/Official_Comment","tmdate":1778662544962,"tcdate":1778662544962,"writers":["TMLR","TMLR/Paper7372/Authors"],"signatures":["TMLR/Paper7372/Authors"],"forum":"4W7sgat04A","number":6,"license":"CC BY 4.0","cdate":1778662544962,"readers":["everyone"],"invitations":["TMLR/Paper7372/-/Official_Comment"],"mdate":1778662544962,"domain":"TMLR","replyto":"SDl13VdSJp","id":"SiiYgIibek","forumContent":{"submission_length":{"value":"Long submission (more than 12 pages of main content)"},"venue":{"value":"Accepted by TMLR"},"code":{"value":"https://github.com/parmanu-lcs2/inform"},"supplementary_material":{"value":"/attachment/171f7e461162276df6370905a64ba727fa7d77db.zip"},"abstract":{"value":"Multi-expert systems, where multiple Large Language Models (LLMs) collaborate to solve complex tasks, are increasingly adopted for high-performance reasoning and generation. However, the orchestration policies governing expert interaction and sequencing remain largely opaque. We introduce INFORM, an interpretability analysis that treats orchestration as an explicit, analyzable computation, enabling the decoupling of expert interaction structure, execution order, and functional attribution. We use INFORM to evaluate an orchestrator on GSM8K, HumanEval, and MMLU using a homogeneous consortium of ten instruction-tuned experts drawn from LLaMA-3.1 8B, Qwen3 8B, and DeepSeek-R1 8B, with controlled decoding-temperature variation, and a secondary heterogeneous consortium spanning 1B-7B parameter models. Across tasks, routing dominance is a poor proxy for functional necessity. We reveal a divergence between relational importance, captured by routing mass and interaction topology, and intrinsic importance, measured via gradient sensitivity: frequently selected experts often act as interaction hubs with limited influence, while sparsely routed experts can be structurally critical. Orchestration behaviors emerge asynchronously, with expert centralization preceding stable routing confidence and expert ordering remaining non-deterministic. Targeted ablations show that masking intrinsically important experts induces disproportionate collapse in interaction structure compared to masking frequent peers, confirming that INFORM exposes functional and structural dependencies beyond accuracy metrics alone."},"_bibtex":{"value":"@article{\nghosh2026disentangling,\ntitle={Disentangling Intrinsic Importance from Emergent Structure in Multi-Expert Orchestration},\nauthor={Sudipto Ghosh and Sujoy Nath and Sunny Manchanda and Tanmoy Chakraborty},\njournal={Transactions on Machine Learning Research},\nissn={2835-8856},\nyear={2026},\nurl={https://openreview.net/forum?id=4W7sgat04A},\nnote={}\n}"},"video":{"value":"https://www.youtube.com/watch?v=f8TwBV-Xl-M"},"title":{"value":"Disentangling Intrinsic Importance from Emergent Structure in Multi-Expert Orchestration"},"changes_since_last_submission":{"value":"A previous version of this manuscript was desk-rejected on grounds of modified template. It was caused by a LaTeX package import. These issues have been fully corrected, and the manuscript now strictly adheres to the official TMLR template. The technical content remains unchanged."},"pdf":{"value":"/pdf/de4cbcdd723008c2af248976743fd6497561470e.pdf"},"venueid":{"value":"TMLR"},"paperhash":{"value":"ghosh|disentangling_intrinsic_importance_from_emergent_structure_in_multiexpert_orchestration"},"authorids":{"value":["~Sudipto_Ghosh4","~Sujoy_Nath1","~Sunny_Manchanda1","~Tanmoy_Chakraborty2"]},"previous_TMLR_submission_url":{"value":"https://openreview.net/forum?id=iKgVAXrVmP"},"assigned_action_editor":{"value":"~Jeff_Phillips1"},"authors":{"value":["Sudipto Ghosh","Sujoy Nath","Sunny Manchanda","Tanmoy Chakraborty"]}},"version":2},{"content":{"thoroughness":{"value":2},"agreement":{"value":3},"code_of_conduct_acknowledgement":{"value":"Yes"},"strengths":{"value":"- **Clear Positioning**: The paper defines a specific problem (monolithic LLM dominance) and offers a concrete alternative: Expert‑Orchestration with specialized models.\n\n- **Concrete Architecture**: The paper introduces tangible components—Judge Models for evaluation and Routers for task assignment—moving beyond high‑level critique to actionable design.\n\n- **Roadmap & Open Questions**: Section 6 outlines explicit research directions and challenges, guiding the community toward future work.\n\n- **Grounded Feasibility**: References existing systems (Mixture‑of‑Experts, FrugalGPT) demonstrate that the proposed framework can be built with current technologies.\n\n- **Interdisciplinary Foundation**: The argument draws on economic theory, organizational science, and cognitive models to justify specialization over generalization.\n\n- **Ethical & Economic Lens**: By incorporating discussions of bias, safety, cost‑efficiency, and societal impact, the paper frames technical choices within broader responsible‑AI considerations.\n\n- **Comprehensive Literature Coverage**: It synthesizes recent research on routing, MoE, continual learning, and foundational texts, establishing a solid knowledge base for the proposed framework."},"weaknesses":{"value":"- **Idealistic Core Dependence**: The framework hinges on Judge models whose performance/objectivity directly determines overall success; no robustness analysis is provided.\n\n- **Engineering Burden**: Practical challenges (API standardisation, versioning, large‑scale reliability) are acknowledged but not addressed or scoped.\n\n- **Uncertain Paradigm Shift**: Similar distributed expert systems already exist; the manuscript fails to demonstrate how its approach would materially shift practice.\n\n- **Missing Trade‑offs**: No discussion of downsides such as increased routing maintenance, MoE gating complexity, heterogeneous hardware, latency constraints, or regulatory hurdles.\n\n- **Ambiguous Expert Definition**: The paper does not clearly distinguish expert models from judges/routers; it conflates fine‑tuned LLMs with truly specialized models, undermining the claimed benefits.\n\n- **Misapplied Interdisciplinary Analogy**: Arguments drawn from economics or cognitive science may not translate to LLM training dynamics (breadth vs. depth), limiting relevance.\n\n- **Limited Novelty of Routing**: Routing has precedent in MoE and LLM training; the paper does not articulate a clear added value over existing routing mechanisms."},"confidence":{"value":2},"questions":{"value":"The Author Survey has answered the questions raised by the reviewers. No further questions need to be articulated."},"rating":{"value":4},"ethics":{"value":"No."}},"parentInvitations":"NeurIPS.cc/2025/Position_Paper_Track/-/Meta_Review","nonreaders":[],"tmdate":1761707394643,"tcdate":1757228478797,"writers":["NeurIPS.cc/2025/Position_Paper_Track","NeurIPS.cc/2025/Position_Paper_Track/Submission63/Area_Chair_M398"],"signatures":["NeurIPS.cc/2025/Position_Paper_Track/Submission63/Area_Chair_M398"],"forum":"g8Fo6qtnMR","number":1,"license":"CC BY 4.0","cdate":1757228478797,"readers":["everyone"],"invitations":["NeurIPS.cc/2025/Position_Paper_Track/Submission63/-/Meta_Review","NeurIPS.cc/2025/Position_Paper_Track/-/Edit"],"mdate":1761707394643,"domain":"NeurIPS.cc/2025/Position_Paper_Track","replyto":"g8Fo6qtnMR","id":"PRd6p3DQ8X","forumContent":{"TLDR":{"value":"Replace the current reliance on monolithic general-purpose LLMs with a \"judges and routers\" architecture that coordinates specialist and generalist models to deliver more accurate, cost-effective, and user-aligned AI responses."},"venue":{"value":"Submitted to NeurIPS 2025 Position Paper Track"},"keywords":{"value":["LLM","judge","router","ensemble","meta-model","transparency","trust","experts"]},"abstract":{"value":"This position paper argues that the prevailing trajectory toward ever larger, more expensive generalist foundation models controlled by a handful of big companies limits innovation and constrains progress. \nWe challenge this approach by advocating for an “Expert Orchestration\" architecture as a superior alternative that democratizes LLM advancement. \nOur proposed architecture intelligently selects from thousands of existing models based on query requirements and decomposition, focusing on identifying what models do well rather than how they work internally. \nIndependent “judge\" models assess various models' capabilities across dimensions that matter to users, while “router\" systems direct queries to the most appropriate specialists within an approved set. \nThis approach delivers superior performance by leveraging targeted expertize rather than forcing costly generalist models to address all user requirements. \nThe expert orchestration paradigm represents a significant advancement in LLM capability by enhancing transparency, control, alignment, and safety through model selection while fostering a more democratic ecosystem."},"_bibtex":{"value":"@misc{\nanonymous2025beyond,\ntitle={Beyond Monoliths: Expert Orchestration for More Capable, Democratic, and Safe Large Language Models},\nauthor={Anonymous},\nyear={2025},\nurl={https://openreview.net/forum?id=g8Fo6qtnMR}\n}"},"title":{"value":"Beyond Monoliths: Expert Orchestration for More Capable, Democratic, and Safe Large Language Models"},"pdf":{"value":"/pdf/f7718725748238ae9cc25f1b32d29c4751f3174b.pdf"},"venueid":{"value":"NeurIPS.cc/2025/Position_Paper_Track/Rejected_Submission"},"paperhash":{"value":"quirke|beyond_monoliths_expert_orchestration_for_more_capable_democratic_and_safe_large_language_models"},"authorids":{"value":["~Philip_Quirke1","~Narmeen_Fatimah_Oozeer1","~Amir_Abdullah1","~Jason_Hoelscher-Obermaier1","~Clement_Neo1","~Jeff_M._Phillips1","~Joshua_Greaves1","~Chaithanya_Bandi2","~Fazl_Barez1","~Shriyash_Kaustubh_Upadhyay1"]},"authors":{"value":["Philip Quirke","Narmeen Fatimah Oozeer","Amir Abdullah","Jason Hoelscher-Obermaier","Clement Neo","Jeff M. Phillips","Joshua Greaves","Chaithanya Bandi","Fazl Barez","Shriyash Kaustubh Upadhyay"]}},"version":2},{"content":{"comment":{"value":"We thank the reviewer for the follow-up questions and are glad our clarifications resolved the question on Corollary 2.2. We address the remaining concerns below.\n\n**Q1: SLM zero-cost.** SLM-LLM collaboration approaches, such as routing, cascading, and speculative decoding, are built on the premise that SLM inference costs are substantially lower than LLM inference costs. The larger this cost differential, the greater the gains in these approaches. Accordingly, the zero-cost SLM setting is widely adopted in the SLM-LLM collaboration literature:\n\n1. Hybrid LLM [1] measures cost as the fraction of queries routed to the small model, with no cost attributed to SLM inference.\n2. RouteLLM [2] measures cost as the percentage of calls to the strong model, excluding the weak model's cost entirely.\n3. ABC [3] acknowledges SLM ensemble cost but excludes it from their edge-to-cloud cost accounting.\n\nEven when SLMs are accessed via paid APIs, the cost gap is typically substantial, as illustrated by representative providers below:\n\n| Provider | SLM| Input ($/1M) | Output ($/1M) | LLM| Input ($/1M) | Output ($/1M) |\n|-|-|-|-|-|-|-|\n| Artificial Analysis| Llama-3.2-3B | 0.08 | 0.08| Llama-3.3-70B | 0.58 | 0.71 |\n| Together AI | Gemma-3n-E4B | 0.02 | 0.04 | Llama-3.3-70B | 0.88 | 0.88 |\n| Groq | Llama-3.1-8B | 0.05 | 0.08 | Llama-3.3-70B | 0.59 | 0.79 |\n\nThe SLM is 6-30× cheaper per token than the LLM and, in some cases, entirely free, reinforcing the zero-cost assumption as reasonable.\n\nFinally, we agree that nonzero SLM costs reduce the cost advantage of all cascading-based methods (including Reactive Shepherding, ABC, and FrugalGPT), as they invoke the SLM multiple times before escalation.This is not unique to the above approaches; it is inherent to all cascading-based methods. However, Proactive Shepherding is minimally affected: it makes the hint decision before SLM inference, so the SLM runs only once per query — the same as routing baselines — largely preserving its cost advantage.\n\nWe will include a sensitivity analysis with nonzero SLM costs in the revised paper and report updated Tables 1-4 to transparently show how each method's advantage changes.\n\n**Q2.** *Sensitivity to hyperparameter choices.* Our method is not brittle to hyperparameter variation. The Pareto front curves demonstrate the performance of Reactive Shepherding across a wide range of hyperparameter configurations ($\\eta_{\\\\text{hint}}$, K, $\\alpha$). Across all four datasets, Reactive Shepherding consistently achieves favorable cost-accuracy trade-offs over a broad region of the hyperparameter space, rather than at a single narrow operating point.\n\n*Generality across tasks*. We acknowledge that Table 5 reports different hyperparameter values across datasets, but highlight three points.  First, this is standard practice across all baselines in our comparison: RouteLLM requires tuning its routing threshold, FrugalGPT requires tuning its confidence threshold, and GraphRouter switches between modes, as shown in Table 5. Our approach is no more demanding in this regard. Second, each hyperparameter serves an intuitive and distinct role: $\\\\alpha$ controls the cost-accuracy trade-off, $\\\\eta_{\\\\text{hint}}$ determines when a query is too complex for hinting and should go directly to the LLM, and K governs the confidence level of the cascading decision. Third, and most importantly, our model itself generalizes across tasks without retraining. For example, the HumanEval and MBPP results in Tables 3-4 are obtained using the model trained solely on GSM8K. This demonstrates that the learned hint allocation captures task-general patterns of query difficulty, and only lightweight hyperparameter calibration (not model retraining) is needed for new tasks. \n\n**Q3: Reactive Shepherding exceeding LLM-only baseline on HumanEval and MBPP.** We thank the reviewer for this keen observation. This occurs because Reactive Shepherding's cascading mechanism introduces an additional source of correct answers that the LLM-only baseline lacks. When the SLM achieves majority agreement across three samples (N=2 per Table 5), its consensus output is used directly without invoking the LLM. On code generation tasks, the SLM's majority-voted solution is sometimes correct on queries where the LLM's single-pass output fails to compile or contains bugs. The system thus exceeds the LLM-only baseline by combining two complementary strengths: SLM consensus for queries where agreement signals reliability, and LLM hints for harder queries. We observed the same phenomenon with ABC, which also uses SLM agreement as a gating mechanism. We will add this explanation when we add these new figures to the revised paper.\n\n[1]Ding, Dujian, et al. \"Hybrid llm: Cost-efficient and quality-aware query routing.\" ICLR (2024)\n\n[2]Ong, Isaac, et al. \"Routellm: Learning to route llms with preference data.\" ICLR (2025)\n\n[3]Kolawole, Steven, et al. \"Agreement-based cascading for efficient inference.\" TMLR(2025)"}},"parentInvitations":"ICML.cc/2026/Conference/-/Reply_Rebuttal_Comment","tmdate":1782369955893,"tcdate":1775325279925,"writers":["ICML.cc/2026/Conference","ICML.cc/2026/Conference/Submission30175/Authors"],"signatures":["ICML.cc/2026/Conference/Submission30175/Authors"],"forum":"H8rywbFXuM","number":1,"license":"CC BY 4.0","cdate":1775325279925,"readers":["everyone"],"invitations":["ICML.cc/2026/Conference/Submission30175/Official_Review4/Rebuttal1/Rebuttal_Acknowledgement1/-/Reply_Rebuttal_Comment","ICML.cc/2026/Conference/-/Edit"],"mdate":1782369955893,"domain":"ICML.cc/2026/Conference","replyto":"7YEc5IoaxK","id":"7vAW1alhmG","forumContent":{"venue":{"value":"Submitted to ICML 2026"},"keywords":{"value":["LLM routing; LLM cascading; SLM-LLM collaboration; Cost-efficient inference"]},"_bibtex":{"value":"@misc{\nanonymous2026pay,\ntitle={Pay for Hints, Not Answers: {LLM} Shepherding for Cost-Efficient Inference},\nauthor={Anonymous},\nyear={2026},\nurl={https://openreview.net/forum?id=H8rywbFXuM}\n}"},"title":{"value":"Pay for Hints, Not Answers: LLM Shepherding for Cost-Efficient Inference"},"paperhash":{"value":"dong|pay_for_hints_not_answers_llm_shepherding_for_costefficient_inference"},"primary_area":{"value":"deep_learning->large_language_models"},"abstract":{"value":"Large Language Models (LLMs) deliver state-of-the-art performance on complex reasoning tasks, but their inference costs limit deployment at scale. Small Language Models (SLMs) offer dramatic cost savings yet lag substantially in accuracy. Existing approaches—routing and cascading—treat the LLM as an all-or-nothing resource: either the query bypasses the LLM entirely, or the LLM generates a complete response at full cost. We introduce **LLM Shepherding**, a framework that requests only a short prefix (a hint) from the LLM and provides it to SLM. This simple mechanism is surprisingly effective for math and coding tasks: even hints comprising 10--30\\% of the full LLM response improve SLM accuracy significantly. Shepherding generalizes both routing and cascading, and it achieves lower cost under oracle decision-making. We develop a two-stage predictor that jointly determines whether a hint is needed and how many tokens to request. On the widely-used mathematical reasoning (GSM8K, CNK12) and code generation (HumanEval, MBPP) benchmarks, Shepherding reduces costs by 42--94\\% relative to LLM-only inference. Compared to state-of-the-art routing and cascading baselines, shepherding delivers up to 2.8$\\times$ cost reduction while matching accuracy. To our knowledge, this is the first work to exploit token-level budget control for SLM-LLM collaboration. Code is available at https://anonymous.4open.science/r/LLM_Sheperding-77C3/."},"pdf":{"value":"/pdf/b5f1805f8847279424d1a89d1e99c69e9d93c924.pdf"},"venueid":{"value":"ICML.cc/2026/Conference/Rejected_Submission"},"authorids":{"value":["~Ziming_Dong1","~Hardik_Sharma3","~Evan_O'Toole1","~Jaya_Prakash_Champati1","~Kui_Wu1"]},"authors":{"value":["Ziming Dong","Hardik Sharma","Evan O'Toole","Jaya Prakash Champati","Kui Wu"]}},"version":2},{"content":{"comment":{"value":"Thanks for your prompt questions. We answer your questions one by one.\n\n**Q1.Did you train separate models for each reward setting?**\n\n**Response:** Actually, similar to meta learning, we pre-trained a unified model on data combined from three different reward settings, and then fine-tuned it on data from each specific reward setting. We found that pre-training followed by fine-tuning converges faster and achieves better results compared to training from scratch. We compared the two methods under the Balanced scenario by fine-tuning them on 400 cases of interaction data and evaluating their performance on 100 cases, ensuring other hyperparameters were consistent. We found that the pre-train then fine-tune approach converged around 200 epochs and achieved a reward of 0.205, while the train from scratch method did not converge until about 900 epochs, reaching a reward of 0.194. \n\nAdditionally, we would like to thank the reviewer again for their constructive suggestions, which inspired us to consider GraphRouter's potential for future work and broader scenarios. We realized that in real-world scenarios, the massive use of LLMs by users would generate a lot of interaction data containing preferences (reward of different weights or preference rankings). Utilizing this data for pretraining GraphRouter could allow our model to better capture users' preferences for LLM selection and thus model more effectively. However, due to the lack of publicly available data and concerns about user privacy, we are currently unable to conduct specific research and experiments. We will discuss this further in the future work section (section 6) of the paper.\n\n**Q2.With a fixed trained model, is it possible to repurpose it for new reward settings, be it alternative weighting of the seen metrics or even unseen metrics?**\n\n**Response:** As discussed in the response to Q1, GraphRouter is pretrained on multiple different reward settings and finetuned on a new reward setting with a small amount of interaction data, enabling it to adapt to new scenarios. Specifically, we will answer your questions one by one. \n\n**[Reward settings with new weights]** To demonstrate GraphRouter's adaptability to new reward configurations, we introduced two novel settings: $Reward=0.3 Performance-0.5 Cost$, $Reward=0.6 Performance-0.4  Cost$. Initially, GraphRouter was pretrained on three standard reward formulas, and subsequently, it was fine-tuned on the new configurations using 400 cases each, followed by an evaluation on 100 test cases. GraphRouter achieved the reward of 0.225, outperforming the best baseline FrugalGPT with 0.193 in reward. The 14.2% increase in reward under the new settings confirms GraphRouter's superior performance.\n\n**[Unseen metrics]** In order to assess GraphRouter's capability to handle previously unseen metrics, we integrated a novel metric called Time, representing the response generation time across various LLM APIs. This metric is crucial as it addresses user concerns over response time efficiency. Accordingly, we established a new reward formula: $Reward=0.5 Performance-0.5  Time$. Following the pretraining on traditional settings, GraphRouter was fine-tuned and tested on this new metric with 400 and 100 cases, respectively. It outperformed FrugalGPT in this metric, with rewards of 0.248 compared to 0.219, marking an improvement of 11.7%. \n\n**Q3.Can GraphRouter be potentially extended for joint usage with optimization solvers? When may this be desirable?**\n\n**Response:**  Indeed, GraphRouter itself is a special deterministic decision-making optimization solver for LLM selection. It constructs tasks, queries, and LLM embeddings through interaction data and message passing, and calculates the probabilities for query selections of LLMs based on these embeddings, finally making deterministic optimization choices based on the highest probabilities. Additionally, we have indeed considered other optimization solvers that could enhance GraphRouter, such as RL-based solvers or bandit solvers, to address the LLM selection problem. GraphRouter can serve as an embedding extraction model, pulling effective task, query, and LLM embeddings from interaction data for these optimization solvers to use as state. Compared to GraphRouter, these solvers can explore actions based on state and make probabilistic decisions (like bandit). However, these optimization solvers, due to not being updated end-to-end with GraphRouter, can lead to potential misalignments making them less desirable. Moreover, our paper mainly focuses on a lightweight and end-to-end elegant framework to solve LLM selection, and since GraphRouter itself has already achieved promising performance compared to baselines, we will consider these optimization solvers in our future work.\n\nWe will incorporate these discussions into the revised PDF. Could you let us know if your concerns have been adequately addressed? If so, we kindly request you consider increasing the score."},"title":{"value":"Further response to Reviewer z8Nx"}},"tmdate":1732434090896,"tcdate":1732434090896,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission10977/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission10977/Authors"],"forum":"eU39PDsZtT","number":14,"license":"CC BY 4.0","cdate":1732434090896,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission10977/-/Official_Comment"],"mdate":1732434090896,"domain":"ICLR.cc/2025/Conference","replyto":"4gInZv942R","id":"inUKSeTz5g","forumContent":{"TLDR":{"value":"We propose GraphRouter, a novel graph-based router that improves LLM selection for diverse tasks by leveraging an inductive graph framework to optimize performance-cost trade-offs and generalize across various models and scenarios."},"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["LLM selection","Graph-based router","Contextual interactions","New LLM settings"]},"primary_area":{"value":"learning on graphs and other geometries & topologies"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The rapidly growing number and variety of Large Language Models (LLMs)\npresent significant challenges in efficiently selecting the appropriate LLM for\na given query, especially considering the trade-offs between performance and\ncomputational cost. Current LLM selection methods often struggle to generalize\nacross new LLMs and different tasks because of their limited ability to leverage\ncontextual interactions among tasks, queries, and LLMs, as well as their depen-\ndence on a transductive learning framework. To address these shortcomings, we\nintroduce a novel inductive graph framework, named as GraphRouter, which\nfully utilizes the contextual information among tasks, queries, and LLMs to en-\nhance the LLM selection process. GraphRouter constructs a heterogeneous\ngraph comprising task, query, and LLM nodes, with interactions represented as\nedges, which efficiently captures the contextual information between the query’s\nrequirements and the LLM’s capabilities. Through an innovative edge prediction\nmechanism, GraphRouter is able to predict attributes (the effect and cost of\nLLM response) of potential edges, allowing for optimized recommendations that\nadapt to both existing and newly introduced LLMs without requiring retraining.\nComprehensive experiments across three distinct effect-cost weight scenarios have\nshown that GraphRouter substantially surpasses existing routers, delivering a\nminimum performance improvement of 12.3%. In addition, it achieves enhanced\ngeneralization across new LLMs settings and supports diverse tasks with at least a\n9.5% boost in effect and a significant reduction in computational demands. This\nwork endeavors to apply a graph-based approach for the contextual and adaptive\nselection of LLMs, offering insights for real-world applications."},"_bibtex":{"value":"@inproceedings{\nfeng2025graphrouter,\ntitle={GraphRouter: A Graph-based Router for {LLM} Selections},\nauthor={Tao Feng and Yanzhen Shen and Jiaxuan You},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=eU39PDsZtT}\n}"},"title":{"value":"GraphRouter: A Graph-based Router for LLM Selections"},"pdf":{"value":"/pdf/2452b8ff98e5d19eedb55a812aabf2db832a2ec4.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"feng|graphrouter_a_graphbased_router_for_llm_selections"},"authorids":{"value":["~Tao_Feng5","~Yanzhen_Shen1","~Jiaxuan_You2"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Tao Feng","Yanzhen Shen","Jiaxuan You"]}},"version":2},{"content":{"comment":{"value":"We respectfully ask the reviewer whether there is a reason for reducing the paper's score, as they have confirmed they will keep their score.\n\nWe thank the reviewer for the follow-up questions. Here are our responses to the reviewer comments.\n\n1. We understand the reviewer’s point, and we agree that this is not a significant theoretical contribution.\n\n2. **Small model cost.** We acknowledge the reviewer's point that the setting with zero SLM cost deserves closer examination. We agree that with a nonzero SLM cost, the cost advantage of cascading-based methods –  including Reactive Shepherding, ABC, and FrugalGPT – is reduced, because these methods invoke the SLM multiple times before escalation. This is not unique to the above approaches; it is inherent to all cascading-based methods. However, Proactive Shepherding is minimally affected: it makes the hint decision before SLM inference, so the SLM runs only once per query. In the nonzero SLM cost regime, Proactive Shepherding's cost advantage over routing baselines is largely preserved, as both incur a single SLM call.\n\n    We will include a sensitivity analysis with nonzero SLM costs (e.g., the AWS Bedrock rate of $0.15/1M tokens) in the revised paper, and report updated Tables 1–4 to transparently show how each method's advantage changes.\n\n3. **Comparison with SD.** We have looked carefully into the cost of SD. Verifying draft tokens in parallel requires the same total FLOPs as generating them autoregressively – the LLM must compute attention over all preceding tokens for each verified position, regardless. When draft tokens are rejected, additional overhead is incurred through speculative sampling, which requires distributions from both the LLM and SLM to generate replacement tokens. Thus, SD's benefit is latency (fewer sequential GPU steps), not reduced computation. If LLM APIs charge per token – reflecting compute cost, not wall-clock time – SD does not reduce the user's monetary cost, even after assuming SLM generation cost is zero. Shepherding, by contrast, directly reduces the number of LLM tokens billed, yielding immediate savings under existing pricing models. The two approaches are complementary: SD reduces latency for the tokens the LLM generates, while shepherding reduces the number of tokens requested in the first place."}},"parentInvitations":"ICML.cc/2026/Conference/-/Reply_Rebuttal_Comment","tmdate":1782369955868,"tcdate":1775234501970,"writers":["ICML.cc/2026/Conference","ICML.cc/2026/Conference/Submission30175/Authors"],"signatures":["ICML.cc/2026/Conference/Submission30175/Authors"],"forum":"H8rywbFXuM","number":1,"license":"CC BY 4.0","cdate":1775234501970,"readers":["everyone"],"invitations":["ICML.cc/2026/Conference/Submission30175/Official_Review1/Rebuttal1/Rebuttal_Acknowledgement1/-/Reply_Rebuttal_Comment","ICML.cc/2026/Conference/-/Edit"],"mdate":1782369955868,"domain":"ICML.cc/2026/Conference","replyto":"5o7aqVmyuX","id":"ckZYjiQhvZ","forumContent":{"venue":{"value":"Submitted to ICML 2026"},"keywords":{"value":["LLM routing; LLM cascading; SLM-LLM collaboration; Cost-efficient inference"]},"_bibtex":{"value":"@misc{\nanonymous2026pay,\ntitle={Pay for Hints, Not Answers: {LLM} Shepherding for Cost-Efficient Inference},\nauthor={Anonymous},\nyear={2026},\nurl={https://openreview.net/forum?id=H8rywbFXuM}\n}"},"title":{"value":"Pay for Hints, Not Answers: LLM Shepherding for Cost-Efficient Inference"},"paperhash":{"value":"dong|pay_for_hints_not_answers_llm_shepherding_for_costefficient_inference"},"primary_area":{"value":"deep_learning->large_language_models"},"abstract":{"value":"Large Language Models (LLMs) deliver state-of-the-art performance on complex reasoning tasks, but their inference costs limit deployment at scale. Small Language Models (SLMs) offer dramatic cost savings yet lag substantially in accuracy. Existing approaches—routing and cascading—treat the LLM as an all-or-nothing resource: either the query bypasses the LLM entirely, or the LLM generates a complete response at full cost. We introduce **LLM Shepherding**, a framework that requests only a short prefix (a hint) from the LLM and provides it to SLM. This simple mechanism is surprisingly effective for math and coding tasks: even hints comprising 10--30\\% of the full LLM response improve SLM accuracy significantly. Shepherding generalizes both routing and cascading, and it achieves lower cost under oracle decision-making. We develop a two-stage predictor that jointly determines whether a hint is needed and how many tokens to request. On the widely-used mathematical reasoning (GSM8K, CNK12) and code generation (HumanEval, MBPP) benchmarks, Shepherding reduces costs by 42--94\\% relative to LLM-only inference. Compared to state-of-the-art routing and cascading baselines, shepherding delivers up to 2.8$\\times$ cost reduction while matching accuracy. To our knowledge, this is the first work to exploit token-level budget control for SLM-LLM collaboration. Code is available at https://anonymous.4open.science/r/LLM_Sheperding-77C3/."},"pdf":{"value":"/pdf/b5f1805f8847279424d1a89d1e99c69e9d93c924.pdf"},"venueid":{"value":"ICML.cc/2026/Conference/Rejected_Submission"},"authorids":{"value":["~Ziming_Dong1","~Hardik_Sharma3","~Evan_O'Toole1","~Jaya_Prakash_Champati1","~Kui_Wu1"]},"authors":{"value":["Ziming Dong","Hardik Sharma","Evan O'Toole","Jaya Prakash Champati","Kui Wu"]}},"version":2},{"content":{"comment":{"value":"Thanks again for your positive rating.\nWe will certainly add the above discussions and experiments to the revision.\n\n---\n\n> Q12. However, after re-reading the paper, I noticed that it currently lacks a comparison or at least a discussion of the existing cascade-based approaches [1-4] for assembling LLMs. \n\n**A12.** \nWe have discussed a cascade-based method (i.e., FrugalGPT) in the related work of the paper (Lines 75-77, Lines 182-184).\nThanks for bringing the four other cascade-based methods to our attention.\nWe agree that cascade-based methods are one direction to achieve cost-effectiveness when choosing LLMs, but they are different from RouterDC in terms of **settings, inference cost, and tasks**.\nWe discuss the differences between our RouterDC and the mentioned cascade-based methods below.\n\n\n(i) \nRouterDC considers **a different setting** compared with the cascade-based methods [1-4]. The cascade-based methods usually **assume that the capacity of LLMs depends on the model size**.\nTheir intuitive idea is to query LLMs from weak (small) to strong  (large) until a satisfactory answer is obtained instead of calling the strong LLMs for all queries. \nOur RouterDC does not require this assumption and can select a suitable LLM from multiple small or large candidate LLMs.\nHence, routing-based methods are more general.\nFurthermore, even if LLMs are of the same size, they may have different specialized capabilities.\n\n(ii)\nCascade-based methods [1-4] may call LLMs **multiple times** for a query (in the worst case, all candidate LLMs need to be called), but our RouterDC only needs to call the selected LLM **once** in inference/testing.\n\n(iii) \nRouterDC is general and can be used for **generation tasks**, but Model Cascading [1] and Online Cascade Learning [4] are limited to **classification task** (e.g., SST2, MRPC, IMDB).\nGeneration tasks are usually more useful and challenging than classification tasks in NLP.\n\nWe will include the above discussion and related works in the revision.\n\n---\n\n#### References\n\n[1] Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systems, EMNLP 2022\n\n[2] Language Model Cascades: Token-level Uncertainty and Beyond, ICLR 2024\n\n[3] Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoning, ICLR 2024\n\n[4] Online Cascade Learning for Efficient Inference over Streams, ICML 2024"},"title":{"value":"Reply to Reviewer 4ok1"}},"tmdate":1730891097365,"tcdate":1723603660319,"writers":["NeurIPS.cc/2024/Conference","NeurIPS.cc/2024/Conference/Submission14840/Authors"],"signatures":["NeurIPS.cc/2024/Conference/Submission14840/Authors"],"forum":"7RQvjayHrM","number":8,"license":"CC BY 4.0","cdate":1723603660319,"readers":["everyone"],"invitations":["NeurIPS.cc/2024/Conference/Submission14840/-/Official_Comment","NeurIPS.cc/2024/Conference/-/Edit"],"mdate":1730891097365,"domain":"NeurIPS.cc/2024/Conference","replyto":"ssX4HBptzP","id":"YrqLVNAOot","forumContent":{"venue":{"value":"NeurIPS 2024 poster"},"keywords":{"value":["LLM Routing; Large Language Models;"]},"primary_area":{"value":"natural_language_processing"},"abstract":{"value":"Recent works show that assembling multiple off-the-shelf large language models (LLMs) can harness their complementary abilities. To achieve this, routing is a promising method, which learns a router to select the most suitable LLM for each query. However, existing routing models are ineffective when multiple LLMs perform well for a query. To address this problem, in this paper, we propose a method called query-based Router by Dual Contrastive learning (RouterDC). The RouterDC model, which consists of an encoder and LLM embeddings, is trained by two proposed contrastive losses (sample-LLM and sample-sample losses). Experimental results show that RouterDC is effective in assembling LLMs and largely outperforms individual top-performing LLMs as well as existing routing methods on both in-distribution (+2.76\\%) and out-of-distribution (+1.90\\%) tasks. The source code is available at https://github.com/shuhao02/RouterDC."},"_bibtex":{"value":"@inproceedings{\nchen2024routerdc,\ntitle={Router{DC}: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Models},\nauthor={Shuhao Chen and Weisen Jiang and Baijiong Lin and James Kwok and Yu Zhang},\nbooktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},\nyear={2024},\nurl={https://openreview.net/forum?id=7RQvjayHrM}\n}"},"title":{"value":"RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Models"},"pdf":{"value":"/pdf/71d8ae93be9d114f2aa1765cbb40cad92623a1c9.pdf"},"venueid":{"value":"NeurIPS.cc/2024/Conference"},"paperhash":{"value":"chen|routerdc_querybased_router_by_dual_contrastive_learning_for_assembling_large_language_models"},"authorids":{"value":["~Shuhao_Chen1","~Weisen_Jiang1","~Baijiong_Lin1","~James_Kwok1","~Yu_Zhang3"]},"authors":{"value":["Shuhao Chen","Weisen Jiang","Baijiong Lin","James Kwok","Yu Zhang"]}},"version":2},{"content":{"summary":{"value":"This paper proposes a reinforcement learning framework for collaborative device-cloud LLM inference where lightweight on-device models learn to autonomously decide when to offload queries to powerful cloud LLMs. Unlike existing approaches using external routers, the method integrates routing capability directly into post-training through a reward maximization problem with hierarchical rewards. The authors develop a Group-Adaptive Policy Gradient (GAPG) algorithm featuring unbiased group-level gradient estimation and adaptive prompt filtering to balance local problem-solving with cloud assistance. Experiments across mathematical benchmarks demonstrate the approach consistently outperforms baselines, maintains stable training, and significantly narrows the performance gap to full cloud LLM usage while respecting offloading constraints."},"soundness":{"value":2},"confidence":{"value":4},"questions":{"value":"Please see weaknesses."},"rating":{"value":4},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"+ This work elegantly integrates routing optimization directly into the post-training process, eliminating the need for separate external routers. This unified approach allows the on-device LLM to simultaneously improve its problem-solving capabilities while learning judicious offloading strategies.\n+ The group-level policy gradient addresses GRPO's limitations in collaborative settings, while adaptive prompt filtering effectively enforces cloud usage constraints, preventing the model from over-relying on external assistance during training.\n+ The evaluation demonstrates performance across multiple model sizes (1B to 3B parameters) and diverse mathematical benchmarks."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"- The work lacks clear motivation and validation for why routing capability should be embedded in the on-device LLM with a first-device-then-cloud cascade way rather than using separate routers for directly choosing which LLM. The optimization goal appears vague—it achieves significantly lower accuracy than cloud LLM alone (cf. Table 1), yet doesn't explicitly quantify or optimize the cost trade-off between on-device computation and cloud invocation, making it difficult to assess whether the approach provides genuine practical benefits. In particular, accuracy should be the first objective in real-world deployment.\n\n- The core technical contribution is incremental. The group-level policy gradient (Proposition 3.1) closely resembles existing GRPO with minor modifications. The only novel component—adaptive prompt filtering—is a simple heuristic that selects training prompts based on response correctness ratios. This lacks theoretical justification and appears to be an ad-hoc empirical design without principled foundations for why this specific filtering strategy is optimal.\n\n- Evaluation is restricted exclusively to mathematical reasoning tasks (Countdown and MATH benchmarks). The paper completely ignores other important domains like general question answering, commonsense reasoning, coding, summarization, or multi-turn dialogue. This severely limits claims about generalizability, as mathematical problems may have unique characteristics that don't transfer to other collaborative inference scenarios where routing decisions differ fundamentally.\n\n- The paper compares against only one routing method (Task-Tuning&Router using DeBERTa) and naive random offloading, missing numerous recent works in LLM routing and cascade systems. Notable omissions include FrugalGPT, RouteLLM variants, context-aware cascading policies, and ensemble-based routing methods cited in related work but never experimentally compared, weakening claims of superiority.\n\n- The paper provides no ablation analysis to isolate the contribution of individual components. Critical questions remain unanswered: How much does hierarchical reward design contribute versus the gradient estimator? What's the impact of different reward weights (αₐ, αc, αf)? How sensitive is performance to the filtering ratio ρ? Without ablations, it's impossible to understand which design choices actually matter or whether simpler alternatives might work equally well."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762919107366,"tcdate":1761797215310,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission6850/Reviewer_Xpht"],"signatures":["ICLR.cc/2026/Conference/Submission6850/Reviewer_Xpht"],"forum":"SwHnok3JtD","number":2,"license":"CC BY 4.0","cdate":1761797215310,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission6850/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762919107366,"domain":"ICLR.cc/2026/Conference","replyto":"SwHnok3JtD","id":"7ReKF9CHhr","forumContent":{"TLDR":{"value":"We propose a unified post-training framework that integrates routing optimization, enabling the on-device LLM to improve its problem-solving ability while learning routing strategies."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Large Language Models","Device-Cloud Collaboration","Post-Training","Routing Optimization","Reinforcement Learning"]},"supplementary_material":{"value":"/attachment/0abec1d957d5dfe99e2938019c58c05fb162a7f5.zip"},"primary_area":{"value":"other topics in machine learning (i.e., none of the above)"},"abstract":{"value":"Device-cloud collaboration has emerged as a promising paradigm for deploying large language models (LLMs), combining the efficiency of lightweight on-device inference with the superior performance of powerful cloud LLMs. An essential problem in this scenario lies in deciding whether a given query is best handled locally or delegated to the cloud. Existing approaches typically rely on external routers, implemented as binary classifiers, which often struggle to determine task difficulty from the prompt's surface pattern. To address these limitations, we propose a framework where the on-device LLM makes routing decisions at the end of its solving process, with this capability instilled through post-training. In particular, we formulate a reward maximization problem with carefully designed rewards that encourage effective problem solving and judicious offloading to the cloud. To solve this problem, we develop a group-adaptive policy gradient algorithm, featuring a group-level policy gradient, designed to yield an unbiased gradient estimator of the reward, and adaptive prompt filtering, developed to provide complementary learning signals. Extensive experiments across models and benchmarks show that the proposed methodology consistently outperforms existing baselines and significantly narrows the gap to full cloud LLM performance."},"_bibtex":{"value":"@misc{\nfang2026collaborative,\ntitle={Collaborative Device-Cloud {LLM} Inference through Reinforcement Learning},\nauthor={Wenzhi Fang and Dong-Jun Han and Liangqi Yuan and Christopher Brinton},\nyear={2026},\nurl={https://openreview.net/forum?id=SwHnok3JtD}\n}"},"title":{"value":"Collaborative Device-Cloud LLM Inference through Reinforcement Learning"},"pdf":{"value":"/pdf/b0a3698c6181d5c8e4245b48f5ea8700da6b0ff7.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"fang|collaborative_devicecloud_llm_inference_through_reinforcement_learning"},"authorids":{"value":["~Wenzhi_Fang1","~Dong-Jun_Han1","~Liangqi_Yuan1","~Christopher_Brinton1"]},"authors":{"value":["Wenzhi Fang","Dong-Jun Han","Liangqi Yuan","Christopher Brinton"]}},"version":2},{"content":{"comment":{"value":"We are encouraged by the reviewers’ positive assessments, recognizing our \"market-based perspective\" as **novel and conceptually powerful** (Reviewers oqqC, VAy3) and our \"Inefficiency Theorem\" as providing **strong theoretical grounding** (Reviewers d1Fk, VAy3).\n\nBelow, we summarize the primary concerns raised and our responses.\n\n**(1) Theoretical Optimality (Global vs. Local)**\n\n* **Reviewer's Concern:** Reviewers (iVUH, d1Fk) noted that Agora’s trading protocol guarantees only a local equilibrium and asked about the gap to the global optimum.\n* **Our Response:** We clarified that finding a global optimum in heterogeneous markets is NP-hard. However, our new **Optimality Gap Analysis (Appendix D)** demonstrates that Agora’s equilibrium is empirically **near-optimal (0.8% gap)**. The Market-Aware Broker acts as a high-quality initializer, placing the system in a favorable basin of attraction for the subsequent deterministic greedy descent.\n\n**(2) Computational Complexity and Overhead**\n\n* **Reviewer's Concern:** Reviewers (oqqC, VAy3) worried that the Broker and trading protocol might introduce significant latency, making the system viable only for non-real-time applications.\n* **Our Response:** The coordination overhead is negligible.\n  * **Complexity:** Broker selection is $O(N)$ and trading is $O(N^2)$ for small $N$ ($\\le 15$), consuming only milliseconds.\n  * **End-to-End Latency:** The \"Average Inference Time\" reported (e.g., 2.5s - 3.5s in Table 5) is **strictly end-to-end**, encompassing all overheads.\n  * **Tractability:** We added Appendix M.3 to clarify that the Strategic Uncertainty Index uses a closed-form marginal approximation, requiring **no additional VLM forward passes**.\n\n**(3) Fairness of Baselines and Generalization**\n\n* **Reviewer's Concern:** Reviewers (d1Fk, oqqC) asked if text-centric routers (e.g., FrugalGPT) were compared fairly in a multimodal setting and requested evidence of generalization.\n* **Our Response:**\n  * **Fairness:** We implemented a **Unified Multimodal Wrapper** (Section 4.1) for text-centric baselines, equipping them with the visual encoder from the strongest agent to ensure a fair comparison.\n  * **Generalization:** As detailed in Note 3, new experiments on **MathVision, InfoVQA, and CC-OCR** (Appendix E.3) confirm that Agora consistently outperforms MoA and KABB-VLM, validating that our gains are not limited to MMBench.\n\n**(4) Strategic Manipulation**\n\n* **Reviewer's Concern:** Reviewer oqqC asked how the system handles strategic agents that might misreport uncertainty.\n* **Our Response:** Agora relies on mechanism design, not trust. The **Profitability-Driven Trading Protocol** (Eq. 5) enforces a hard economic constraint computed from system-defined parameters, preventing \"fake\" profitable trades. Additionally, the Broker’s Thompson Sampling naturally penalizes agents that under-report uncertainty but fail, reducing their future selection probability."},"title":{"value":"Note 2 for the New AC: Highlights of Reviewer Concerns and Our Responses"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764764636020,"tcdate":1764764612336,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission582/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission582/Authors"],"forum":"zeqCjGQB4U","number":17,"license":"CC BY 4.0","cdate":1764764612336,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission582/-/Official_Comment"],"mdate":1764764636020,"domain":"ICLR.cc/2026/Conference","replyto":"QX1mesZXPr","id":"0t6QKC8rpJ","forumContent":{"TLDR":{"value":"We presents a framework that reframes coordination as a decentralized market for uncertainty."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["agent; Vision Language Model; Uncernity"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Vision-Language Models (VLMs) enable powerful multi-agent systems, but scaling them is economically unsustainable: coordinating heterogeneous agents under information asymmetry often spirals costs. Existing paradigms, such as Mixture-of-Agents and knowledge-based routers, rely on heuristic proxies that ignore costs and collapse uncertainty structure, leading to provably suboptimal coordination.\nWe introduce Agora, a framework that reframes coordination as a decentralized market for uncertainty. Agora formalizes epistemic uncertainty into a structured, tradable asset (perceptual, semantic, inferential), and enforces profitability-driven trading among agents based on rational economic rules. A market-aware broker, extending Thompson Sampling, initiates collaboration and guides the system toward cost-efficient equilibria. Experiments on five multimodal benchmarks (MMMU, MMBench, MathVision, InfoVQA, CC-OCR) show that Agora outperforms strong VLMs and heuristic multi-agent strategies, e.g., achieving +8.5% accuracy over the best baseline on MMMU while reducing cost by over 3×. These results establish market-based coordination as a principled and scalable paradigm for building economically viable multi-agent visual intelligence systems."},"_bibtex":{"value":"@inproceedings{\nzhang2026why,\ntitle={Why Keep Your Doubts to Yourself? Trading Visual Uncertainties among Vision-Language Models},\nauthor={Jusheng Zhang and Yijia Fan and Kaitong Cai and Jing Yang and Jiawei Yao and Jian Wang and Guanlong Qu and Ziliang Chen and Keze Wang},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=zeqCjGQB4U}\n}"},"title":{"value":"Why Keep Your Doubts to Yourself? Trading Visual Uncertainties among Vision-Language Models"},"pdf":{"value":"/pdf/03f4bead771e441ade14013c814de209cd0630ea.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"zhang|why_keep_your_doubts_to_yourself_trading_visual_uncertainties_among_visionlanguage_models"},"authorids":{"value":["~Jusheng_Zhang3","~Yijia_Fan1","~Kaitong_Cai1","~Jing_Yang39","~Jiawei_Yao3","~Jian_Wang10","~Guanlong_Qu2","~Ziliang_Chen1","~Keze_Wang1"]},"authors":{"value":["Jusheng Zhang","Yijia Fan","Kaitong Cai","Jing Yang","Jiawei Yao","Jian Wang","Guanlong Qu","Ziliang Chen","Keze Wang"]}},"version":2},{"content":{"comment":{"value":"(cont’d from previous comment)\n\n## Generalization to different SLMs\n\n>How sensitive is SDR to changing the local SLM?\n\nDuring the rebuttal period, we conducted additional experiments in the one-remote scenario using an SLM from a different model family (Llama3.1-8B), while keeping all other experimental settings identical to those described in Section 4.1 of the paper. The following table reports the AUC comparison between our SDR method and the baseline methods. As shown, the effectiveness of our approach extends to a completely different model architecture.\n\n|Method \\ Dataset | MMLU | SQuAD | GSM8K |\n| --- | --- | --- | --- |\n|FrugalGPT | 0.6361   | 0.8493   | 0.5054   |\n|HybridLLM | 0.5371   | 0.5404   | 0.5031   |\n|SDR (ours)| **0.6714**  | **0.8557**   | **0.5062**    |\n\nMoreover, as you suggested, we will continue to include experiments with SLMs of different parameter scales and quantization configurations in future revisions. However, we would like to note that the variations introduced by different parameter sizes or quantization levels within the same model family are typically much smaller than the architectural differences across model families. Therefore, the above results already provide strong evidence of the generalizability of our method across different SLMs.\n\n>Can the decider trained on one SLM transfer without re-labeling?\n\nLabels essentially reflects the relative capability between the SLM and the LLM on a given dataset. If we replace the SLM, it is unavoidable that we must collect the new SLM’s responses on the dataset, since different models naturally exhibit non-negligible performance differences. On the other hand, if no new remote LLM is introduced, the LLM response labels can be fully reused.\n\n## Exploring additional approaches in the multi-remote scenario\n>For the multi-remote case, why is averaging the scores the right aggregation? Have you compared to learned policies (e.g., budget-constrained bandits or cost-aware ranking) or provided a regret bound?\n\nThe directions you suggested are indeed promising. However, the primary focus of this work is to introduce improved training objectives and module designs within a single-remote setting. And a simple aggregation strategy already produces performance superior to the baselines in the multi-remote scenario is an encouraging byproduct, demonstrating preliminary extensibility across different application settings. Exploring more sophisticated decision mechanisms under multi-remote configurations would undoubtedly further unlock the potential of our method and represents an important direction for future research. We appreciate your constructive suggestion.\n\nOnce again, we are grateful for your thoughtful review and valuable feedback. We hope that our responses have satisfactorily addressed all of your concerns.\n\nSincerely,\nAll authors"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764312193327,"tcdate":1764312193327,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission6798/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission6798/Authors"],"forum":"uSMttGgyuZ","number":3,"license":"CC BY 4.0","cdate":1764312193327,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission6798/-/Official_Comment"],"mdate":1764312193327,"domain":"ICLR.cc/2026/Conference","replyto":"ovnqjfeTD6","id":"eh4rhCFxnv","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"TLDR":{"value":"To achieve high performance with limited monetary cost with collaboration of local SLMs and remote LLMs."},"keywords":{"value":["LLM Routing","Cost-Efficiency","Local-Remote Collaboration"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"The rapid advancement of large language models (LLMs) has led to remarkable performance across diverse domains such as question answering, creative writing, programming, etc., making them indispensable assistants in daily life and work. Currently, LLM services are primarily accessed in two ways: (i) paid access to cloud-hosted LLMs, which are powerful but introduce nontrivial cost; and (ii) deployment of small language models (SLMs) on personal devices or small clusters, which, while less powerful, are sufficient for handling relatively simple tasks. To achieve a balanced trade-off between monetary cost and task performance, we propose Selective Deferred Routing, a paradigm that enables cost-efficient collaboration between local SLMs and remote LLMs. In this framework, a user request is first processed by the local SLM, which not only generates a preliminary response but also provides rich semantic representations of the request. A lightweight decider module then leverages this information to either adopt the initial response or route the request in a single step to the most suitable remote LLM for a higher-quality response. Extensive experiments on 5 LLMs and 3 datasets demonstrate that our approach consistently outperforms existing multi-LLM collaboration methods across different cost–performance trade-off preferences."},"_bibtex":{"value":"@misc{\nmiao2026selective,\ntitle={Selective Deferred Routing: Enabling Cost-Efficient Collaboration between Local {SLM}s and Remote {LLM}s},\nauthor={Qijun Miao and Zhixuan Fang},\nyear={2026},\nurl={https://openreview.net/forum?id=uSMttGgyuZ}\n}"},"title":{"value":"Selective Deferred Routing: Enabling Cost-Efficient Collaboration between Local SLMs and Remote LLMs"},"pdf":{"value":"/pdf/53c64af1a280cc8a3b45cf0c65c03f4c04ee44ec.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"miao|selective_deferred_routing_enabling_costefficient_collaboration_between_local_slms_and_remote_llms"},"authorids":{"value":["~Qijun_Miao1","~Zhixuan_Fang1"]},"authors":{"value":["Qijun Miao","Zhixuan Fang"]}},"version":2},{"content":{"comment":{"value":"**Continuation of the response to Q1:**\n\n**Table 3: Comparison of various methods on large multi-task interaction dataset across Three Distinct Performance-Cost Weight Scenarios.** Bold denotes the best results. Each metric reflects average values from multiple evaluation rounds.\n\n| Model         | Performance (Performance First) | Cost (Performance First) | Reward (Performance First) | Performance (Balance) | Cost (Balance) | Reward (Balance) | Performance (Cost First) | Cost (Cost First) | Reward (Cost First) |\n|---------------|---------------------------------|--------------------------|----------------------------|-----------------------|----------------|------------------|--------------------------|-------------------|---------------------|\n| Largest LLM   | 0.321                           | 0.611                    | 0.321                      | 0.321                 | 0.611          | -0.145           | 0.321                    | 0.611             | -0.425             |\n| Smallest LLM  | 0.180                           | 0.018                    | 0.180                      | 0.180                 | 0.018          | 0.081            | 0.180                    | 0.018             | 0.022              |\n| Prompt LLM    | 0.260                           | 0.654                    | 0.260                      | 0.180                 | 0.026          | 0.077            | 0.184                    | 0.024             | 0.018              |\n| Hybrid LLM    | 0.350                           | 0.611                    | 0.350                      | 0.311                 | 0.256          | 0.028            | 0.184                    | 0.103             | -0.046             |\n| FrugalGPT     | 0.277                           | 0.357                    | 0.277                      | 0.259                 | 0.143          | 0.058            | 0.272                    | 0.034             | 0.027              |\n| C2MAB-V       | 0.254                           | 0.366                    | 0.254                      | 0.295                 | 0.122          | 0.087            | 0.261                    | 0.036             | 0.023              |\n| **GraphRouter** | 0.393                    | 0.220              | **0.393**                  | 0.297                 | 0.052          | **0.122**        | 0.299                | 0.018             | **0.046**          |\n| Oracle        | 0.432                           | 0.429                    | 0.432                      | 0.432                 | 0.398          | 0.171            | 0.330                    | 0.018             | 0.052              |\n\n\n**[1]** Evaluating large language models trained on code, arXiv 2021.\n\n**[2]** HotpotQA: A dataset for diverse, explainable multi-hop question answering, EMNLP 2018.\n\n**Q2. The current methodology overlooks the inherent relationships among LLMs. For instance, some LLMs belong to the same series (e.g., LLaMA3-7B, LLaMA3-8B). Adding links to indicate such parent or sibling connections could provide a more comprehensive graph modeling. I appreciate that the authors discuss this point in the limitations section. Additionally, common prior knowledge could enhance the LLM descriptions: e.g., CodeLLaMA series are effective for code-related tasks, while BaiChuan performs better in Chinese contexts.**   \n\n**Response:** Thanks for your constructive advice. We answer your questions from two aspects.\n\n**[Add inner edges among LLM nodes]**. Following the suggestions of the reviewer, we add inner edges among LLM nodes based on GraphRouter. Specifically, we proposed an LLM-link method that connects nodes within the same size or the same LLM series (such as those within the LLaMA series). We compared the performance of LLM-link and GraphRouter across three scenarios, as shown in the first table below. We observed that adding inner edges to LLM nodes impaired the performance of GraphRouter.We suspect that in the settings where all LLMs are observable, the similarities and differences in capabilities of all LLMs have already been captured through extensive message passing via query nodes, task nodes, and their interaction edges. This makes adding these inner edges of LLMs redundant in message passing."},"title":{"value":"Response to Reviewer pHun (2/5)"}},"tmdate":1732305246095,"tcdate":1732305246095,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission10977/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission10977/Authors"],"forum":"eU39PDsZtT","number":8,"license":"CC BY 4.0","cdate":1732305246095,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission10977/-/Official_Comment"],"mdate":1732305246095,"domain":"ICLR.cc/2025/Conference","replyto":"XMADM9GHEQ","id":"Oj5xoDO0yO","forumContent":{"TLDR":{"value":"We propose GraphRouter, a novel graph-based router that improves LLM selection for diverse tasks by leveraging an inductive graph framework to optimize performance-cost trade-offs and generalize across various models and scenarios."},"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["LLM selection","Graph-based router","Contextual interactions","New LLM settings"]},"primary_area":{"value":"learning on graphs and other geometries & topologies"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The rapidly growing number and variety of Large Language Models (LLMs)\npresent significant challenges in efficiently selecting the appropriate LLM for\na given query, especially considering the trade-offs between performance and\ncomputational cost. Current LLM selection methods often struggle to generalize\nacross new LLMs and different tasks because of their limited ability to leverage\ncontextual interactions among tasks, queries, and LLMs, as well as their depen-\ndence on a transductive learning framework. To address these shortcomings, we\nintroduce a novel inductive graph framework, named as GraphRouter, which\nfully utilizes the contextual information among tasks, queries, and LLMs to en-\nhance the LLM selection process. GraphRouter constructs a heterogeneous\ngraph comprising task, query, and LLM nodes, with interactions represented as\nedges, which efficiently captures the contextual information between the query’s\nrequirements and the LLM’s capabilities. Through an innovative edge prediction\nmechanism, GraphRouter is able to predict attributes (the effect and cost of\nLLM response) of potential edges, allowing for optimized recommendations that\nadapt to both existing and newly introduced LLMs without requiring retraining.\nComprehensive experiments across three distinct effect-cost weight scenarios have\nshown that GraphRouter substantially surpasses existing routers, delivering a\nminimum performance improvement of 12.3%. In addition, it achieves enhanced\ngeneralization across new LLMs settings and supports diverse tasks with at least a\n9.5% boost in effect and a significant reduction in computational demands. This\nwork endeavors to apply a graph-based approach for the contextual and adaptive\nselection of LLMs, offering insights for real-world applications."},"_bibtex":{"value":"@inproceedings{\nfeng2025graphrouter,\ntitle={GraphRouter: A Graph-based Router for {LLM} Selections},\nauthor={Tao Feng and Yanzhen Shen and Jiaxuan You},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=eU39PDsZtT}\n}"},"title":{"value":"GraphRouter: A Graph-based Router for LLM Selections"},"pdf":{"value":"/pdf/2452b8ff98e5d19eedb55a812aabf2db832a2ec4.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"feng|graphrouter_a_graphbased_router_for_llm_selections"},"authorids":{"value":["~Tao_Feng5","~Yanzhen_Shen1","~Jiaxuan_You2"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Tao Feng","Yanzhen Shen","Jiaxuan You"]}},"version":2},{"content":{"comment":{"value":"We thank all the reviewers for their valuable and constructive comments. As reviewers are concerned about the novelty and contribution of this work, we summarize the novelty and contributions below:\n\n- Problem setup: 1) we focus on a new type of QA task where the model needs to write code to fetch relevant information from APIs; 2) we particularly consider the ecologically valid scenario where queries appear sequentially over time. For 1) existing work relies on pre-defined tools/functions to address queries that require external APIs, while we focus on using free-form code to fetch the information, which is more flexible and challenging, requiring no offline preparation. For 2), existing works typically consider queries as independent tasks, which is not how most real world systems will be deployed. We study how to make a system deployable by reducing its compute-cost over a sequence of streaming queries. \n\n- EcoAssistant consists of two simple yet effective techniques: assistant hierarchy and solution demonstration. First, the assistant hierarchy technique has not been studied before; to the best of our knowledge, the most relevant work is another ICLR’2024 submission [1], where the authors present a similar idea called model cascade, but it is designed for single-turn text generation task, while we focus on the direction of LLM agents where a chat LLM has to cooperate with a code executor to address the user queries. Second, the solution demonstration is unique, as far as we know there is no existing work that presents a similar technique. The core idea is that when a query is successfully addressed, the conversation history from the two-agent system could be useful for helping solve future queries. In particular, we identify the last code snippet of the conversation history as a useful solution, and store it in a database to help with future queries.\n\n- The two techniques collectively contribute to the “indirect advising” effect. In particular, since the query-code database is shared by the assistants, the solution from a more powerful mode (GPT-4) stored in the database would be later retrieved to guide a weaker model (GPT-3.5). Such a mechanism requires no offline preparation but could further reduce the overall cost, because by looking at the solution from the more powerful model, the weaker model could solve more tasks and reduce the invocation of more expensive models. \n\n\n[1] FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. ICLR submission id 6216"},"title":{"value":"General Response"}},"tmdate":1700694336425,"tcdate":1700630489442,"writers":["ICLR.cc/2024/Conference","ICLR.cc/2024/Conference/Submission436/Authors"],"signatures":["ICLR.cc/2024/Conference/Submission436/Authors"],"forum":"vxZW1aROZA","number":6,"license":"CC BY 4.0","cdate":1700630489442,"readers":["everyone"],"invitations":["ICLR.cc/2024/Conference/Submission436/-/Official_Comment"],"mdate":1700694336425,"domain":"ICLR.cc/2024/Conference","replyto":"vxZW1aROZA","id":"NnT6uHfrif","forumContent":{"venue":{"value":"Submitted to ICLR 2024"},"TLDR":{"value":"We study how to affordably and accurately use conversational large language models as assistants to tackle the code-driven question answering task."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Larger language models (LLMs)","Conversational LLMs","LLM as assistant","Cost effective deployment of LLMs"]},"primary_area":{"value":"general machine learning (i.e., none of the above)"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide."},"abstract":{"value":"Today, users ask Large language models (LLMs) as assistants to answer queries that require external knowledge; they ask about the weather in a specific city, about stock prices, and even about where specific locations are within their neighborhood. These queries require the LLM to produce code that invokes external APIs to answer the user's question, yet LLMs rarely produce correct code on the first try, requiring iterative code refinement upon execution results. In addition, using LLM assistants to support high query volumes can be expensive. In this work, we contribute a framework, EcoAssistant, that enables LLMs to answer code-driven queries more affordably and accurately. EcoAssistant contains three components. First, it allows the LLM assistants to converse with an automatic code executor to iteratively refine code or to produce answers based on the execution results. Second, we use a hierarchy of LLM assistants, which attempts to answer the query with weaker, cheaper LLMs before backing off to stronger, expensive ones. Third, we retrieve solutions from past successful queries as in-context demonstrations to help subsequent queries. Empirically, we show that EcoAssistant offers distinct advantages for affordability and accuracy, surpassing GPT-4 by 10 points of success rate with less than 50% of GPT-4's cost."},"_bibtex":{"value":"@misc{\nzhang2024ecoassistant,\ntitle={EcoAssistant: Using {LLM} Assistant More Affordably and Accurately},\nauthor={Jieyu Zhang and Ranjay Krishna and Ahmed Hassan Awadallah and Chi Wang},\nyear={2024},\nurl={https://openreview.net/forum?id=vxZW1aROZA}\n}"},"title":{"value":"EcoAssistant: Using LLM Assistant More Affordably and Accurately"},"pdf":{"value":"/pdf/7219ca8952383a1e1ae050c3a2698d13ca1712a0.pdf"},"venueid":{"value":"ICLR.cc/2024/Conference/Rejected_Submission"},"paperhash":{"value":"zhang|ecoassistant_using_llm_assistant_more_affordably_and_accurately"},"authorids":{"value":["~Jieyu_Zhang1","~Ranjay_Krishna1","~Ahmed_Hassan_Awadallah1","~Chi_Wang3"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors' identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Jieyu Zhang","Ranjay Krishna","Ahmed Hassan Awadallah","Chi Wang"]}},"version":2},{"content":{"comment":{"value":"The review thread by hRVj has become unconstructive, as they fixated on the similarity of the **problem statement (model routing) in [1] and our paper, and dismissed the novelties and contributions of our solution**.  \n\nTo provide clarity for other reviewers and the area chair, we summarize our contributions and key differences from [1] to ensure a fair and comprehensive evaluation:  \n\n*Architecture innovations:*\n\n- Performance Predictor: a lightweight LLM that estimates the expected performance of various underlying LLMs on a given query without executing them\n- Cost-Aware Selector: a novel module to effectively select the best-performing model to maximize the overall effectiveness of the responses within a given budget constraint\n- Universe Constructor.  a novel approach to select models that are complementary to achieve the highest performance, given a predefined limitation on the number of serving models.  \n\n*Empirical results:*\n- Outperforming Mixtral, the state-of-the-art open-source LLM at the time, by a significant margin.\n- Surpassing GPT-4’s performance while significantly cutting inference costs.  \n\n**Key Differences from [1]:** \n- [1] relied on hand-picked underlying models (ada, babbage, curie, davinci), whereas our Universe Constructor dynamically identifies 56 models from over 1,000 available open-source models in Huggingface's OpenLLM benchmark.\n- According to Table 3 in [1], their approach failed to outperform the underlying models, whereas our method significantly surpasses these models in performance and reduces inference costs (see Table 1 and Figure 2).\n- The models used in [1] are outdated and the experimental results can not be interpreted in the context of today’s LLMs, while our approach leverages the latest and most powerful models, including variations of LLaMA2, Mixtral, and GPT-4.\nWhile we recognize that [1] partially addressed the routing problem, their method failed to resolve it effectively. Our manuscript (Lines 159–165) also details differences from [2].  \n\n**Regarding the Reviewer’s Requests:**\n\n The suggestion to **re-train the model proposed in [1]** and run the inference on MMLU for comparison **contradicts ICLR guidelines (FAQ)**, which discourage requests for **significant additional experiments**. We strongly believe that the authors of [1] should provide their trained models for comparison, as training models is costly and time-consuming. Nonetheless, we added a **strong KNN baseline** in our revision to further validate the effectiveness of our Routoo model.  \n\n**Request to the Area Chair:**  \n\nWe urge the area chair to assess reviewer hRVj’s evaluation to ensure a fair judgment. The insistence on **focusing solely on [1] diverges from the broader evaluation** provided by other reviewers.  \n\n[1] Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modeling. https://arxiv.org/abs/2308.06077.  \n[2] FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance, https://arxiv.org/pdf/2305.05176"},"title":{"value":"Official Comment by Author"}},"tmdate":1733309298958,"tcdate":1733309287390,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission11514/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission11514/Authors"],"forum":"RQ9fQLEajC","number":24,"license":"CC BY 4.0","cdate":1733309287390,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission11514/-/Official_Comment"],"mdate":1733309298958,"domain":"ICLR.cc/2025/Conference","replyto":"XlHdSvGmQ7","id":"TAqwuXRrVb","forumContent":{"TLDR":{"value":"An LLM-based Router trained to route to the most effective LLM optimizing performance, cost, and efficiency, creating a new state-of-the-art model by integrating knowledge of underlying models."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["LLM","efficiency","knowledge integration","generation"]},"supplementary_material":{"value":"/attachment/2e84b1f1c381960311d11bdd23f4d081c4e4d14a.zip"},"primary_area":{"value":"generative models"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"LLMs with superior response quality—particularly larger or closed-source models—often come with higher inference costs, making their deployment inefficient and costly. Meanwhile, developing foundational LLMs from scratch is becoming increasingly resource-intensive and impractical for many applications. To address the challenge of balancing quality and cost, we introduce Routoo, an architecture designed to optimize the selection of LLMs for specific prompts based on performance, cost, and efficiency. Routoo provides controllability over the trade-off between inference cost and quality, enabling significant reductions in inference costs for a given quality requirement.\nRoutoo comprises two key components: a performance predictor and cost-aware selector. The performance predictor is a lightweight LLM that estimates the expected performance of various underlying LLMs on a given prompt without executing them. The cost-aware selector module then selects the most suitable model based on these predictions and constraints such as cost and latency, significantly reducing inference costs for the same quality. \nWe evaluated Routoo using the MMLU benchmark across 57 domains employing open-source models. Our results show that Routoo matches the performance of the Mixtral 8x7b model while reducing inference costs by one-third. Additionally, by allowing increased costs, Routoo surpasses Mixtral's accuracy by over 5\\% at equivalent costs, achieving an accuracy of 75.9\\%. When integrating GPT4 into our model pool, Routoo nearly matches GPT4's performance at half the cost and exceeds it with a 25\\% cost reduction. \nThese outcomes highlight Routoo's potential to significantly reduce inference costs without compromising quality, and even to establish new state-of-the-art results by leveraging the collective capabilities of multiple LLMs."},"_bibtex":{"value":"@misc{\nmohammadshahi2025routoo,\ntitle={Routoo: Learning to Route to Large Language Models Effectively},\nauthor={Alireza Mohammadshahi and Arshad Rafiq Shaikh and Majid Yazdani},\nyear={2025},\nurl={https://openreview.net/forum?id=RQ9fQLEajC}\n}"},"title":{"value":"Routoo: Learning to Route to Large Language Models Effectively"},"pdf":{"value":"/pdf/454cdce2a12d959ded0bc50a6c5b05c494a2d4dd.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"mohammadshahi|routoo_learning_to_route_to_large_language_models_effectively"},"authorids":{"value":["~Alireza_Mohammadshahi1","~Arshad_Rafiq_Shaikh1","~Majid_Yazdani1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Alireza Mohammadshahi","Arshad Rafiq Shaikh","Majid Yazdani"]}},"version":2},{"content":{"comment":{"value":"Dear AC and Reviewers,\n\nWe sincerely thank you for your insightful reviews and constructive feedback. We are encouraged by the consensus that:\n\n1.  **Novelty:** Our market-based perspective, treating uncertainty as a tradable asset to resolve information asymmetry, is novel and conceptually powerful (Reviewers $oqqC$, $VAy3$).\n2.  **Theoretical Foundation:** The formalization of \"Agnostic Coordination\" and the Inefficiency Theorem provides a strong theoretical grounding for the problem (Reviewer $d1Fk$, $VAy3$).\n3.  **Effectiveness:** Agora demonstrates substantial gains in accuracy (e.g., +8.5% on MMMU) and cost-efficiency across multiple benchmarks (Reviewers $oqqC$, $iVUH$).\n\nWe have updated our manuscript to incorporate your suggestions. Changes are highlighted in the revised PDF. Below, we address the three most common concerns regarding **Theoretical Optimality**, **Computational Complexity**, and **Generalization**.\n\n>**1. Local vs. Global Optimality (Response to $iVUH$, $d1Fk$)**\nA primary theoretical concern was whether Agora converges to a global optimum. We clarified in Section 3.3 that finding a global optimum in a decentralized, heterogeneous agent market is NP-hard. Therefore, Agora is designed to guarantee convergence to a **Locally Optimal Equilibrium** via deterministic greedy descent on the global cost function.\n\nTo quantify the quality of this equilibrium, we added an **Optimality Gap Analysis** (Appendix D). We constructed a Brute-Force Global Oracle on a subset of MMBench tasks. The results (Table R1 below) show that Agora's cost is only **0.80%** higher than the theoretical global minimum, while significantly outperforming greedy and random baselines. This empirically confirms that our Market-Aware Broker (MAB) initialization effectively guides the system into a high-quality basin of attraction.\n\n**Table R1. Optimality Gap Analysis (from Appendix D)**\n\n| Method                    | Avg. Cost (Normalized) | Gap to Global Opt. | Accuracy (%) |\n| :------------------------ | :--------------------- | :----------------- | :----------- |\n| Global Optimal (Oracle)   | 1.000                  | 0.00%              | 89.6         |\n| **Agora (Decentralized)** | **1.008**              | **+0.80%**         | **89.5**     |\n| Greedy (Local Only)       | 1.152                  | +15.2%             | 84.3         |\n\n---\n\n\n---\n\n\n>**2. Computational Complexity & Real-Time Viability (Response to $oqqC$, $VAy3$)**\nConcerns were raised regarding the overhead of the Broker and trading protocol. We have clarified in Section 3.3 and 4.1 that:\n\n* **Complexity:** The Broker selection is $O(N)$ and the trading protocol is $O(N^2)$. For typical pool sizes ($N \\le 15$), these operations consume milliseconds.\n* **End-to-End Latency:** The \"Average Inference Time\" reported in our experiments (Figure 4 & Table 9) is **strictly end-to-end**, encompassing all coordination overheads. Agora achieves state-of-the-art accuracy with inference times (2.5s - 3.5s) comparable to standard routing baselines.\n* **Tractability:** We added Appendix M.3 to explain that the Strategic Uncertainty Index ($U_{strategic}$) uses a closed-form marginal approximation, avoiding expensive model re-execution.\n\n>**3. Robustness and Cross-Dataset Generalization (Response to $d1Fk$, $oqqC$)**\nTo address concerns about fairness and generalization:\n\n* **Fairness:** We implemented a **Unified Multimodal Wrapper** for text-centric baselines (FrugalGPT, RouteLLM) to ensure fair multimodal comparison (Section 4.1).\n* **Generalization:** We extended the head-to-head comparison against strong baselines (MoA, KABB) to three additional datasets (MathVision, InfoVQA, CC-OCR) in Appendix E.3. Agora consistently outperforms baselines across reasoning-heavy and perception-heavy tasks."},"title":{"value":"General Response (1)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763927080779,"tcdate":1763785052370,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission582/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission582/Authors"],"forum":"zeqCjGQB4U","number":9,"license":"CC BY 4.0","cdate":1763785052370,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission582/-/Official_Comment"],"mdate":1763927080779,"domain":"ICLR.cc/2026/Conference","replyto":"zeqCjGQB4U","id":"0ODm8GM5Ws","forumContent":{"TLDR":{"value":"We presents a framework that reframes coordination as a decentralized market for uncertainty."},"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["agent; Vision Language Model; Uncernity"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Vision-Language Models (VLMs) enable powerful multi-agent systems, but scaling them is economically unsustainable: coordinating heterogeneous agents under information asymmetry often spirals costs. Existing paradigms, such as Mixture-of-Agents and knowledge-based routers, rely on heuristic proxies that ignore costs and collapse uncertainty structure, leading to provably suboptimal coordination.\nWe introduce Agora, a framework that reframes coordination as a decentralized market for uncertainty. Agora formalizes epistemic uncertainty into a structured, tradable asset (perceptual, semantic, inferential), and enforces profitability-driven trading among agents based on rational economic rules. A market-aware broker, extending Thompson Sampling, initiates collaboration and guides the system toward cost-efficient equilibria. Experiments on five multimodal benchmarks (MMMU, MMBench, MathVision, InfoVQA, CC-OCR) show that Agora outperforms strong VLMs and heuristic multi-agent strategies, e.g., achieving +8.5% accuracy over the best baseline on MMMU while reducing cost by over 3×. These results establish market-based coordination as a principled and scalable paradigm for building economically viable multi-agent visual intelligence systems."},"_bibtex":{"value":"@inproceedings{\nzhang2026why,\ntitle={Why Keep Your Doubts to Yourself? Trading Visual Uncertainties among Vision-Language Models},\nauthor={Jusheng Zhang and Yijia Fan and Kaitong Cai and Jing Yang and Jiawei Yao and Jian Wang and Guanlong Qu and Ziliang Chen and Keze Wang},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=zeqCjGQB4U}\n}"},"title":{"value":"Why Keep Your Doubts to Yourself? Trading Visual Uncertainties among Vision-Language Models"},"pdf":{"value":"/pdf/03f4bead771e441ade14013c814de209cd0630ea.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"zhang|why_keep_your_doubts_to_yourself_trading_visual_uncertainties_among_visionlanguage_models"},"authorids":{"value":["~Jusheng_Zhang3","~Yijia_Fan1","~Kaitong_Cai1","~Jing_Yang39","~Jiawei_Yao3","~Jian_Wang10","~Guanlong_Qu2","~Ziliang_Chen1","~Keze_Wang1"]},"authors":{"value":["Jusheng Zhang","Yijia Fan","Kaitong Cai","Jing Yang","Jiawei Yao","Jian Wang","Guanlong Qu","Ziliang Chen","Keze Wang"]}},"version":2},{"content":{"comment":{"value":"**Q6. Could you elaborate on how the framework would handle dynamic updates to LLM capabilities, such as when models are fine-tuned or updated? Would this require rebuilding the entire graph?** \n\n**Response:** Thanks for your insightful question. When LLM models are fine-tuned or updated, we can view these LLM models as new LLMs. In addition, the strong inductive power enables GraphRouter to effectively learn new LLM nodes' embeddings by modeling their few-shot interactions with task nodes and query nodes. In this process, we can simply add on the previous graph, so there is no need to rebuild the entire graph. The only update required is the new LLM nodes' connections with the few-shots queries. In ***[section 4.2 of our paper and Table 5]***, we demonstrated through experiments that our method generalizes well to new LLMs. Therefore, when models are fine-tuned or updated, GraphRouter can seamlessly adapt by incorporating these changes without the significant computational overhead. \n\nWe greatly appreciate you raising this question, as it aligns with a similar point mentioned by Reviewer pHun in Q2. This has also inspired us to explore a method for addressing dynamic updates through a zero-shot approach, requiring only minor modifications to the edges in GraphRouter. Specifically, we draw an analogy to recommendation systems, which rely on the social networks of new and old users to address the cold start problem of new users. In our task, we considered using the addition of inner edges within LLM nodes to enable zero-shot capabilities for new LLMs (dynamically updated) in GraphRouter. We connect LLM nodes within the same size or the same LLM series (such as those within the LLaMA series). Based on this, we followed the training data settings described in section 4.2, but without the need for few-shots data, to conduct zero-shot LLM selection experiments. We set the first six LLMs in Table 3 as visible during training, while the last four are used solely for zero-shot experiments. We connected nodes within the same size or LLM series and conducted zero-shot experiments using this version of GraphRouter in the Balance scenario, as shown in the table below. We observed that under the zero-shot setting, GraphRouter (zero-shot) not only had an extremely low time cost but also approached the reward of the strongest baseline, C2MAB-V, in this scenario. All these findings highlight the great potential of leveraging the inner links of LLM nodes to enhance the zero-shot capabilities of GraphRouter. All the discussions and experimental results mentioned above have been updated in Section D of the Appendix and Table 29 of the revised pdf version.\n\n| **Method**              | **Reward** | **Reward Improvement (%)** | **Time Cost** | **Time Cost Reduction (%)** |\n|-------------------------|------------|-----------------------------|---------------|-----------------------------|\n| HybridLLM               | 0.01       | -94.71                      | 273.45        | 49.57                       |\n| FrugalGPT               | 0.171      | -9.52                       | 63.15         | 88.35                       |\n| C2MAB-V                 | 0.189      | 0.00                        | 542.25        | 0.00                        |\n| GraphRouter (zero-shot) | 0.182      | -3.7                        | 1.00          | 99.82                       |\n| GraphRouter (few-shots) | 0.207      | 9.52                        | 3.00          | 99.45                       |\n| GraphRouter (Trained)   | 0.219      | 15.87                       | 30.00         | 94.47                       |"},"title":{"value":"Response to Reviewer c8tQ (4/4)"}},"tmdate":1732558259622,"tcdate":1732304770119,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission10977/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission10977/Authors"],"forum":"eU39PDsZtT","number":6,"license":"CC BY 4.0","cdate":1732304770119,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission10977/-/Official_Comment"],"mdate":1732558259622,"domain":"ICLR.cc/2025/Conference","replyto":"fYp4hXfqc9","id":"29w48Sw1rH","forumContent":{"TLDR":{"value":"We propose GraphRouter, a novel graph-based router that improves LLM selection for diverse tasks by leveraging an inductive graph framework to optimize performance-cost trade-offs and generalize across various models and scenarios."},"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["LLM selection","Graph-based router","Contextual interactions","New LLM settings"]},"primary_area":{"value":"learning on graphs and other geometries & topologies"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The rapidly growing number and variety of Large Language Models (LLMs)\npresent significant challenges in efficiently selecting the appropriate LLM for\na given query, especially considering the trade-offs between performance and\ncomputational cost. Current LLM selection methods often struggle to generalize\nacross new LLMs and different tasks because of their limited ability to leverage\ncontextual interactions among tasks, queries, and LLMs, as well as their depen-\ndence on a transductive learning framework. To address these shortcomings, we\nintroduce a novel inductive graph framework, named as GraphRouter, which\nfully utilizes the contextual information among tasks, queries, and LLMs to en-\nhance the LLM selection process. GraphRouter constructs a heterogeneous\ngraph comprising task, query, and LLM nodes, with interactions represented as\nedges, which efficiently captures the contextual information between the query’s\nrequirements and the LLM’s capabilities. Through an innovative edge prediction\nmechanism, GraphRouter is able to predict attributes (the effect and cost of\nLLM response) of potential edges, allowing for optimized recommendations that\nadapt to both existing and newly introduced LLMs without requiring retraining.\nComprehensive experiments across three distinct effect-cost weight scenarios have\nshown that GraphRouter substantially surpasses existing routers, delivering a\nminimum performance improvement of 12.3%. In addition, it achieves enhanced\ngeneralization across new LLMs settings and supports diverse tasks with at least a\n9.5% boost in effect and a significant reduction in computational demands. This\nwork endeavors to apply a graph-based approach for the contextual and adaptive\nselection of LLMs, offering insights for real-world applications."},"_bibtex":{"value":"@inproceedings{\nfeng2025graphrouter,\ntitle={GraphRouter: A Graph-based Router for {LLM} Selections},\nauthor={Tao Feng and Yanzhen Shen and Jiaxuan You},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=eU39PDsZtT}\n}"},"title":{"value":"GraphRouter: A Graph-based Router for LLM Selections"},"pdf":{"value":"/pdf/2452b8ff98e5d19eedb55a812aabf2db832a2ec4.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"feng|graphrouter_a_graphbased_router_for_llm_selections"},"authorids":{"value":["~Tao_Feng5","~Yanzhen_Shen1","~Jiaxuan_You2"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Tao Feng","Yanzhen Shen","Jiaxuan You"]}},"version":2},{"content":{"summary":{"value":"This paper addresses **cost-effective LLM selection for tasks decomposed into sequential pipelines**, where different LLMs handle each subtask and outputs feed forward. This creates complex interdependencies: earlier LLM choices affect input quality and optimal selections for downstream subtasks.\n\nThe authors propose **Sequential Bandits**, a neural contextual bandit algorithm with two key innovations: (1) maintaining **separate neural networks for each (subtask, LLM) pair** to model rewards, and (2) **sequentially** selecting LLMs after observing previous outputs rather than committing to all choices simultaneously. The UCB-based selection strategy balances exploration-exploitation while incorporating costs via tunable parameters $\\alpha_i$.\n\n**Main contributions** include: (1) novel problem formulation for sequential LLM pipeline selection, (2) the Sequential Bandits algorithm grounded in neural contextual bandits, (3) a new medical diagnosis dataset from MIMIC-III, and (4) experiments showing 7.6% and 6.5% improvements over baselines on medical and telecommunications tasks. The work assumes task decomposition is pre-defined rather than learned."},"soundness":{"value":3},"confidence":{"value":4},"questions":{"value":"See weakness above"},"rating":{"value":4},"details_of_ethics_concerns":{"value":"N/A"},"code_of_conduct":{"value":"Yes"},"presentation":{"value":3},"contribution":{"value":2},"strengths":{"value":"**Originality:** The paper identifies a new problem that sits between existing LLM routing and cascading work. The key insight—that LLM choices for earlier subtasks affect optimal selections downstream—is well-motivated. \n\n**Quality:** The experimental methodology demonstrates careful design in several respects. The baselines are thoughtfully constructed to isolate specific contributions: Random shows learning value and fixed Llama/Tele shows adaptation value.\n\n**Clarity:** The paper is well-written with clear motivation and straightforward presentation. Figure 2 effectively illustrates the problem, and Algorithm 1 is easy to follow.\n\n**Significance:** The problem is timely given the rise of agentic AI and custom LLM assistants. The framework is general and shows meaningful practical improvements."},"flag_for_ethics_review":{"value":["No ethics review needed."]},"weaknesses":{"value":"**Limited Baseline Comparisons:** The most significant experimental weakness is the absence of comparisons to recent LLM routing and cascading methods extensively cited in related work, such as FrugalGPT, Hybrid LLM, and many other routing/cascading frameworks. While many routing methods require historical training data—and learning without such data is indeed a key motivation for the online approach—several evaluation strategies could address this gap: (1) provide a warm-start period where methods collect initial training data before comparison, or (2) evaluate after the online learning phase to compare converged performance. More fundamentally, demonstrating when and by how much online learning outperforms offline-trained routers (in terms of sample efficiency, final performance, or adaptability) would significantly strengthen the contribution. Without these comparisons, readers cannot assess whether the observed improvements come from genuinely superior algorithmic design.\n\n**Task Decomposition Assumption:** The paper assumes task decomposition is given, which is a major practical limitation explicitly acknowledged but not addressed. In real deployments, determining *how* to break down tasks is often harder than selecting models. The medical example (summarize → diagnose) seems intuitive, but for arbitrary complex queries, the optimal decomposition is non-obvious and likely query-dependent. The paper provides no guidance on when pipelines help versus hurt, or how to design good decompositions.\n\n**Lack of Theoretical Analysis:** While the paper adapts NeuralUCB's framework, it provides no formal regret bounds for the sequential setting. The theoretical contribution is essentially \"use existing neural bandit algorithms but with separate networks per arm.\" It's unclear whether standard NeuralUCB regret guarantees transfer to this setting where: (1) contexts change dynamically based on previous LLM outputs, (2) subtask rewards are interdependent, and (3) the super-arm reward is a complex function of base rewards. The exploration-exploitation trade-off in sequential decisions differs fundamentally from standard bandits. \n\n**Limited Experimental Scale and Scope:** The evaluation uses only 100 medical reports and 100 rounds of online learning, which is quite modest. More critically, there's no analysis of: (1) how performance scales with pipeline depth (only 2-3 subtasks tested), (2) sensitivity to the cost parameter α (no ablation shown), (3) what happens with more LLMs per subtask (experiments use 5-7 total models), or (4) performance in non-stationary settings where LLM capabilities or costs change. The token length prediction model's accuracy isn't reported—if predictions are poor, the cost-aware objective could be misleading. \n\n**Cost Model Limitations:** The cost model assumes costs are proportional to tokens, but ignores latency, which can be a critical \"cost\" in interactive applications. The token predictor is trained on LMSYS-Chat-1M, which may not generalize well to specialized domains (medical, telecom). The fixed token predictor never adapts during online learning, potentially leading to systematic errors."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Review","nonreaders":[],"tmdate":1762924174237,"tcdate":1761966283572,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission13581/Reviewer_CrjG"],"signatures":["ICLR.cc/2026/Conference/Submission13581/Reviewer_CrjG"],"forum":"EiJbxZTtDj","number":2,"license":"CC BY 4.0","cdate":1761966283572,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission13581/-/Official_Review","ICLR.cc/2026/Conference/-/Edit"],"mdate":1762924174237,"domain":"ICLR.cc/2026/Conference","replyto":"EiJbxZTtDj","id":"C4VZVDflGQ","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["neural bandits","online LLM selection"]},"supplementary_material":{"value":"/attachment/a9a80a4aff5ba151c2f87476eb660c59f0e0a5da.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"With the increasing popularity of large language models (LLMs) for a variety of tasks, there has been a growing interest in strategies that can predict which out of a set of LLMs will yield a successful answer at low cost. This problem promises to become more and more relevant as providers like Microsoft allow users to easily create custom LLM \"assistants'' specialized to particular types of queries. However, some tasks (i.e., queries) may benefit from breaking down the task into smaller subtasks, each of which can then be executed by a LLM expected to perform well on that specific subtask. For example, in extracting a diagnosis from medical records, one can first select an LLM to summarize the record, select another to validate the summary, and then select another, possibly different, LLM to extract the diagnosis from the summarized record. Unlike existing LLM selection or routing algorithms, this setting requires selecting a sequence of LLMs, with the output of each LLM feeding into the next and potentially influencing its success. Thus, unlike single LLM selection, the quality of each subtask's output directly affects the inputs, and hence the cost and success rate, of downstream LLMs, creating complex performance dependencies that must be learned during selection. We propose a neural contextual bandit-based algorithm that trains neural networks that model LLM success on each subtask in an online manner, thus learning to guide the LLM selections for the different subtasks, even in the absence of historical LLM performance data. Experiments on telecommunications question answering and medical diagnosis prediction datasets illustrate the effectiveness of our proposed approach compared to other LLM selection algorithms."},"_bibtex":{"value":"@misc{\natalar2026neural,\ntitle={Neural Bandit Based Optimal {LLM} Selection for a Pipeline of Tasks},\nauthor={Baran Atalar and Eddie Zhang and Carlee Joe-Wong},\nyear={2026},\nurl={https://openreview.net/forum?id=EiJbxZTtDj}\n}"},"title":{"value":"Neural Bandit Based Optimal LLM Selection for a Pipeline of Tasks"},"pdf":{"value":"/pdf/0ea0038c815706f637e52253b61de3097ff3ae5b.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"atalar|neural_bandit_based_optimal_llm_selection_for_a_pipeline_of_tasks"},"authorids":{"value":["~Baran_Atalar1","~Eddie_Zhang3","~Carlee_Joe-Wong1"]},"authors":{"value":["Baran Atalar","Eddie Zhang","Carlee Joe-Wong"]}},"version":2},{"content":{"comment":{"value":"We appreciate the reviewer’s insightful observations and constructive criticism.\n\n1. **Choice of Baseline**:\n\n    - (i) The baseline we choose [1] is the first paper in LLM Cascade which provides theoretical guarantee on the error rate. Their work reaches SOTA compared to predictive probability[2] (which is widely used but easy to overestimate), verbalized confidence[3]. In our experiment, we also add ablations settings which contains two variants of the LLM Cascades.  Last but not least, our inter-Cascade can be integrated to any LLM Cascade, we choose the SOTA as the reference.\n\n    - (ii) The performance of sole LLM is in Table 1. \n\n    - (iii) We are adding experiments on the benchmarks that reviewers mentioned. We will update the result once finished.\n\n    [1] Jung, Jaehun, Faeze Brahman, and Yejin Choi. \"Trust or escalate: Llm judges with provable guarantees for human agreement.\" arXiv preprint arXiv:2407.18370 (2024).\n\n    [2] Chen, Lingjiao, Matei Zaharia, and James Zou. \"Frugalgpt: How to use large language models while reducing cost and improving performance.\" arXiv preprint arXiv:2305.05176(2023).\n\n    [3] Tian, Katherine, et al. \"Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback.\" arXiv preprint arXiv:2305.14975 (2023).\n\n2. **Strategy Matching Function**: Our framework is general to any similarity matching functions. The empirical method we used is listed in Section 3.2 Experiments settings. In our experience, We employ the all-MiniLM-L6-v2 transformer to produce 384-dimensional sentence embeddings and calculate cosine distances between the question embedding vector and each strategy-related embedding vector. Rank them with the distance and return top-$k$ vectors, together with corresponding strategy texts.\n\n 3. **Refinement on Related Work**:\n\n    - (i) Thanks for the suggestions, we enlarged the related work part, and added more details in the main text.\n\n    - (ii) We emphasized the categories of the related work, e.g., In LLM Cascade part, we address that LLM Cascade falls into the category of  Ensemble after inference LLM Ensemble. In Retrieval-Augmented Generation (RAG) part, we split the related work into three groups and made comparisons to each sub-category. \n\n    - (iii) As multiple reviewers mentioned, we group the \"in-context learning involving the retrieval of the most relevant samples, dynamic RAG as History-aware RAG and made comparisons with our Inter-Cascade. We have tried our best but with the limit of pages, we just add brief outlines of related work in Introductions. The extensive comparison is placed in the appendix. \n\n4. **Refinement on writing and presentation**\n\n    - (i) To be more rigorous, we rewrite that sentence as”the deferral decision rule is trained or derived by algorithms offline.” to conclude the deferral functions that are trained from. \n\n    - (ii) Here offline means the model training stage, online means the inference stage. We mentioned online because we want to emphasize that our method doesn't require training and can conduct during inference/generation of answers.\n\n    - (iii) we modified and added more supporting materials (motivations on the consideration of similar problems; related works.) to the abstract and introduction to try to describe the problem and our ideas clearer.  We also add clarification on motivations and applications in **common response 1 & 2**."},"title":{"value":"Response to Weaknesses:"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763844564644,"tcdate":1763842939785,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission15383/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission15383/Authors"],"forum":"fIFYBtjn2h","number":12,"license":"CC BY 4.0","cdate":1763842939785,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission15383/-/Official_Comment"],"mdate":1763844564644,"domain":"ICLR.cc/2026/Conference","replyto":"DaxC1Rtyws","id":"BjVINBq0Jl","forumContent":{"TLDR":{"value":"We propose Interactive-LLM-Cascade, a multi-LLM system where weak LLM can not only defer to strong LLM for difficult questions, but also get boosted by strong LLM's feedback in a long term."},"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Large language models","LLM Cascade","query routing","LLM Router"]},"supplementary_material":{"value":"/attachment/2816942e4b7a3f0e5f8013391386a1f45379c68f.zip"},"primary_area":{"value":"generative models"},"abstract":{"value":"Large Language Models (LLMs) vary widely in their capabilities, with larger models often having better performance but higher cost: choosing an LLM model often involves trading off performance and cost. The LLM Cascade is a paradigm that defers difficult queries from weak/cheap to strong/expensive models. This approach is nonadaptive: the deferral decision rule is trained or derived by algorithms offline. When confronted with similar or repeated queries, the LLM Cascade may then repeatedly consult the expensive model and incur higher cost. To improve the cascading efficiency, we propose Inter-Cascade, an online and interactive LLM Cascade that extends the role of strong model from a backup helper to a long-term teacher. In our system, when a strong model resolves a difficult query, it also distills its solution into a generalized, reusable problem-solving strategy that boosts the weak model on subsequent queries. Adding strategies to queries enables the weak model to dynamically improve its performance over time, avoiding computationally and time-intensive fine-tuning. Empirically, compared with standard LLM Cascade baselines across multiple benchmarks, the Inter-Cascade significantly improves the accuracy of the weak model (by up to 33.06 absolute percentage points) and the overall system (by up to 5.53 absolute percentage points), while reducing the calls to strong models (by up to 48.05% relative reduction) and saving the corresponding fees (by up to 49.63% relative reduction). Inter-Cascade demonstrates the effective in-context knowledge transfer between LLMs, and provides a general, scalable framework applicable to both open-source and API-based LLMs."},"_bibtex":{"value":"@misc{\nwu2026not,\ntitle={Not only a helper, but also a teacher: Interactive {LLM} Cascade},\nauthor={Yu Wu and Shuo Wu and Ye Tao and Yansong Li and Anand D. Sarwate},\nyear={2026},\nurl={https://openreview.net/forum?id=fIFYBtjn2h}\n}"},"title":{"value":"Not only a helper, but also a teacher: Interactive LLM Cascade"},"pdf":{"value":"/pdf/57c9608871aa4c4e8d40375ed9bd85616c38b364.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"wu|not_only_a_helper_but_also_a_teacher_interactive_llm_cascade"},"authorids":{"value":["~Yu_Wu12","~Shuo_Wu1","~Ye_Tao2","~Yansong_Li2","~Anand_D._Sarwate2"]},"authors":{"value":["Yu Wu","Shuo Wu","Ye Tao","Yansong Li","Anand D. Sarwate"]}},"version":2},{"content":{"comment":{"value":"We thank the reviewer for their thoughtful feedback and constructive suggestions, and are very grateful that they found our work well-motivated and well-executed. \n\n**Cost-latency analysis.**\nThe reviewer raises an important point about the cost-latency trade-off. We fully agree that this topic warrants further exploration. We are excited to share additional theoretical and empirical results that explore this trade-off. \n\n\n1. **Static breadth-focused speculation (Algorithm 1) and selection of k.** For Algorithm 1, we obtain closed-form expressions for relative latency improvement ratio and relative cost increase ratio. These expressions allow practitioners to tune the number of speculative branches k *offline* by directly assessing this tradeoff.\n1. **Dynamic branch selection substantially reduces cost.** If additionally, one could obtain an estimation of the per-speculation accuracies online, then we prove theoretically the optimal strategy is a threshold policy: continue only when the predicted accuracy of a speculative branch exceeds a computable threshold. Empirically, this policy performs strongly—on the chess environment, it **achieves the same speedups with 40% fewer tokens**. \n1. **No exponential blowup in depth-focused speculation..** We analyze a depth-focused speculative policy supporting multi-step speculation. We show that the number of parallel branches remains bounded. We also provide a characterization of the resulting cost-latency tradeoff.\n1. **Environments that can reduce both cost and latency.** Interestingly, cost and latency do not always trade off against each other. In our OS-tuning environment where losslessness is not required, the Speculator accelerates convergence to the optimum so effectively that the system reaches the solution **using only 8% of the original time and 20% of the original token cost**.\n\n1. **No/low cost for self-serving deployments.** The discussions above measure cost in terms of *token count*, but in **self-hosted inference** the relevant cost is GPU-hours or fixed procurement costs—not per-token pricing. Under low-traffic conditions with continuous batching, speculative calls become **effectively free** because they utilize otherwise idle GPU compute bandwidth. Moreover, this opens up new directions in scheduling and workload balancing for optimizing latency and throughput in agentic LLM deployments.\n\nWe refer the reviewer to the response to hwQA for a detailed summary. \n\n**Choose a proper speculator.** The reviewer also asks how users can identify an appropriate speculative model. In practice, this is a **model-selection** problem, closely related to recent work on routing queries to the most suitable LLM [1,2]. Our framework naturally supports both offline profiling and online dynamic selection, making speculative actions essentially a new application of this model-selection paradigm.\n\nIn addition, we also find that small models are great candidates for Speculators. In fact, our E-commerce and HotpotQA environment both uses smaller counterparts of their big Actor models  as Speculators (GPT-5-nano for GPT-5, Gemini-2.5-Flash for Gemini-2.5-Pro, etc.) and obtained roughly 30% speculation accuracy. \n\n**Reference**\n1. Chen, Lingjiao, Matei Zaharia, and James Zou. \"Frugalgpt: How to use large language models while reducing cost and improving performance.\" arXiv preprint arXiv:2305.05176 (2023).\n1. Srivatsa, Kv Aditya, Kaushal Maurya, and Ekaterina Kochmar. \"Harnessing the power of multiple minds: Lessons learned from LLM routing.\" In Proceedings of the Fifth Workshop on Insights from Negative Results in NLP, pp. 124-134. 2024."}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1763692581498,"tcdate":1763692581498,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission22399/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission22399/Authors"],"forum":"P0GOk5wslg","number":9,"license":"CC BY 4.0","cdate":1763692581498,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission22399/-/Official_Comment"],"mdate":1763692581498,"domain":"ICLR.cc/2026/Conference","replyto":"ZM5iN4CT1M","id":"aem4yiwdM9","forumContent":{"venue":{"value":"ICLR 2026 Oral"},"TLDR":{"value":"We introduce speculative actions—a lossless framework that predicts likely actions using faster models, enabling multiple API calls to be executed in parallel and thus yields substantial acceleration."},"keywords":{"value":["AI Agents","Speculative Decoding","Parallel Execution","Agentic Serving","Agentic Simulation"]},"primary_area":{"value":"infrastructure, software libraries, hardware, systems, etc."},"abstract":{"value":"AI agents are increasingly deployed in complex, interactive environments, yet their runtime remains a major bottleneck for training, evaluation, and real-world use. Typical agent behavior unfolds sequentially, where each action requires an API call that can incur substantial latency. For example, a game of chess between two state-of-the-art agents can take hours. We introduce speculative actions, a lossless acceleration framework for general agentic systems. Inspired by speculative execution in microprocessors and speculative decoding in LLM inference, our method uses faster models to predict likely future actions and executes them in parallel, committing only when predictions match. We evaluate speculative actions across gaming, e-commerce, and web search environments, and additionally study a lossy extension in an operating systems setting. Across domains, we achieve up to 55% next-action prediction accuracy, translating into substantial latency reductions. Finally, we present a cost–latency analysis that formalizes the tradeoff between speculative breadth and time savings. This analysis enables principled tuning and selective branch launching, to ensure multi-branch speculation delivers practical speedups without prohibitive cost growth."},"_bibtex":{"value":"@inproceedings{\nye2026speculative,\ntitle={Speculative Actions: A Lossless Framework for Faster {AI} Agents},\nauthor={Naimeng Ye and Arnav Ahuja and Georgios Liargkovas and Yunan Lu and Kostis Kaffes and Tianyi Peng},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=P0GOk5wslg}\n}"},"title":{"value":"Speculative Actions: A Lossless Framework for Faster AI Agents"},"pdf":{"value":"/pdf/b87fe88f08fe928f9c1f66d781783635dde3af9a.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"ye|speculative_actions_a_lossless_framework_for_faster_ai_agents"},"authorids":{"value":["~Naimeng_Ye1","~Arnav_Ahuja1","~Georgios_Liargkovas1","~Yunan_Lu2","~Kostis_Kaffes1","~Tianyi_Peng1"]},"authors":{"value":["Naimeng Ye","Arnav Ahuja","Georgios Liargkovas","Yunan Lu","Kostis Kaffes","Tianyi Peng"]}},"version":2},{"content":{"comment":{"value":"**Continuation of the response to Q2:**\n\nOn the other hand, we are very pleased with the reviewer raising this question, as well as Reviewer c8tQ's Q6. We considered that adding inner edges among LLM nodes could enable GraphRouter to address zero-shot scenarios. Specifically, inspired by recommendation systems that often rely on the social networks of new and old users to address the cold start problem of new users, we considered using the addition of inner edges within LLM nodes to enable zero-shot capabilities for new LLMs in GraphRouter. Based on this, we followed the training data settings described in section 4.2, without the need for few-shots data, to conduct zero-shot LLM selection experiments. We set the first six LLMs in Table 3 as visible during training, while the last four are used solely for zero-shot experiments. We connected nodes within the same size or LLM series and conducted zero-shot experiments using this version of GraphRouter in the Balance scenario, as shown in the second table below. We observed that under the zero-shot setting, GraphRouter (zero-shot) not only had an extremely low time cost but also approached the reward of the strongest baseline, C2MAB-V, in this scenario. All these findings demonstrate the great potential of modeling inner links of LLM nodes for the zero-shot capabilities of GraphRouter.\n\nTo conclude, although adding inner edges among LLM nodes does not necessarily enhance GraphRouter’s performance when all LLMs are observable, we have demonstrated that modeling internal connections between LLM nodes is promising to enhance GraphRouter in the zero-shot setting. All the discussions and experimental results mentioned above have been updated in Section D of the Appendix and Table 28-29 of the revised pdf version.\n\n\n**Table 1: Comparison of GraphRouter and LLM-link across different scenarios focusing on Reward**\nThis table evaluates reward variations between two models under three distinct scenarios. Each number is formatted to three decimal places for precision.\n\n| Model      | Performance First | Balance | Cost First |\n|------------|-------------------|---------|------------|\n| GraphRouter| 0.539             | 0.209   | 0.064      |\n| LLM-link   | 0.530             | 0.192   | 0.061      |\n \n**Table 2: Comparison of methods in the zero-shot and few-shot setting on Reward, Time Cost, and the corresponding percentage Reward improvements and Time Cost reduction rate, relative to the most costly method (C2MAB-V (dai2024cost)). The experiment is conducted in the Balance scenario.**\n\n| **Method**              | **Reward** | **Reward Improvement (%)** | **Time Cost** | **Time Cost Reduction (%)** |\n|-------------------------|------------|-----------------------------|---------------|-----------------------------|\n| HybridLLM               | 0.01       | -94.71                      | 273.45        | 49.57                       |\n| FrugalGPT               | 0.171      | -9.52                       | 63.15         | 88.35                       |\n| C2MAB-V                 | 0.189      | 0.00                        | 542.25        | 0.00                        |\n| GraphRouter (zero-shot) | 0.182      | -3.7                        | 1.00          | 99.82                       |\n| GraphRouter (few-shots) | 0.207      | 9.52                        | 3.00          | 99.45                       |\n| GraphRouter (Trained)   | 0.219      | 15.87                       | 30.00         | 94.47                       |\n\n**[Common prior knowledge]**. We agree that this prior knowledge could potentially enhance the LLM descriptions, thereby improving the performance of LLM selection. However, such prior knowledge is difficult to obtain or requires expert experience, thus limiting the method's automation. Moreover, with the emergence of a large number of new LLMs and updates to existing LLMs, acquiring and utilizing this prior knowledge becomes even more challenging. The core focus of our paper is to model the embeddings of LLMs with varying capabilities more effectively. This is achieved by utilizing the interaction data among tasks, queries, and LLMs, along with the message-passing capabilities of GNNs to capture and differentiate these embeddings. We believe that in the real scenario of massive LLM user interactions, this interaction data can enable GraphRouter to model the descriptions of LLMs more accurately and automatically."},"title":{"value":"Response to Reviewer pHun (3/5)"}},"tmdate":1732305320750,"tcdate":1732305320750,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission10977/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission10977/Authors"],"forum":"eU39PDsZtT","number":9,"license":"CC BY 4.0","cdate":1732305320750,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission10977/-/Official_Comment"],"mdate":1732305320750,"domain":"ICLR.cc/2025/Conference","replyto":"Oj5xoDO0yO","id":"GGeYyz25IZ","forumContent":{"TLDR":{"value":"We propose GraphRouter, a novel graph-based router that improves LLM selection for diverse tasks by leveraging an inductive graph framework to optimize performance-cost trade-offs and generalize across various models and scenarios."},"venue":{"value":"ICLR 2025 Poster"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["LLM selection","Graph-based router","Contextual interactions","New LLM settings"]},"primary_area":{"value":"learning on graphs and other geometries & topologies"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"The rapidly growing number and variety of Large Language Models (LLMs)\npresent significant challenges in efficiently selecting the appropriate LLM for\na given query, especially considering the trade-offs between performance and\ncomputational cost. Current LLM selection methods often struggle to generalize\nacross new LLMs and different tasks because of their limited ability to leverage\ncontextual interactions among tasks, queries, and LLMs, as well as their depen-\ndence on a transductive learning framework. To address these shortcomings, we\nintroduce a novel inductive graph framework, named as GraphRouter, which\nfully utilizes the contextual information among tasks, queries, and LLMs to en-\nhance the LLM selection process. GraphRouter constructs a heterogeneous\ngraph comprising task, query, and LLM nodes, with interactions represented as\nedges, which efficiently captures the contextual information between the query’s\nrequirements and the LLM’s capabilities. Through an innovative edge prediction\nmechanism, GraphRouter is able to predict attributes (the effect and cost of\nLLM response) of potential edges, allowing for optimized recommendations that\nadapt to both existing and newly introduced LLMs without requiring retraining.\nComprehensive experiments across three distinct effect-cost weight scenarios have\nshown that GraphRouter substantially surpasses existing routers, delivering a\nminimum performance improvement of 12.3%. In addition, it achieves enhanced\ngeneralization across new LLMs settings and supports diverse tasks with at least a\n9.5% boost in effect and a significant reduction in computational demands. This\nwork endeavors to apply a graph-based approach for the contextual and adaptive\nselection of LLMs, offering insights for real-world applications."},"_bibtex":{"value":"@inproceedings{\nfeng2025graphrouter,\ntitle={GraphRouter: A Graph-based Router for {LLM} Selections},\nauthor={Tao Feng and Yanzhen Shen and Jiaxuan You},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=eU39PDsZtT}\n}"},"title":{"value":"GraphRouter: A Graph-based Router for LLM Selections"},"pdf":{"value":"/pdf/2452b8ff98e5d19eedb55a812aabf2db832a2ec4.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference"},"paperhash":{"value":"feng|graphrouter_a_graphbased_router_for_llm_selections"},"authorids":{"value":["~Tao_Feng5","~Yanzhen_Shen1","~Jiaxuan_You2"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Tao Feng","Yanzhen Shen","Jiaxuan You"]}},"version":2},{"content":{"comment":{"value":"> The router is evaluated in a batch setting, so that it assumes all tasks may be known upfront. In real-world settings, the router makes per-task decisions without knowing the full distribution. Also, the binary \"easy/hard\" classification with constrained decoding is simplistic; a regression-based difficulty score might perform better, but no alternatives are explored or discussed.\n\n* We thank the reviewer for the suggestion.\n* **On regression-based difficulty scores:** As stated in Section 7.2, the router already outputs a continuous numerical difficulty score between 0 (easiest) and 1 (hardest) for each task \\- this is precisely the regression-based scoring the reviewer suggests.   \n* **On batch vs. per-task routing:** We agree that real-world deployment requires per-task routing decisions. We would like to clarify that our setup already supports this: since scores are numerical, an appropriate threshold $\\\\tau$ can be chosen such that any task scoring above $\\\\tau$ is routed to the large model.   \n  * Crucially, this per-task threshold setting and our batch setting are equivalent \\- they route the same tasks and achieve the same RCRPS: the \"fraction of tasks routed\" reported in Table 2 simply corresponds to exploring different percentile-based thresholds (20%, 40%, 60%, 80% quantiles of each router's score distribution), making the results directly transferable to the per-task setting.\n\n* We explore this further and find that score ranges vary meaningfully across router sizes: larger routers spread scores across their range (e.g., Qwen2.5-32B: \\[0.03, 0.39\\], Qwen2.5-72B: \\[0.05, 0.95\\]), while smaller routers collapse tasks into a narrow band (e.g., Qwen2.5-0.5B: \\[0.76, 0.84\\]). Once a router is chosen, calibrating $\\\\tau$ would require only a small held-out set of tasks to estimate the relevant percentile of that router's score distribution \\- a lightweight, one-time step identical to standard practice in confidence-based routing \\[1, 2\\]. \n \n\n* We have updated the paper to reflect this: Section 7.2 now clarifies that our setup supports both batch and per-task routing, and a new Appendix F.2 provides a dedicated discussion with score range analysis across router sizes (Figure F.2), showing that both settings yield equivalent results.\n\n\\[1\\] Chen, Lingjiao, Matei Zaharia, and James Zou. \"FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance.\" *Transactions on Machine Learning Research*.                  \n   \n\\[2\\] Ong, Isaac, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M. Waleed Kadous, and Ion Stoica. \"RouteLLM: Learning to Route LLMs from Preference Data.\" In *The Thirteenth International Conference on Learning Representations*.\n\n> Table 1 shows that CorDP performance varies depending on the base forecaster, but the paper provides limited discussion on how to select the right base forecaster.\n\n* We thank the reviewer for the question. As we discuss in the \"SampleWise vs Median variants\" paragraph of Sec 5.3, the performance of CorDP methods does differ depending on the base forecaster used; however, all winning methods except one use Lag-Llama as the base forecaster. This is because Lag-Llama is the best performing quantitative forecaster on this data, as shown in the base forecaster row of Table 1\\.  \n* This points to a clear and practical selection criterion: to choose the best available quantitative forecaster as the base model for CorDP. Crucially, this selection does not require any labeled context-aided forecasting data \\- one can simply evaluate candidate base forecasters on standard held-out no-context data (a routine step in any forecasting pipeline) and use the best performer. This is also practically convenient: in real deployments, practitioners typically already have a quantitative forecasting model in place, and CorDP is designed to augment it with contextual reasoning rather than replace it. We show that LLMs with CorDP are flexible enough to use forecasts from any base model and incorporate context on top.   \n* We have made this selection guideline explicit in Section 5.3 in the revised version of the paper."}},"parentInvitations":"TMLR/-/Official_Comment","tmdate":1777843067561,"tcdate":1777840529675,"writers":["TMLR","TMLR/Paper7655/Authors"],"signatures":["TMLR/Paper7655/Authors"],"forum":"dkjHHFJkVI","number":5,"license":"CC BY 4.0","cdate":1777840529675,"readers":["everyone"],"invitations":["TMLR/Paper7655/-/Official_Comment"],"mdate":1777843067561,"domain":"TMLR","replyto":"5MOhYgGkWU","id":"aDr0dAYNMx","forumContent":{"submission_length":{"value":"Regular submission (no more than 12 pages of main content)"},"venue":{"value":"Accepted by TMLR"},"code":{"value":"https://github.com/ashok-arjun/beyond-naive-prompting"},"abstract":{"value":"Real-world forecasting requires models to integrate not only historical data but also relevant contextual information provided in textual form. While large language models (LLMs) show promise for context-aided forecasting, critical challenges remain: we lack diagnostic tools to understand failure modes, performance remains far below their potential, and high computational costs limit practical deployment. We introduce a unified framework of four strategies that address these limitations along three orthogonal dimensions: model diagnostics, accuracy, and efficiency. Through extensive evaluation across model families from small open-source models to frontier models including Gemini, GPT, and Claude, we uncover both fundamental insights and practical solutions. Our findings span three key dimensions: diagnostic strategies reveal the “Execution Gap” where models correctly explain how context affects forecasts but fail to apply this reasoning; accuracy-focused strategies achieve substantial performance improvements of 25-50%; and efficiency-oriented approaches show that adaptive routing between small and large models can approach large model accuracy on average while significantly reducing inference costs. These orthogonal strategies can be flexibly integrated based on deployment constraints, providing practitioners with a comprehensive toolkit for practical LLM-based context-aided forecasting. Code is made available at \\url{https://github.com/ashok-arjun/beyond-naive-prompting}."},"_bibtex":{"value":"@article{\nashok2026beyond,\ntitle={Beyond Na{\\\"\\i}ve Prompting: Strategies for Improved Context-aided Forecasting with {LLM}s},\nauthor={Arjun Ashok and Andrew Robert Williams and Vincent Zhihao Zheng and Irina Rish and Nicolas Chapados and {\\'E}tienne Marcotte and Valentina Zantedeschi and Alexandre Drouin},\njournal={Transactions on Machine Learning Research},\nissn={2835-8856},\nyear={2026},\nurl={https://openreview.net/forum?id=dkjHHFJkVI},\nnote={}\n}"},"title":{"value":"Beyond Naïve Prompting: Strategies for Improved Context-aided Forecasting with LLMs"},"changes_since_last_submission":{"value":"# Changes post-rebuttal (May 2026):                       \n                                              \nAll post-rebuttal changes appear in blue text in the PDF. \n\n  - CorDP (Sec 5.3). Promoted the task-type finding into an explicit practitioner selection criterion (SampleWise-CorDP for partial-RoI tasks; Median-CorDP for full-shape/constrained tasks).                  \n  - CorDP (Sec 5.2). Clarified that Median-CorDP and SampleWise-CorDP are mechanistically distinct and complementary, not combinable.                                                                            \n  - RouteDP (Table 2). Added a note that percentage labels are nominal and actual task counts are floored.  \n  - RouteDP (Sec 7.2 + App F.2). Clarified that the setup supports per-task routing via a score threshold, equivalent to the batch setting; added score-range analysis across router sizes.                          \n  - FxDP (Sec 4.1).  Added a note that without FxDP, both steps are performed implicitly in a single forward pass, and that FxDP externalizes this process to make intermediate failures diagnosable.    \n  - FxDP (Sec 4.3). Replaced Fig 3 with an improved version: the Execution Gap segment is now highlighted in orange (previously gray) to draw attention to the central finding, and bracket annotations below the x-axis explicitly group models and label the two key results. \n  - FxDP (Sec 4.3). Added the evaluation reliability finding (no incorrect explanation leads to an improved forecast) in the results section.    \n  - Discussion and Future Work (Sec 9). Expanded on the connections between CorDP and tool-use agents, emphasized on the need for more unconstrained benchmarks, and elaborated on future directions related to finetuning.\n\n# Initial submission (Feb 2026):                       \n\nIn this submission we address all major concerns from the previous submission through new human evaluation studies that validate our ground truth and confirm agreement between our LLM judges and human annotators, a unified presentation of our strategies, and updated results with recent frontier models.\n\n## Major changes\n\n* **FxDP: Ground truth forecast effects.** We ran a human evaluation on the Prolific platform to verify the ground truth forecast effects. Each task was evaluated by five independent annotators; we aggregated responses by majority vote and used attention checks to ensure annotation quality. We obtained at least 4 out of 5 annotator agreement on all tasks and 80–100% agreement per item, thereby validating the gold standard. Further details are given in Appendix C.2.\n\n* **FxDP: LLM judges.** We compare models’ forecast effects to the ground truth using a panel of LLM judges (GPT-5.2, Claude-Sonnet-4.5, Gemini-2.5-Pro). All pairs of judges show high agreement, which indicates that the evaluation protocol provides clear, consistent, and objective criteria.\n\n* **FxDP: Human–LLM alignment.** To verify that LLM judge results align with human judgment, we designed a human evaluation on a subset of items. Five independent annotators per item performed the same comparison task as the judges, with majority-vote aggregation and attention checks, following standard annotation practice. Agreement between human evaluators and the LLM judges is 91.7–93.3% (Cohen’s κ = 0.81–0.84), with high inter-annotator and inter-judge agreement; see Appendix C.3. We open-source all evaluation artifacts, and our original findings remain unchanged.\n\n* **FxDP: Threshold analysis.** We use a 50% relative improvement threshold in RoI as the primary criterion and report sensitivity analyses at 30%–80% in Appendix C.4. We show that our findings hold across this range of thresholds.\n\n* **FxDP, CorDP, IC-DP.** We updated results for GPT-5.2, Gemini-2.5-Pro, and Claude-Sonnet-4.5. We also report costs and inference times for all of these models.\n\n* **Unification.** We present a unified view of our proposed strategies in a new Figure 1 and report results with multiple strategies combined.\n\n* **Supplementary material.** We provide code for all proposed methods.\n\n## Minor changes\n\n* **Renaming ReDP to FxDP:** We renamed ReDP (Direct Prompting with Reasoning over Context) to FxDP (Direct Prompt with Forecast Effect Explanation) to reflect our focus on the model’s perceived effect of the context on the forecast rather than full step-by-step reasoning traces. We now call the outputs “forecast effect explanations.” The methodology and main results are unchanged.\n\n* **FxDP:** We add Figure 2, which summarizes the major results, and add the full results table in the appendix."},"pdf":{"value":"/pdf/963cd6cbcfd30ca6fbab8f051d1c3bbd6dd3c144.pdf"},"venueid":{"value":"TMLR"},"paperhash":{"value":"ashok|beyond_naïve_prompting_strategies_for_improved_contextaided_forecasting_with_llms"},"authorids":{"value":["~Arjun_Ashok1","~Andrew_Robert_Williams1","~Vincent_Zhihao_Zheng1","~Irina_Rish1","~Nicolas_Chapados1","~Étienne_Marcotte1","~Valentina_Zantedeschi2","~Alexandre_Drouin2"]},"previous_TMLR_submission_url":{"value":"https://openreview.net/forum?id=CLuE0GL5xE"},"assigned_action_editor":{"value":"~Zachary_B._Charles1"},"authors":{"value":["Arjun Ashok","Andrew Robert Williams","Vincent Zhihao Zheng","Irina Rish","Nicolas Chapados","Étienne Marcotte","Valentina Zantedeschi","Alexandre Drouin"]}},"version":2},{"content":{"comment":{"value":"Dear Reviewer cQJE,\n\nWe sincerely appreciate the time and effort you devoted to reviewing our manuscript and for providing insightful comments and suggestions. Below, we provide detailed responses to your concerns and questions.\n\n## The latency of the proposed method\n>The latency is not reported. \n\nThe SDR method introduced in this work is primarily designed to optimize the trade-off between monetary cost and task performance. The lightweight design is intended to ensure minimal additional deployment overhead and thereby preserve the practical applicability of the method. It is **not specifically aimed at optimizing latency**, which is why the current version of the manuscript does not include latency-focused analyses or experiments. \n\nThat said, we acknowledge that incorporating latency evaluations in future revisions would further strengthen the practical value of our approach, and we thank you for pointing out this aspect.\n\n## The idea of performing routing earlier\n>Judge after SLM finishes generation can take very long. After SLM generates a few tokens, it might be sufficient to start routing.\n\nWe agree that this is a valuable idea, but it introduces new challenges and may drift away from the core research question of the cost–performance trade-off. A typical example can help illustrate this: current LLMs often exhibit extensive **reflective–refining** behavior during reasoning. And you have likely observed cases where they repeatedly output phrases such as \"Wait, let’s see......\" before revising their answers. If routing decisions were made immediately after the SLM produces an initial, unrefined draft, the reasonable behavior would likely be routing to the remote LLM prematurely, even in cases where the local SLM might eventually reach a correct answer on its own. \n\nConsequently, exploring this direction would likely lead to a new trade-off between latency and performance and possibly require additional designs regarding to SLM behavior. Overall, while this line of inquiry deviates from our original goal of optimizing the cost–performance trade-off, it indeed represents a valuable direction for future extension. We sincerely appreciate you sharing this insightful thought with us.\n\n## Latency & cost versus finetuning a BERT model\n>Fine tuning a BERT before SLM and LLMs may have lower latency and cost.\n\nAs described in Section 3.3, the decider module in our method is constructed primarily by reusing one transformer layer from the SLM. With a 7B-scale SLM, the additional module contains approximately 200M parameters, **comparable to the scale of BERT**. Moreover, the module can operate partially in parallel with the original SLM, so qualitatively it will not introduce greater latency than a BERT-based approach.\n\nConcerning training cost, a straightforward implementation indeed requires running the original SLM online to obtain hidden states, which requires memory to store the SLM parameters. However, since our method does not update the SLM parameters, no additional memory is needed for activations or similar information of SLM. Furthermore, if sufficient disk storage is available, one can precompute and store the hidden states offline and simply load them during training, thus avoiding online SLM inference entirely.\n\n## The experimental results on GSM8K\nThe relatively small advantage over FrugalGPT on GSM8K is explained in Section 4.2 of the manuscript. This is due to the near-saturated performance of LLMs on this dataset, under which the **new training objective we propose becomes intrinsically similar to those used in prior work**. The reason for using Llama-4 as the LLM for this dataset is also stated in the first sentence of Section 4.2: this model achieves the strongest overall performance on GSM8K. This decision was made due to space limits; however, we acknowledge that adding the experimental results for all remaining LLMs to the appendix would enhance the completeness of the paper.\n\nFinally, we once again thank you for your thoughtful review and valuable suggestions. We hope that our responses have satisfactorily addressed all of your concerns.\n\nSincerely,\nAll authors"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764311897990,"tcdate":1764311897990,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission6798/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission6798/Authors"],"forum":"uSMttGgyuZ","number":1,"license":"CC BY 4.0","cdate":1764311897990,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission6798/-/Official_Comment"],"mdate":1764311897990,"domain":"ICLR.cc/2026/Conference","replyto":"r74htWv0HA","id":"QZZJwWCLgr","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"TLDR":{"value":"To achieve high performance with limited monetary cost with collaboration of local SLMs and remote LLMs."},"keywords":{"value":["LLM Routing","Cost-Efficiency","Local-Remote Collaboration"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"The rapid advancement of large language models (LLMs) has led to remarkable performance across diverse domains such as question answering, creative writing, programming, etc., making them indispensable assistants in daily life and work. Currently, LLM services are primarily accessed in two ways: (i) paid access to cloud-hosted LLMs, which are powerful but introduce nontrivial cost; and (ii) deployment of small language models (SLMs) on personal devices or small clusters, which, while less powerful, are sufficient for handling relatively simple tasks. To achieve a balanced trade-off between monetary cost and task performance, we propose Selective Deferred Routing, a paradigm that enables cost-efficient collaboration between local SLMs and remote LLMs. In this framework, a user request is first processed by the local SLM, which not only generates a preliminary response but also provides rich semantic representations of the request. A lightweight decider module then leverages this information to either adopt the initial response or route the request in a single step to the most suitable remote LLM for a higher-quality response. Extensive experiments on 5 LLMs and 3 datasets demonstrate that our approach consistently outperforms existing multi-LLM collaboration methods across different cost–performance trade-off preferences."},"_bibtex":{"value":"@misc{\nmiao2026selective,\ntitle={Selective Deferred Routing: Enabling Cost-Efficient Collaboration between Local {SLM}s and Remote {LLM}s},\nauthor={Qijun Miao and Zhixuan Fang},\nyear={2026},\nurl={https://openreview.net/forum?id=uSMttGgyuZ}\n}"},"title":{"value":"Selective Deferred Routing: Enabling Cost-Efficient Collaboration between Local SLMs and Remote LLMs"},"pdf":{"value":"/pdf/53c64af1a280cc8a3b45cf0c65c03f4c04ee44ec.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"miao|selective_deferred_routing_enabling_costefficient_collaboration_between_local_slms_and_remote_llms"},"authorids":{"value":["~Qijun_Miao1","~Zhixuan_Fang1"]},"authors":{"value":["Qijun Miao","Zhixuan Fang"]}},"version":2},{"content":{"comment":{"value":"***Q1. As the heterogeneous graph is constructed directly from the interaction history, it should not contain more information than the history itself. Could the authors elaborate on why such a graph-based representation improves preference elicitation compared to directly learning from the raw interaction table?***\n\n**Author Response:**\n\nWe thank the reviewer for raising the question regarding the advantages of a graph-based representation. While the topological structure of the graph is indeed derived from the interaction history, **our graph-based approach provides superior preference elicitation because it explicitly models the relationships among key entities in the interactions.**\n\nGMTRouter extracts the key entities from the interaction history and carefully designs the graph structure to capture the rich relational patterns within the interactions. Through message passing, each node explicitly aggregates information from its related nodes, dynamically refining its representation. **The graph structure imposes a strong relational inductive bias, allowing the model to capture user preferences more effectively than treating interaction rows independently.**\n\nTo further ground this discussion, **we selected TIGER [5], a recent method that learns directly from raw interaction data, and RouteLLM [6] as new baselines**. The experimental results are shown below:\n\n**Table of TIGER & GMTRouter performance over datasets**\n\n| Model | Chatbot-Arena | MT-Bench | GSM8K | MMLU | LaMP |\n|--------|:--------:|:--------:|:--------:|:--------:|:--------:|\n|        | ACC / AUC | ACC / AUC | ACC / AUC | ACC / AUC | ACC / AUC |\n| RouteLLM   | 0.492 / 0.485 | 0.480 / 0.475 | 0.499 / 0.498 | 0.532 / 0.513 | - |\n| TIGER  | 0.739 / 0.735 | 0.656 / 0.691 | 0.639 / 0.683 | 0.710 / 0.764 | 0.339 / 0.698 |\n| GMTRouter  | 0.774 / 0.875 | 0.784 / 0.859 | 0.773 / 0.859 | 0.771 / 0.870 | 0.349 / 0.662 |\n\n*Note that RouteLLM is designed for routing between two LLMs, whereas the LaMP dataset requires routing among five LLMs; therefore, RouteLLM was not evaluated on LaMP.*\n\nMoreover, as reported in ***Table 2***, we have included several non-graph-based baselines (e.g FrugalGPT, Personalized LLM) as comparisons.\nThe results confirm that representing the interaction history as a heterogeneous graph enables GMTRouter to achieve more effective personalization compared to learning directly from raw data. These empirical results complement the conceptual argument: the graph structure imposes a relational inductive bias that  encourages the model to exploit dependencies across users, queries, responses, and LLMs, rather than treating interaction rows as independent.\n\n***Q2. The graph construction includes multiple users within the same graph (e.g., Figure 3(b)). However, user preferences are typically independent of one another. Would constructing individual user-specific graphs (and learning over them) be more intuitive? How does the shared multi-user graph influence personalization performance?***\n\n**Author Response:**\n\nWe thank the reviewer for this insightful question. As the reviewer correctly notes, user preferences are diverse, and signals from different users can interfere with one another. \nAlthough we conceptually construct a global interaction graph (Fig. 3(b)), **message passing is always performed on per-user subgraphs**:\n- During training ***[Line 248-251]***, each sampled subgraph only contains interactions from a single user.\n- During inference ***[Line 273-277]***, we also build a subgraph using only that user's history and candidate LLMs.\n\nThus, in practice, GMTRouter operates on **individual user-specific graphs**, and there is no cross-user message passing; the global graph is simply the union of these user-specific components for convenience of storage and indexing. Personalization arises from user-specific node representations together with parameter sharing across users.\nWe have clarified these details in the paper and will update ***Figure 3*** in future revisions to better illustrate the design. We, once again, thank the reviewer for the insightful comments to help us better polish our work. \n\n**[5]** Rajput, Shashank, et al. \"Recommender systems with generative retrieval.\" Advances in Neural Information Processing Systems 36 (2023): 10299-10315.\n\n**[6]** Ong, Isaac, et al. \"Routellm: Learning to route llms with preference data.\" arXiv preprint arXiv:2406.18665 (2024)."},"title":{"value":"Author response to Reviewer 8ann (3/3)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764046878381,"tcdate":1764046571670,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission14657/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission14657/Authors"],"forum":"oRe9tHpSpn","number":9,"license":"CC BY 4.0","cdate":1764046571670,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission14657/-/Official_Comment"],"mdate":1764046878381,"domain":"ICLR.cc/2026/Conference","replyto":"ryB9mqPWFS","id":"9vTMaudVNw","forumContent":{"venue":{"value":"ICLR 2026 Conference Withdrawn Submission"},"keywords":{"value":["Heterogeneous GNN","LLM routing","Preference Learning"]},"primary_area":{"value":"learning on graphs and other geometries & topologies"},"abstract":{"value":"Large Language Model (LLM) routing has demonstrated strong capability in balancing response quality with computational cost. As users exhibit diverse preferences, personalization has attracted increasing attention in LLM routing, since even identical queries may require different models to generate responses tailored to individual needs. However, existing approaches are not fully personalized and often fail to faithfully capture the complex interactions between specific users and LLMs. Moreover, user preference data is typically scarce, noisy, and inconsistent in format, which limits the effectiveness of methods that rely solely on user-specific data. To address these challenges, we propose GMTRouter, which represents multi-turn user–LLM interactions as a heterogeneous graph with four node types: user, LLM, query, and response, thereby maximally preserving the rich relational structure of the interaction. Through a tailored message-passing mechanism, GMTRouter learns to capture user preferences from few-shot data within a lightweight inductive graph learning framework, enabling effective personalization. Extensive experiments demonstrate that GMTRouter consistently outperforms the strongest baselines, achieving 0.9%–21.6% higher accuracy and 0.006–0.309 higher AUC across multiple datasets. More importantly, we further demonstrate that GMTRouter can adapt to new users and evolving preferences using only few-shot data, without extensive fine-tuning."},"_bibtex":{"value":"@misc{\nxie2026gmtrouter,\ntitle={{GMTR}outer: Personalized {LLM} Router over Multi-turn User Interactions},\nauthor={Encheng Xie and Yihang Sun and Tao Feng and Jiaxuan You},\nyear={2026},\nurl={https://openreview.net/forum?id=oRe9tHpSpn}\n}"},"title":{"value":"GMTRouter: Personalized LLM Router over Multi-turn User Interactions"},"pdf":{"value":"/pdf/b4f227b8ee59ab4f2a14d8fc028c0e06e9455c44.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Withdrawn_Submission"},"paperhash":{"value":"xie|gmtrouter_personalized_llm_router_over_multiturn_user_interactions"},"authorids":{"value":["~Encheng_Xie1","~Yihang_Sun2","~Tao_Feng5","~Jiaxuan_You2"]},"authors":{"value":["Encheng Xie","Yihang Sun","Tao Feng","Jiaxuan You"]}},"version":2},{"content":{"comment":{"value":"Thank you for your comments and for appreciating our technical contributions. We have made the following updates and clarifications to address your remaining concerns.\n\n---\n\n1. **Literature Review on Bandits with Long-Term Constraints**:\n\nWe have now added a **formal review of relevant works on Bandits with Constraints** in **Appendix A.2**, which includes a comparison table with existing works on **bandits with long-term constraints**. While prior works use a combination of UCB/LCB, they are typically restricted to **linear CMAB settings** with **semi-bandit feedback**. In contrast, our framework extends this approach to handle **non-linear reward functions** through different **relaxation and rounding techniques**, making it applicable to broader scenarios.\n\n---\n\n2. **Clarification on Assumptions for Reward and Cost Functions**:\n\nWe would like to clarify that we **do not** make prior assumptions about the **cost** and **reward functions** of individual LLMs; instead, they are learned through **online feedback**. The **overall reward definitions for multi-LLM tasks** (AWC, SUC, AIC) are drawn from common multi-LLM collaboration scenarios and are validated by existing literature [1, 2, 3].\n\nThe **L-Lipschitz continuity** is **not assumed** but **derived as an intrinsic property** of our reward models. Specifically, we establish that all three reward models are **1-Lipschitz continuous** with respect to LLM performance $\\boldsymbol{\\mu}$.\n\n- *AIC*: \n\n$ |r(S;\\boldsymbol{\\mu})-r(S;\\boldsymbol{\\mu}')|=|\\prod\\_{k \\in S} \\mu\\_k - \\prod\\_{k \\in S} \\mu'\\_k|= |\\sum\\_{k=1}^K (\\prod\\_{j=1}^{k-1}{\\mu\\_{j}}) (\\boldsymbol{\\mu}\\_{k} - \\mu'\\_{k})(\\prod\\_{j=k+1}^K\\mu\\_{j})| \n\\le  \\sum\\_{k=1}^K | (\\prod\\_{j=1}^{k-1}{\\mu'\\_{j}}) (\\boldsymbol{\\mu}\\_{k} - \\mu'\\_{k})(\\prod_{j=k+1}^K \\mu\\_{j})|\n\\le  \\sum\\_{k=1}^K |\\mu\\_{k} - \\mu'\\_{k}|.\n$\n\n- *SUC*: \n\n$|r(S;\\boldsymbol{\\mu})-r(S;\\boldsymbol{\\mu}')|=|\\sum_{k \\in S} \\mu_{k} - \\sum_{k \\in S} \\mu_{k}'| \\le  \\sum_{k=1}^K |\\mu_{k} - \\mu'_{k}|$.\n\n- *AWC*:\n\n$|r(S;\\boldsymbol{\\mu})-r(S;\\boldsymbol{\\mu}')|=|\\prod_{k \\in S} (1-\\mu_k)) - \\prod_{k \\in S} (1-\\mu_k)')|$, and we can follow the same derivation as AIC by introducing a new variable $\\lambda = \\mu$.\n\n\n Thus, our theoretical analysis extends to the more general **L-Lipschitz setting**. We have clarified these in  **Section 5** and **Appendix C.2**. Beyond the contribution of solving the multi-LLM selection problem, our general theoretical analysis itself is novel and of independent interest. \n\n---\n\n3. **Exploration in LLM Selection**:\n\nRegarding the **exploration** aspect of LLM selection, it is not \"unreliable\" as suggested. Rather, our exploration strategy is **carefully designed**, relying on **Lemma 1** to ensure **high-probability** events where empirical estimates for both rewards and costs align closely with their true means. Without this exploration component, the approach would **reduce to a greedy algorithm**, which may be unreliable and could lead to **sub-optimal** choices without considering potentially better LLMs.\n\n---\n\n4. **Contextual Bandit for Human Preferences**:\n\nOur current implementation focuses on a **non-contextual setup**, but we recognize the potential of extending our framework to include **contextual combinatorial multi-armed bandits** to better capture human preferences. We have highlighted our intention to explore this extension in **Section 7** as part of future research directions.\n\n---\n\n[1]. Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. Dynamic llm-agent network: An llm-agent collaboration framework with agent team optimization. *arXiv preprint arXiv:2310.02170*, 2023c.\n\n[2]. Lingjiao Chen, Matei Zaharia, and James Zou. Frugalgpt: How to use large language models while reducing cost and improving performance. *arXiv preprint arXiv:2305.05176*, 2023.\n\n\n[3]. Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al. Metagpt: Meta programming for multi-agent collaborative framework. *arXiv preprint arXiv:2308.00352*, 2023."},"title":{"value":"Response to Remaining Concerns"}},"tmdate":1732446216109,"tcdate":1732446216109,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission9524/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission9524/Authors"],"forum":"JLDAWbzTUg","number":12,"license":"CC BY 4.0","cdate":1732446216109,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission9524/-/Official_Comment"],"mdate":1732446216109,"domain":"ICLR.cc/2025/Conference","replyto":"WtPh4zm2bi","id":"L0gfZxaCYJ","forumContent":{"TLDR":{"value":"We introduce \\textit{C2MAB-V}, a cost-effective combinatorial multi-armed bandit system with versatile reward models for multi-LLM selection, balancing performance and cost-efficiency across multiple models and tasks."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["Combinatorial multi-armed bandit","Online learning","Large language model","Long-term cost constraint"]},"supplementary_material":{"value":"/attachment/b977437053394fef5e4fe6bd4700478e686b01d4.zip"},"primary_area":{"value":"reinforcement learning"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"With the rapid advancement of large language models (LLMs),  the diversity of multi-LLM tasks and the variability in their pricing structures have become increasingly important, as costs can vary greatly between different LLMs. To tackle these challenges, we introduce the \\textit{C2MAB-V}, a \\underline{C}ost-effective \\underline{C}ombinatorial \\underline{M}ulti-armed \\underline{B}andit with \\underline{V}ersatile reward models for optimal LLM selection and usage.  This online model differs from traditional static approaches or those reliant on a single LLM without cost consideration. With multiple LLMs deployed on a scheduling cloud and a local server dedicated to handling user queries, \\textit{C2MAB-V} facilitates the selection of multiple LLMs over a combinatorial search space, specifically tailored for various collaborative task types with different reward models. Based on our designed online feedback mechanism and confidence bound technique, \\textit{C2MAB-V} can effectively address the multi-LLM selection challenge by managing the exploration-exploitation trade-off across different models, while also balancing cost and reward for diverse tasks. The NP-hard integer linear programming problem for selecting multiple LLMs with trade-off dilemmas is addressed by: i) decomposing the integer problem into a relaxed form by the local server, ii) utilizing a discretization rounding scheme that provides optimal LLM combinations by the scheduling cloud, and iii) continual online updates based on feedback. Theoretically, we prove that \\textit{C2MAB-V} offers strict guarantees over versatile reward models,  matching state-of-the-art results for regret and violations in some degenerate cases. Empirically, we show that \\textit{C2MAB-V} effectively balances performance and cost-efficiency with nine LLMs for three application scenarios."},"_bibtex":{"value":"@misc{\ndai2025costeffective,\ntitle={Cost-Effective Online Multi-{LLM} Selection with Versatile Reward Models},\nauthor={Xiangxiang Dai and Jin Li and Xutong Liu and Anqi Yu and John C.S. Lui},\nyear={2025},\nurl={https://openreview.net/forum?id=JLDAWbzTUg}\n}"},"title":{"value":"Cost-Effective Online Multi-LLM Selection with Versatile Reward Models"},"pdf":{"value":"/pdf/2153c724d07868387e8d78fe42646a0f28fc8740.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"dai|costeffective_online_multillm_selection_with_versatile_reward_models"},"authorids":{"value":["~Xiangxiang_Dai2","~Jin_Li22","~Xutong_Liu1","~Anqi_Yu1","~John_C.S._Lui2"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Xiangxiang Dai","Jin Li","Xutong Liu","Anqi Yu","John C.S. Lui"]}},"version":2},{"content":{"comment":{"value":"> \"removing more than 30% of weights leads to significant (> 5%, relative) accuracy degradation on fact recall related tasks (Figure 1, left). Fact recall suffers similarly from dense down-scaling.\" <- Please have some text which segways from the first paragraph (page 2) to the list of 3 bold-faced items. The intro currently reads like a collection of text/paragraphs which do not blend into one another. ...\n\nThank you for your suggestions! We applied your suggested comments; since this involves substantial change in our intro text, we expect to continue updating and improving it during the discussion period.\n\n> Same comment for italicized-starting-text, which proceed bold-starting-text; please segway the various paragraphs together. It is very difficult for a reader to understand the point that is trying to be made when sentences exist independently.\n\nUnlike the previous point, these individual items are indeed discussing largely independent topics. We feel like they may be better left as independent paragraphs.\n\n>-Why did the authors not consider the GINC dataset for ICL, from Xie et al's \"An Explanation of In-context Learning as Implicit Bayesian Inference?\"\n\nGINC dataset requires pre-training on the synthetically generated pre-training dataset for the evaluation task to make sense. The task is purely synthetic, a model pre-trained on natural language corpus cannot not solve it. Since our focus is on evaluating pre-trained LLMs, this task does not appear to suit our needs. Please see the eval dataset sample here [9]. \n\n> -\"From the OPT family, we evaluate the two largest models that fit in our hardware setup\" <- Please state the hardware setup\n\nOur hardware setup is available in appendix J. All our evaluations are performed on TPU v3-8; each core of TPU v3 has 16GB HBM memory. We used batch size 1 and additional tricks such as CPU-offloading to fit larger models inside TPUs.  \n\n> -In Table 1, please define what is meant by \"Context Type\"\n\nIt means what is the type of supporting evidence we insert into the model’s context to help the model solve the task. Empty means no evidence provided, and is used in closebook QA evaluation. Factual means we provide a paragraph of supporting evidence containing the factual answer to the question, and is used in openbook QA evaluation. Synthetic means we provide a paragraph of evidence containing the counterfactual answer to the question, and is used in counterfactual QA evaluation. Exemplary means we provide in-context examples to guide models to learn parametric functions in-context.\n\n[1] SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot, Frantar et al., ICML 2023.\n\n[2] A Simple and Effective Pruning Approach for Large Language Models, Sun et al., arXiv 2023.\n\n[3] LLM-Pruner: On the Structural Pruning of Large Language Models, Ma et al., NeurIPS 2023.\n\n[4] LLaMA: Open and Efficient Foundation Language Models, Touvron et al., arXiv 2023.\n\n[5] Pythia eval code: https://github.com/EleutherAI/lm-evaluation-harness/blob/master/lm_eval/tasks/triviaqa.py\n\n[6] OPT: Open Pre-trained Transformer Language Models, Zhang et al., arXiv 2022.\n\n[7] Language Models are Few-Shot Learners, Brown et al., NeurIPS 2020.\n\n[8] FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance, Chen et al., arXiv, 2023.\n\n[9] GINC sample eval data https://raw.githubusercontent.com/p-lambda/incontext-learning/main/data/GINC_trans0.1_start10.0_nsymbols50_nvalues10_nslots10_vic0.9_nhmms10/id_prompts_randomsample_3.json\n\n[10] A Survey on In-context Learning, Dong et al., arXiv 2022.\n\n[11] OpenICL: An Open-Source Framework for In-context Learning, Wu et al., arXiv 2023.\n\n[12] Entity-Based Knowledge Conflicts in Question Answering, Longpre et al., ACL 2021.\n\n[13] DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answering, Neeman et al., arXiv 2022."},"title":{"value":"Response -- Presentation & Questions"}},"tmdate":1700204034604,"tcdate":1700204034604,"writers":["ICLR.cc/2024/Conference","ICLR.cc/2024/Conference/Submission3569/Authors"],"signatures":["ICLR.cc/2024/Conference/Submission3569/Authors"],"forum":"ldJXXxPE0L","number":8,"license":"CC BY 4.0","cdate":1700204034604,"readers":["everyone"],"invitations":["ICLR.cc/2024/Conference/Submission3569/-/Official_Comment"],"mdate":1700204034604,"domain":"ICLR.cc/2024/Conference","replyto":"LoISLMvuOK","id":"T5qa2B9fq8","forumContent":{"venue":{"value":"ICLR 2024 poster"},"TLDR":{"value":"Moderate down-scaling harms fact recall, and yet the ability to learn from a few input-output examples from context withstands aggressive down-scaling."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["large language model","scaling","in-context learning","pruning"]},"primary_area":{"value":"generative models"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide."},"abstract":{"value":"We study how down-scaling large language model (LLM) size impacts LLM capabilities. We begin by measuring the effects of weight pruning – a popular technique for reducing model size – on the two abilities of LLMs: (a) recalling facts presented during pre-training and (b) processing information presented in context. Surprisingly, we find that existing pruning techniques affect these two abilities of LLMs differently. For example, pruning more than 30% of weights significantly decreases an LLM’s ability to recall facts presented during pre-training. Yet pruning 60-70% of weights largely preserves an LLM’s ability to process information in-context, ranging from retrieving answers based on information presented in context to learning parameterized functions such as a linear classifier based on a few examples. Moderate pruning impairs LLM’s ability to recall facts learnt from pre-training. However, its effect on model’s ability to process information presented in context is much less pronounced. The said disparate effects similarly arise when replacing the original model with a smaller dense one with reduced width and depth. This similarity suggests that model size reduction in general underpins the said disparity."},"_bibtex":{"value":"@inproceedings{\njin2024the,\ntitle={The Cost of Scaling Down Large Language Models: Reducing Model Size Affects Memory before In-context Learning},\nauthor={Tian Jin and Nolan Clement and Xin Dong and Vaishnavh Nagarajan and Michael Carbin and Jonathan Ragan-Kelley and Gintare Karolina Dziugaite},\nbooktitle={The Twelfth International Conference on Learning Representations},\nyear={2024},\nurl={https://openreview.net/forum?id=ldJXXxPE0L}\n}"},"title":{"value":"The Cost of Scaling Down Large Language Models: Reducing Model Size Affects Memory before In-context Learning"},"pdf":{"value":"/pdf/6a6c078824f86b9c8353c38a86794c78253b804f.pdf"},"venueid":{"value":"ICLR.cc/2024/Conference"},"paperhash":{"value":"jin|the_cost_of_scaling_down_large_language_models_reducing_model_size_affects_memory_before_incontext_learning"},"authorids":{"value":["~Tian_Jin1","~Nolan_Clement1","~Xin_Dong1","~Vaishnavh_Nagarajan3","~Michael_Carbin1","~Jonathan_Ragan-Kelley1","~Gintare_Karolina_Dziugaite1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors' identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Tian Jin","Nolan Clement","Xin Dong","Vaishnavh Nagarajan","Michael Carbin","Jonathan Ragan-Kelley","Gintare Karolina Dziugaite"]}},"version":2},{"content":{"comment":{"value":"We thank the reviewer for the follow-up questions. \n\n**KQ1.** We identified this phenomenon during our algorithm development, and it directly motivated the hint size threshold $\\\\eta_{\\text{hint}}$ in our policy design (Section 3.4). The gap at large hint sizes is structural: even with a near-complete LLM response as a prefix, the SLM generates a response that may still produce errors – wasting the LLM cost while failing to match LLM accuracy.\nOur framework addresses this with a three-way decision: (1) no-hint → SLM only, (2) predicted hint size ≤ $\\\\eta_{\\text{hint}}$ → shepherding, (3) predicted hint > $\\\\eta_{\\text{hint}}$ → route directly to LLM, bypassing the SLM entirely. The threshold $\\\\eta_{\\text{hint}}​$ is tuned per dataset (Table 5, Appendix F): GSM8K = 58, CNK12 = 60, HumanEval = 110, MBPP = 130 tokens. This ensures shepherding operates only in the regime where partial hints are most cost-effective, while queries requiring near-complete LLM reasoning are handled by the LLM at full accuracy. The reviewer's observation about the turning point is precisely what $\\\\eta_{\\text{hint}}$ captures.\n\n**KQ2.** Thank you for this important clarification – we now better understand the reviewer's concern about possible alternative SLM-LLM collaboration architectures that could achieve similar cost-accuracy trade-offs. We address each suggestion.\n\n*On DEER / early answer inducing.* We appreciate this suggestion and the analogy between increasing accuracy with hint size in our paper and increasing accuracy with exit positions in Figure 2 of DEER (https://openreview.net/pdf?id=NpU7ZXafRi). However, we highlight two important differences that make direct comparison challenging.\n\nFirst, DEER requires fine-grained control over the LLM's generation process: it must interrupt generation mid-stream at specific token positions, inject an answer-inducing prompt, evaluate confidence via logprobs, and conditionally resume or roll back. Standard LLM APIs – including Groq – treat generation as a single atomic call with no mechanism to pause, inspect, inject, and conditionally continue. While some APIs expose logprobs (e.g., OpenAI's logprobs parameter), the mid-stream interruption and injection mechanism DEER relies on is not supported by standard LLM APIs. Shepherding, by contrast, requires only the standard max_new_tokens parameter available in every LLM API, making it compatible with any provider.\n\nSecond, DEER increases LLM compute overhead rather than reducing it. At every candidate exit point, DEER performs additional forward passes to generate trial answers and evaluate confidence (DEER-Pro uses N=4 passes per exit point). The savings come from stopping the main CoT early, not from fewer model calls. Shepherding makes at most one LLM API call per query, making the cost structures fundamentally different.\n\nThe reviewer’s question about where the turning point lies – when it is cheaper to let the LLM finish rather than hand off to the SLM – is precisely what our threshold $\\\\eta_{\\text{hint}}$ addresses. \n\n*On SLM CoT → LLM final answer.* This is a legitimate alternative architecture that exploits the input/output price differential. However, prior work by [Yue et al.,2024] shows that passing weaker-model outputs as context to an LLM can degrade performance, particularly when the SLM reasoning contains errors that mislead the LLM. Our approach avoids this failure mode entirely: the hint flows from the LLM to the SLM, ensuring the reasoning prefix is always high-quality.\n\nWe did not intend to suggest that no other SLM-LLM combinations exist. Nevertheless, within the routing/cascading paradigm, our baselines (RouteLLM, GraphRouter, FrugalGPT, ABC) are representative of the state of the art. We totally agree that the design space of SLM-LLM collaboration is broader. Yet, different paradigms like DEER and Speculative Decoding often operate with different structures or optimize for disparate objectives – making a head-to-head benchmark inherently challenging. \n\n*[Yue et al.,2024] Yue, Murong, Jie Zhao, Min Zhang, Liang Du, and Ziyu Yao. \"Large Language Model Cascades with Mixture of Thought Representations for Cost-Efficient Reasoning.\" in Proc. ICLR, 2024.\nhttps://arxiv.org/abs/2310.03094*\n\nWe will add a discussion of DEER, the ICLR 2024 paper, and similar approaches to the related work section of the revised paper."}},"parentInvitations":"ICML.cc/2026/Conference/-/Reply_Rebuttal_Comment","tmdate":1782369956852,"tcdate":1775235878795,"writers":["ICML.cc/2026/Conference","ICML.cc/2026/Conference/Submission30175/Authors"],"signatures":["ICML.cc/2026/Conference/Submission30175/Authors"],"forum":"H8rywbFXuM","number":1,"license":"CC BY 4.0","cdate":1775235878795,"readers":["everyone"],"invitations":["ICML.cc/2026/Conference/Submission30175/Official_Review2/Rebuttal1/Rebuttal_Acknowledgement1/-/Reply_Rebuttal_Comment","ICML.cc/2026/Conference/-/Edit"],"mdate":1782369956852,"domain":"ICML.cc/2026/Conference","replyto":"klJnnoXAWa","id":"y7HnYAG7LC","forumContent":{"venue":{"value":"Submitted to ICML 2026"},"keywords":{"value":["LLM routing; LLM cascading; SLM-LLM collaboration; Cost-efficient inference"]},"_bibtex":{"value":"@misc{\nanonymous2026pay,\ntitle={Pay for Hints, Not Answers: {LLM} Shepherding for Cost-Efficient Inference},\nauthor={Anonymous},\nyear={2026},\nurl={https://openreview.net/forum?id=H8rywbFXuM}\n}"},"title":{"value":"Pay for Hints, Not Answers: LLM Shepherding for Cost-Efficient Inference"},"paperhash":{"value":"dong|pay_for_hints_not_answers_llm_shepherding_for_costefficient_inference"},"primary_area":{"value":"deep_learning->large_language_models"},"abstract":{"value":"Large Language Models (LLMs) deliver state-of-the-art performance on complex reasoning tasks, but their inference costs limit deployment at scale. Small Language Models (SLMs) offer dramatic cost savings yet lag substantially in accuracy. Existing approaches—routing and cascading—treat the LLM as an all-or-nothing resource: either the query bypasses the LLM entirely, or the LLM generates a complete response at full cost. We introduce **LLM Shepherding**, a framework that requests only a short prefix (a hint) from the LLM and provides it to SLM. This simple mechanism is surprisingly effective for math and coding tasks: even hints comprising 10--30\\% of the full LLM response improve SLM accuracy significantly. Shepherding generalizes both routing and cascading, and it achieves lower cost under oracle decision-making. We develop a two-stage predictor that jointly determines whether a hint is needed and how many tokens to request. On the widely-used mathematical reasoning (GSM8K, CNK12) and code generation (HumanEval, MBPP) benchmarks, Shepherding reduces costs by 42--94\\% relative to LLM-only inference. Compared to state-of-the-art routing and cascading baselines, shepherding delivers up to 2.8$\\times$ cost reduction while matching accuracy. To our knowledge, this is the first work to exploit token-level budget control for SLM-LLM collaboration. Code is available at https://anonymous.4open.science/r/LLM_Sheperding-77C3/."},"pdf":{"value":"/pdf/b5f1805f8847279424d1a89d1e99c69e9d93c924.pdf"},"venueid":{"value":"ICML.cc/2026/Conference/Rejected_Submission"},"authorids":{"value":["~Ziming_Dong1","~Hardik_Sharma3","~Evan_O'Toole1","~Jaya_Prakash_Champati1","~Kui_Wu1"]},"authors":{"value":["Ziming Dong","Hardik Sharma","Evan O'Toole","Jaya Prakash Champati","Kui Wu"]}},"version":2},{"content":{"comment":{"value":"We thank the reviewer for their constructive and insightful feedback.\n\n### 1. Evidence of Phenomenon with Simpler Objective\n\nWe provide hyperparameter sweeps for terms like oracle alignment, asymmetry, and sparsity in Appendix C (Table 5). We acknowledge that the current text does not isolate the routing-attribution mismatch under a fundamentally simplified loss function. To address the reviewer's concerns, we use a homogeneous setup of Qwen3-8B experts and perform the following additional experiment.\n\nWe perform pairwise term ablations to capture higher-order interactions. We categorize the individual loss terms into three functional groups and perform ablation studies on the GSM8K task. The groupings are as follows:\n\n- Core: Distillation + Oracle Terms\n- Structure: Symmetry + Sparsity Terms\n- Selection: Selection + Sparsity Terms\n\n| Expert Consortium | Full Objective | w/o Core | w/o Structure | w/o Selection |\n| ----------------- | -------------: | -------: | ------------: | ------------: |\n| Qwen3-8B $\\\\times$ 2       |           88.6 |     86.3 |          89.1 |          85.8 |\n| Qwen3-8B $\\\\times$ 3\t| 92.4\t|90.9\t|88.0\t|86.9|\n| Qwen3-8B $\\\\times$ 4\t| 92.4\t| 89.3\t| 85.8\t| 88.0 |\n| Qwen3-8B $\\\\times$ 5\t|92.4\t|90.1\t|86.9\t|89.1|\n\nRemoving any of these groups almost always degrades performance substantially compared to the full objective function. These grouped ablations confirm that no terms are actively conflicting -- they are complementary. The structural and selection constraints are necessary to regularize the core distillation process and prevent degenerate routing. This shows that these terms are actually necessary.\n\n### 2. Detailed Analysis of Heterogeneous Consortium\n\nWe partially address the heterogeneous consortium in Section 5, where we observe that it introduces significant volatility, slower convergence in routing confidence, and weaker centralization compared to the homogeneous setting (Figure 6). However, as the reviewer correctly noted, our primary detailed analyses appear to focus on the homogeneous setup. We will extend the full suite of analyses -- including attribution heatmaps, perturbation analyses, and masking interventions -- to the heterogeneous consortium and will include these results in the revised appendix.\n\n**UPDATE:** We have added the attribution heatmaps for the heterogeneous consortium in Section 5 (Fig 9, 10) of the revised version. While a complete perturbation and masking analysis for the heterogeneous setting is beyond the scope of the current revision, the newly added attribution results already indicate that the core conclusions remain consistent under the heterogeneous setup.\n\n### 3. Robustness of the Intrinsic-Importance Metric\n\nWe thank the reviewer for pointing this out. Ensuring the metric is not an artifact of the specific encoder is important. As a preliminary step, we swapped the frozen BERT-base encoder for several alternative embedding models (DistilBERT, GTE-small, DeBERTaV3-large). We observe that the orchestrator is largely encoder-agnostic, and overall task performance remains highly stable regardless of the embedding model used:\n\n|Expert Consortium|BERT-base|DistilBERT|GTE-small|DeBERTaV3-large|\n|---|---|---|---|---|\n|Qwen3-8B $\\\\times$ 5 | 96.1| 90.9|90.1|95.4|\n\nWhile these results confirm that the overall orchestration capability does not artificially depend on the specific encoder, we agree that the stability of the expert ranking itself must be explicitly verified. To fully address your critique, we are currently computing the gradient-based attribution scores using alternative strategies -- comparing our current `[CLS]` pooling against mean pooling, as well as extracting representations from intermediate layers. We commit to including these full rank correlation analyses in the revised appendix to confirm the robustness of the metric.\n\n**UPDATE:** We have added the results and robustness analysis for this point in Appendix M of the revised version.\n\n### 4. Scope of the Generality Claim\n\nWe currently discuss the application of INFORM beyond our primary setup by interpreting confidence-based cascade orchestration (FrugalGPT-style) in Section 5, and Appendix I. Appendix K discusses the boundaries of the framework's applicability, specifically addressing how structural analyses can be applied to black-box and API-based systems even when exact intrinsic importance cannot be computed. To strengthen the work and clarify our claims, we will add a dedicated \"Scope and Limitations\" paragraph in the main text that explicitly outlines the orchestration architectures we have validated and clearly states the environments where INFORM's applicability remains untested (e.g., fully decentralized multi-agent topologies without a centralized routing mechanism).\n\n**UPDATE:** We have added the scope and limitations in Section 6 of the revised version."},"title":{"value":"Response to Reviewer LSJ9 Comments"}},"parentInvitations":"TMLR/-/Official_Comment","tmdate":1778658405538,"tcdate":1778268531947,"writers":["TMLR","TMLR/Paper7372/Authors"],"signatures":["TMLR/Paper7372/Authors"],"forum":"4W7sgat04A","number":4,"license":"CC BY 4.0","cdate":1778268531947,"readers":["everyone"],"invitations":["TMLR/Paper7372/-/Official_Comment"],"mdate":1778658405538,"domain":"TMLR","replyto":"XnkhbrvX7R","id":"EXvl1rpcAr","forumContent":{"submission_length":{"value":"Long submission (more than 12 pages of main content)"},"venue":{"value":"Accepted by TMLR"},"code":{"value":"https://github.com/parmanu-lcs2/inform"},"supplementary_material":{"value":"/attachment/171f7e461162276df6370905a64ba727fa7d77db.zip"},"abstract":{"value":"Multi-expert systems, where multiple Large Language Models (LLMs) collaborate to solve complex tasks, are increasingly adopted for high-performance reasoning and generation. However, the orchestration policies governing expert interaction and sequencing remain largely opaque. We introduce INFORM, an interpretability analysis that treats orchestration as an explicit, analyzable computation, enabling the decoupling of expert interaction structure, execution order, and functional attribution. We use INFORM to evaluate an orchestrator on GSM8K, HumanEval, and MMLU using a homogeneous consortium of ten instruction-tuned experts drawn from LLaMA-3.1 8B, Qwen3 8B, and DeepSeek-R1 8B, with controlled decoding-temperature variation, and a secondary heterogeneous consortium spanning 1B-7B parameter models. Across tasks, routing dominance is a poor proxy for functional necessity. We reveal a divergence between relational importance, captured by routing mass and interaction topology, and intrinsic importance, measured via gradient sensitivity: frequently selected experts often act as interaction hubs with limited influence, while sparsely routed experts can be structurally critical. Orchestration behaviors emerge asynchronously, with expert centralization preceding stable routing confidence and expert ordering remaining non-deterministic. Targeted ablations show that masking intrinsically important experts induces disproportionate collapse in interaction structure compared to masking frequent peers, confirming that INFORM exposes functional and structural dependencies beyond accuracy metrics alone."},"_bibtex":{"value":"@article{\nghosh2026disentangling,\ntitle={Disentangling Intrinsic Importance from Emergent Structure in Multi-Expert Orchestration},\nauthor={Sudipto Ghosh and Sujoy Nath and Sunny Manchanda and Tanmoy Chakraborty},\njournal={Transactions on Machine Learning Research},\nissn={2835-8856},\nyear={2026},\nurl={https://openreview.net/forum?id=4W7sgat04A},\nnote={}\n}"},"video":{"value":"https://www.youtube.com/watch?v=f8TwBV-Xl-M"},"title":{"value":"Disentangling Intrinsic Importance from Emergent Structure in Multi-Expert Orchestration"},"changes_since_last_submission":{"value":"A previous version of this manuscript was desk-rejected on grounds of modified template. It was caused by a LaTeX package import. These issues have been fully corrected, and the manuscript now strictly adheres to the official TMLR template. The technical content remains unchanged."},"pdf":{"value":"/pdf/de4cbcdd723008c2af248976743fd6497561470e.pdf"},"venueid":{"value":"TMLR"},"paperhash":{"value":"ghosh|disentangling_intrinsic_importance_from_emergent_structure_in_multiexpert_orchestration"},"authorids":{"value":["~Sudipto_Ghosh4","~Sujoy_Nath1","~Sunny_Manchanda1","~Tanmoy_Chakraborty2"]},"previous_TMLR_submission_url":{"value":"https://openreview.net/forum?id=iKgVAXrVmP"},"assigned_action_editor":{"value":"~Jeff_Phillips1"},"authors":{"value":["Sudipto Ghosh","Sujoy Nath","Sunny Manchanda","Tanmoy Chakraborty"]}},"version":2},{"content":{"comment":{"value":"> (strengthen) Can the two CorDP approaches be combined, or can we predict beforehand which one is more likely to succeed?\n\n* Thank you for the questions. The two variants cannot be directly combined, as they are fundamentally different in how they handle forecast distributions (Sec 5.2). Median-CorDP discards the original distribution entirely \\- it uses only the base forecast median as a starting point and draws new samples from the LLM to form a corrected distribution. SampleWise-CorDP, by contrast,  preserves the original probabilistic structure by correcting each individual sample from the base forecast. The two variants are instead complementary by design, each suited to different task types, as discussed in Sec 5.3.  \n* On predicting which variant is more likely to succeed: the choice depends weakly on the LLM \\- among the 13 models where CorDP outperforms DP,  SampleWise-CorDP is best for 7 models and Median-CorDP for the remaining 6, with no clear LLM-level predictor. However,  task type is a stronger and more actionable signal (Sec 5.3 \\- paragraph titled “Performance varies by task type”) - SampleWise-CorDP excels on partial-RoI tasks, while Median-CorDP dominates full-shape and constrained tasks. This task-type specialization provides practitioners with a principled guide for variant selection.  \n* We have made this distinction and the task-type guidance explicit in Sections 5.2 and 5.3 of the revised version of the paper.\n\n> (strengthen) p10 If you have 71 tasks, how do you round/truncate to pick 20% increments?\n\n\n* Thank you for the question. The reported percentages are nominal labels for readability; the actual number of tasks routed is obtained by flooring (e.g., 20% of 71 tasks gives 14 tasks, 40% gives 28, and so on). We note that Appendix F.2 already provides the full routing curve without any rounding, so the coarse increments in the main text table are purely a presentational choice for conciseness.  \n* We have made this rounding convention clearer in the caption of Table 2\\.\n\n> (strengthen) For routing, if you didn't have multiple tasks to establish relative rankings, how would you decide which model to pick for a single task?\n    \n* We thank the reviewer for the question. As stated in Section 7.2, the router outputs a continuous numerical difficulty score between 0 (easiest) and 1 (hardest) for each task, so per-task routing is already supported: if we do not have multiple tasks to establish relative ranking, a threshold $\\\\tau$ can be chosen such that any task scoring above $\\\\tau$ is routed to the large model. This per-task setting and our batch setting are equivalent \\- they route the same tasks and achieve the same RCRPS: the \"fraction of tasks routed\" in Table 2 simply corresponds to exploring different percentile-based thresholds (20%, 40%, 60%, 80% quantiles of the router's score distribution).   \n\n\n* We explore this further and find that score ranges vary meaningfully across router sizes: larger routers (e.g., Qwen2.5-32B: \\[0.03, 0.39\\], Qwen2.5-72B: \\[0.05, 0.95\\]) spread scores across their range, while smaller routers collapse tasks into a narrow band (e.g., Qwen2.5-0.5B: \\[0.76, 0.84\\]). Once a router is chosen, calibrating $\\\\tau$ would require only a small held-out set of 5–10 tasks to estimate the relevant percentile of that router's score distribution \\- a lightweight, one-time step identical to standard practice in confidence-based routing \\[1, 2\\]. \n\n* We have updated the paper to reflect this: Section 7.2 now clarifies that our setup supports both batch and per-task routing, and a new Appendix F.2 provides a dedicated discussion with score range analysis across router sizes (Figure 48), showing that both settings yield equivalent results.\n\n\\[1\\] Chen, Lingjiao, Matei Zaharia, and James Zou. \"FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance.\" *Transactions on Machine Learning Research*.                  \n   \n\\[2\\] Ong, Isaac, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M. Waleed Kadous, and Ion Stoica. \"RouteLLM: Learning to Route LLMs from Preference Data.\" In *The Thirteenth International Conference on Learning Representations*.\n\n\n---\nWe hope these revisions fully address the raised concerns and that the updated paper provides a clearer and more complete picture of our contributions; we welcome any further questions."}},"parentInvitations":"TMLR/-/Official_Comment","tmdate":1777843344052,"tcdate":1777841612884,"writers":["TMLR","TMLR/Paper7655/Authors"],"signatures":["TMLR/Paper7655/Authors"],"forum":"dkjHHFJkVI","number":10,"license":"CC BY 4.0","cdate":1777841612884,"readers":["everyone"],"invitations":["TMLR/Paper7655/-/Official_Comment"],"mdate":1777843344052,"domain":"TMLR","replyto":"GQSfVq1Evn","id":"p4ONbLWvzm","forumContent":{"submission_length":{"value":"Regular submission (no more than 12 pages of main content)"},"venue":{"value":"Accepted by TMLR"},"code":{"value":"https://github.com/ashok-arjun/beyond-naive-prompting"},"abstract":{"value":"Real-world forecasting requires models to integrate not only historical data but also relevant contextual information provided in textual form. While large language models (LLMs) show promise for context-aided forecasting, critical challenges remain: we lack diagnostic tools to understand failure modes, performance remains far below their potential, and high computational costs limit practical deployment. We introduce a unified framework of four strategies that address these limitations along three orthogonal dimensions: model diagnostics, accuracy, and efficiency. Through extensive evaluation across model families from small open-source models to frontier models including Gemini, GPT, and Claude, we uncover both fundamental insights and practical solutions. Our findings span three key dimensions: diagnostic strategies reveal the “Execution Gap” where models correctly explain how context affects forecasts but fail to apply this reasoning; accuracy-focused strategies achieve substantial performance improvements of 25-50%; and efficiency-oriented approaches show that adaptive routing between small and large models can approach large model accuracy on average while significantly reducing inference costs. These orthogonal strategies can be flexibly integrated based on deployment constraints, providing practitioners with a comprehensive toolkit for practical LLM-based context-aided forecasting. Code is made available at \\url{https://github.com/ashok-arjun/beyond-naive-prompting}."},"_bibtex":{"value":"@article{\nashok2026beyond,\ntitle={Beyond Na{\\\"\\i}ve Prompting: Strategies for Improved Context-aided Forecasting with {LLM}s},\nauthor={Arjun Ashok and Andrew Robert Williams and Vincent Zhihao Zheng and Irina Rish and Nicolas Chapados and {\\'E}tienne Marcotte and Valentina Zantedeschi and Alexandre Drouin},\njournal={Transactions on Machine Learning Research},\nissn={2835-8856},\nyear={2026},\nurl={https://openreview.net/forum?id=dkjHHFJkVI},\nnote={}\n}"},"title":{"value":"Beyond Naïve Prompting: Strategies for Improved Context-aided Forecasting with LLMs"},"changes_since_last_submission":{"value":"# Changes post-rebuttal (May 2026):                       \n                                              \nAll post-rebuttal changes appear in blue text in the PDF. \n\n  - CorDP (Sec 5.3). Promoted the task-type finding into an explicit practitioner selection criterion (SampleWise-CorDP for partial-RoI tasks; Median-CorDP for full-shape/constrained tasks).                  \n  - CorDP (Sec 5.2). Clarified that Median-CorDP and SampleWise-CorDP are mechanistically distinct and complementary, not combinable.                                                                            \n  - RouteDP (Table 2). Added a note that percentage labels are nominal and actual task counts are floored.  \n  - RouteDP (Sec 7.2 + App F.2). Clarified that the setup supports per-task routing via a score threshold, equivalent to the batch setting; added score-range analysis across router sizes.                          \n  - FxDP (Sec 4.1).  Added a note that without FxDP, both steps are performed implicitly in a single forward pass, and that FxDP externalizes this process to make intermediate failures diagnosable.    \n  - FxDP (Sec 4.3). Replaced Fig 3 with an improved version: the Execution Gap segment is now highlighted in orange (previously gray) to draw attention to the central finding, and bracket annotations below the x-axis explicitly group models and label the two key results. \n  - FxDP (Sec 4.3). Added the evaluation reliability finding (no incorrect explanation leads to an improved forecast) in the results section.    \n  - Discussion and Future Work (Sec 9). Expanded on the connections between CorDP and tool-use agents, emphasized on the need for more unconstrained benchmarks, and elaborated on future directions related to finetuning.\n\n# Initial submission (Feb 2026):                       \n\nIn this submission we address all major concerns from the previous submission through new human evaluation studies that validate our ground truth and confirm agreement between our LLM judges and human annotators, a unified presentation of our strategies, and updated results with recent frontier models.\n\n## Major changes\n\n* **FxDP: Ground truth forecast effects.** We ran a human evaluation on the Prolific platform to verify the ground truth forecast effects. Each task was evaluated by five independent annotators; we aggregated responses by majority vote and used attention checks to ensure annotation quality. We obtained at least 4 out of 5 annotator agreement on all tasks and 80–100% agreement per item, thereby validating the gold standard. Further details are given in Appendix C.2.\n\n* **FxDP: LLM judges.** We compare models’ forecast effects to the ground truth using a panel of LLM judges (GPT-5.2, Claude-Sonnet-4.5, Gemini-2.5-Pro). All pairs of judges show high agreement, which indicates that the evaluation protocol provides clear, consistent, and objective criteria.\n\n* **FxDP: Human–LLM alignment.** To verify that LLM judge results align with human judgment, we designed a human evaluation on a subset of items. Five independent annotators per item performed the same comparison task as the judges, with majority-vote aggregation and attention checks, following standard annotation practice. Agreement between human evaluators and the LLM judges is 91.7–93.3% (Cohen’s κ = 0.81–0.84), with high inter-annotator and inter-judge agreement; see Appendix C.3. We open-source all evaluation artifacts, and our original findings remain unchanged.\n\n* **FxDP: Threshold analysis.** We use a 50% relative improvement threshold in RoI as the primary criterion and report sensitivity analyses at 30%–80% in Appendix C.4. We show that our findings hold across this range of thresholds.\n\n* **FxDP, CorDP, IC-DP.** We updated results for GPT-5.2, Gemini-2.5-Pro, and Claude-Sonnet-4.5. We also report costs and inference times for all of these models.\n\n* **Unification.** We present a unified view of our proposed strategies in a new Figure 1 and report results with multiple strategies combined.\n\n* **Supplementary material.** We provide code for all proposed methods.\n\n## Minor changes\n\n* **Renaming ReDP to FxDP:** We renamed ReDP (Direct Prompting with Reasoning over Context) to FxDP (Direct Prompt with Forecast Effect Explanation) to reflect our focus on the model’s perceived effect of the context on the forecast rather than full step-by-step reasoning traces. We now call the outputs “forecast effect explanations.” The methodology and main results are unchanged.\n\n* **FxDP:** We add Figure 2, which summarizes the major results, and add the full results table in the appendix."},"pdf":{"value":"/pdf/963cd6cbcfd30ca6fbab8f051d1c3bbd6dd3c144.pdf"},"venueid":{"value":"TMLR"},"paperhash":{"value":"ashok|beyond_naïve_prompting_strategies_for_improved_contextaided_forecasting_with_llms"},"authorids":{"value":["~Arjun_Ashok1","~Andrew_Robert_Williams1","~Vincent_Zhihao_Zheng1","~Irina_Rish1","~Nicolas_Chapados1","~Étienne_Marcotte1","~Valentina_Zantedeschi2","~Alexandre_Drouin2"]},"previous_TMLR_submission_url":{"value":"https://openreview.net/forum?id=CLuE0GL5xE"},"assigned_action_editor":{"value":"~Zachary_B._Charles1"},"authors":{"value":["Arjun Ashok","Andrew Robert Williams","Vincent Zhihao Zheng","Irina Rish","Nicolas Chapados","Étienne Marcotte","Valentina Zantedeschi","Alexandre Drouin"]}},"version":2},{"content":{"comment":{"value":"Dear AC:\n\nWe thank you for the time and effort devoted to our submission. Below is a concise summary of the strengths raised by reviewers and how we addressed the reviewers' concerns.\n\n-----------\n**Strength**\n\n**Clear motivation**  \nReviewers appreciate the clear insight of long-tail distribution that early turns are information-dense while later turns remain dependent but lightweight, motivating a local surrogate to cut multi-turn serving cost. \n\n\n**Strong, well-scoped pipeline**  \nThe soft-prompt mining, localized LoRA, and switch/rollback steps form a well-structured system that is easy to follow and compatible with existing inference engines. \n\n\n**Effective and consistent empirical performance**  \nSOMA achieves the highest response similarity to the original model across real-world datasets and model families while being efficient, showing strong multi-turn consistency and capability improvements. \n\n\n**Theoretical grounding that informs design**  \nTheoretical results help justify the mining objective, switching rule, and prompt coverage, and reviewers note that empirical behaviors match these predictions. \n \n\n**Relevance and practicality**  \nReviewers agree the method addresses an important bottleneck in practical LLM serving and provides a deployable framework for long multi-turn interactions. \n\n-----------\n**Addressed Concerns**\n\n**1. More Experimentas and theoretical support.** Reviewers noted that the original experiments did not sufficiently validate the theory or demonstrate how SOMA behaves under its stated assumptions. In response, we added new evaluations including MATH exact-match accuracy, within-family LLaMA experiments, and new baselines such as speculative decoding and FrugalGPT in the global response. We also included detailed switching-window analyses showing that early turns carry most information and that switching behavior matches the predictions of Lemma 1 and Theorem 2 in Appendix E.6. These additions directly support the theoretical claims and show that SOMA’s local-manifold approximation behaves as expected in practice.\n\n**2. Similarity versus task quality.** Reviewers were concerned that similarity scores may not reflect correctness. We clarified that SOMA is designed as a drop-in replacement for the original model, making response similarity the appropriate measure. To further strengthen the evaluation, we added a EM test on MATH (see global responses) that confirms SOMA preserves factual accuracy while maintaining high similarity.\n\n**3. Model pairing fairness and surrogate capacity.** Reviewer ejFa questioned whether large capability gaps made the evaluation unfair or unrealistic. We clarified that all models are from the same family and the large gap is intentional to test SOMA under difficult settings; if it works there, it should work even better when the surrogate is closer in size. We added LLaMA-3.1-8B experiments showing stronger fidelity in milder gaps, confirming robustness across surrogate strengths.\n\n**4. Dataset filtering.** Reviewer ejFa requested more explanation of filtering and token-usage statistics. We expanded Appendix B.4 with the filtering process and clarified how normalized turn-level token counts are computed and analyzed in Section 2.2. This shows that the long-tail structure is consistent across datasets and motivates SOMA’s design.\n\n**5. Methodological clarity.** Reviewers asked why the teacher receives verbalized prompts while the surrogate uses continuous prompts and requested clearer explanations of semantic divergence loss and gate behavior. We rewrote Sections 3.1–3.3 to explain that verbalization is necessary because the teacher is API-only and cannot accept embeddings. We also added an intuitive description of expectation-weighted divergence as distribution-level alignment and clarified the fast semantic-closeness test used for switching and rollback. We also corrected a typo in the fine-tuning loss and explained why maximal-divergence mining identifies the surrogate’s most important failure directions.\n\n**6. Theory assumptions and their justification.** Reviewers questioned the low-rank smoothness and sub-Gaussian assumptions. We clarified that our analysis relies only on local smoothness and approximate low-rank structure, consistent with prior manifold-learning literature. We added empirical evidence in Appendix E.5–E.6 showing early-turn information concentration and smooth switching curves that align with the assumptions used in Lemma 1 and Theorem 2.\n\n**7. Applicability concerns.** Reviewers worried about warm-start cost and imperfect drift detection. We clarified SOMA targets long context-dependent multi-turn use and falls back to the teacher in short chats, and emphasized rollback plus natural capacity limits rather than solving extreme size gaps.\n\n-----\n\nWe believe these revisions address all major reviewer concerns and significantly strengthen the manuscript. We greatly appreciate your time and considerations."},"title":{"value":"Summarization of reviews and addressed concerns"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1770918031016,"tcdate":1764817321276,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission22291/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission22291/Authors"],"forum":"B821Pc5huD","number":10,"license":"CC BY 4.0","cdate":1764817321276,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission22291/-/Official_Comment","ICLR.cc/2026/Conference/-/Edit"],"mdate":1770918031016,"domain":"ICLR.cc/2026/Conference","replyto":"B821Pc5huD","id":"rpARp5dGyl","forumContent":{"venue":{"value":"Submitted to ICLR 2026"},"keywords":{"value":["Small Language Model","multi-turn conversations","local manifold approximation"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Large Language Models (LLMs) are increasingly deployed in multi-turn dialogue settings where preserving conversational context across turns is essential. A standard serving practice concatenates the full dialogue history at every turn, which reliably maintains coherence but incurs substantial cost in latency, memory, and API expenditure, especially when queries are routed to large proprietary models. Existing approaches often struggle to balance the trade-off between response quality and efficiency. We propose a framework that exploits the early turns of a session to estimate a local response manifold and then adapt a smaller surrogate model to this local region for the remainder of the conversation. Concretely, we learn soft prompts that maximize semantic divergence between the large and surrogate small language models' responses to surface least-aligned local directions, stabilize training with anti-degeneration control, and distill the mined cases into localized LoRA fine-tuning so the surrogate runs without prompts at inference. A simple gate enables a one-time switch with rollback on drift. We further provide a theoretical analysis for key components in SOMA. Extensive experiments show the effectiveness of SOMA. The source code is provided at: https://anonymous.4open.science/r/SOMA-D377."},"_bibtex":{"value":"@misc{\ncheng2026soma,\ntitle={{SOMA}: Efficient Multi-turn {LLM} Serving via Small Language Model},\nauthor={Xueqi Cheng and Qiong Wu and Zhengyi Zhou and Xugui Zhou and Tyler Derr and Yushun Dong},\nyear={2026},\nurl={https://openreview.net/forum?id=B821Pc5huD}\n}"},"title":{"value":"SOMA: Efficient Multi-turn LLM Serving via Small Language Model"},"pdf":{"value":"/pdf/b48d6bf40b465bca46fe81bea8253ea534b19776.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference/Rejected_Submission"},"paperhash":{"value":"cheng|soma_efficient_multiturn_llm_serving_via_small_language_model"},"authorids":{"value":["~Xueqi_Cheng2","~Qiong_Wu3","~Zhengyi_Zhou1","~Xugui_Zhou1","~Tyler_Derr1","~Yushun_Dong1"]},"authors":{"value":["Xueqi Cheng","Qiong Wu","Zhengyi Zhou","Xugui Zhou","Tyler Derr","Yushun Dong"]}},"version":2},{"content":{"comment":{"value":"**Dear AC,**\n\nThank you for accepting our paper as a re-assigned submission. We truly appreciate the additional workload this may have caused. We are also grateful for the reviewers’ thoughtful and constructive feedback.\n\n**We appreciate that the reviewers acknowledged the contributions of our work:**\n\n- **Reviewer JRcd** highlighted that the problem formulation is clear and that our **\"idea will be of interest to the community\".**\n- **Reviewer xhvr** praised the \"originality\" of shifting from centralized to distributed routing and noted the \"rigorous\" experimental design.\n- **Reviewer qdad** recognized that DiSRouter achieves a utility score \"significantly superior\" to baselines and appreciated its \"plug-and-play\" modularity.\n- **Reviewer dfTT** acknowledged the \"concrete technical contribution\" of our Self-Awareness Training pipeline.\n\n---\n### **Summary of Key Concerns and Our Responses**\nWe consolidated the feedback into four core questions and present our responses below.\n\n**Core Concern A: Cross-Architecture Generalization (Homogeneity)**\n\n*(Raised by Reviewers xhvr [W1], dfTT [W2])*\n\n**Reviewer Comment:** Concerns were raised that our experiments relied solely on the Qwen series, questioning applicability to heterogeneous agent pools.\n\n**Our Response:** We conducted **new cross-architecture experiments** (as presented in Appendix A.10 of the revised paper) by integrating `Gemma3-1B-Instruct` and `Phi4-mini-Instruct` into the routing chain. The results confirm that our Self-Awareness Training is **architecture-agnostic**. The heterogeneous system (Gemma/Phi/Qwen) maintained high scenario adaptability and superior utility, demonstrating that DiSRouter generalizes effectively to diverse model families.\n\n---\n**Core Concern B: Routing Overhead and Utility Metric Formulation**\n\n*(Raised by Reviewers dfTT [W1], qdad [Q2], JRcd [W2])*\n\n**Reviewer Comment:** Reviewer dfTT argued that our Utility metric was flawed for ignoring cumulative latency (routing overhead) in the cascade. Reviewers qdad and JRcd are also concerned about the overhead of multiple rejection responses.\n\n**Our Response:** We clarified that the omission of the routing overhead term in the original formula was a **deliberate simplification for mathematical conciseness**, predicated on our analysis that this term is statistically negligible as presented in Appendix A.1. To validate this:\n\n1. **Empirical Verification:** We measured that the cumulative \"routing cost\" (time spent rejecting) constitutes **<5%** of the inference cost even in worst-case paths.\n2. **Adjusted Metric:** We re-calculated our results using a stricter Utility metric that explicitly subtracts routing overhead ($Cost_{routing}$). As shown in Table R2 (Response to dfTT), the utility drop is marginal (~1.5%) and DiSRouter retains its superiority over baselines.\n3. **Time-to-First-Token:** We demonstrated that while DiSRouter incurs slightly higher latency than predictive routers (e.g.,s RouteLLM) which use lightweight classifier, it is **significantly faster** than generative baselines that rely on full model outputs or verification steps (e.g., FrugalGPT, AutoMix).\n\n---\n**Core Concern C: Training Efficiency and Overhead**\n\n*(Raised by Reviewers xhvr [W2], qdad [W1])*\n\n**Reviewer Comment:** Reviewers expressed concern that the two-stage training approach (SFT+RL) is computationally heavy compared to training a lightweight centralized router.\n\n**Our Response:** We argue that the training overhead is justified and optional:\n\n1. **Optionality (GPT Series Experiments):** We demonstrated that high-capability models (e.g., GPT-4o-mini) possess **intrinsic self-awareness** and do not require training. We constructed a **\"Training-Free DiSRouter\"** using GPT models, which achieved comparable utility on in-domain tasks and much higher utility on OOD tasks without any gradient updates.\n2. **Amortized Cost:** Unlike centralized routers that require full retraining whenever the model pool and tasks change, DiSRouter is decoupled. Training is a one-time investment per agent; once trained, an agent is a modular \"plug-and-play\" component.\n---\n**Core Concern D: Generalization to Open-Ended Tasks**\n\n*(Raised by Reviewers dfTT [W3], qdad [W2])*\n\n**Reviewer Comment:** Questions regarding whether the binary \"reject/answer\" training applies to open-ended tasks like summarization where correctness is non-binary.\n\n**Our Response:** We extended our framework to **Text Summarization** (SAMSum, XSum) in Appendix A.11. We adapted the reward function to use continuous metrics (BERTScore) and percentile-based thresholds. The results show that DiSRouter successfully distinguishes high-quality generations from low-quality ones, proving the method is **transferable to open-ended generation tasks.**"},"title":{"value":"Summary of Reviews & New Experiments: Addressing All Reviewer Concerns with Definitive Evidence"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764670644614,"tcdate":1764670644614,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission7635/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission7635/Authors"],"forum":"KDcwXKr0NU","number":16,"license":"CC BY 4.0","cdate":1764670644614,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission7635/-/Official_Comment"],"mdate":1764670644614,"domain":"ICLR.cc/2026/Conference","replyto":"KDcwXKr0NU","id":"oiG3tNvUb3","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"TLDR":{"value":"We introduce DiSRouter, a distributed system where LLMs leverage \"self-awareness\" to route queries among themselves, outperforming conventional centralized routers."},"keywords":{"value":["query routing","model selection","distributed system","self-awareness of LLM"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"The proliferation of Large Language Models (LLMs) has created a diverse ecosystem of models with highly varying performance and costs, necessitating effective query routing to balance performance and expense. Current routing systems often rely on a centralized external router trained on a fixed set of LLMs, making them inflexible and prone to poor performance since the small router can not fully understand the knowledge boundaries of different LLMs. We introduce DiSRouter (Distributed Self-Router), a novel paradigm that shifts from centralized control to distributed routing. In DiSRouter, a query traverses a network of LLM agents, each independently deciding whether to answer or route to other agents based on its own self-awareness—its ability to judge its competence. This distributed design offers superior flexibility, scalability, and generalizability. To enable this, we propose a two-stage Self-Awareness Training pipeline that enhances each LLM's self-awareness. Extensive experiments demonstrate that DiSRouter significantly outperforms existing routing methods in utility across various scenarios, effectively distinguishes between easy and hard queries, and shows strong generalization to out-of-domain tasks. Our work validates that leveraging an LLM's intrinsic self-awareness is more effective than external assessment, paving the way for more modular and efficient multi-agent systems."},"_bibtex":{"value":"@inproceedings{\nzheng2026disrouter,\ntitle={Di{SR}outer: Distributed Self-Routing for {LLM} Selections},\nauthor={Hang Zheng and Hongshen Xu and Yongkai.lin and Shuai Fan and Lu Chen and Kai Yu},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=KDcwXKr0NU}\n}"},"title":{"value":"DiSRouter: Distributed Self-Routing for LLM Selections"},"pdf":{"value":"/pdf/0190f98d94890b700d03febdee678e22a2a5e380.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"zheng|disrouter_distributed_selfrouting_for_llm_selections"},"authorids":{"value":["~Hang_Zheng3","~Hongshen_Xu1","~Yongkai.lin1","~Shuai_Fan1","~Lu_Chen3","~Kai_Yu3"]},"authors":{"value":["Hang Zheng","Hongshen Xu","Yongkai.lin","Shuai Fan","Lu Chen","Kai Yu"]}},"version":2},{"content":{"comment":{"value":"Thank you for your thorough review and insightful questions, and for highlighting the novelty and effectiveness of our method. \n\n\n**Comparison with Mixture-of-experts and Test-time Adaptation Methods**\n\nOur approach is complementary to these methods and can be used in conjunction with them. Specifically, although state-of-the-art open and closed models already use mixture-of-experts architectures, they have different strenghts and weaknesses, and do not dominate other models on all existing benchmarks. In such settings, post-hoc model routing can be used to leverage the streghts of multiple models instead of relying on a single one.\n\n\n**Discussion of Cascade-based Routing Methods**\n\nOur method makes a single routing decision before any model processes the query. In contrast, cascades (e.g., frugalGPT) make dynamic mid-inference decisions, which can trigger model switches, lose KV-cache benefits, and force full reprocessing which is especially costly for long docQA queries. As a result, cascades incur higher inference-time overhead and are less plug-and-play with existing models.\n\nOur design choices in RADAR are a deliberate trade-off between routing complexity, inference-time compute, and accuracy. RADAR is simpler and faster (routing adds 7ms per query in our setup), though potentially less accurate than sophisticated cascades. Our hypervolume performance shows we are close to Pareto-optimal. The marginal accuracy gains from cascading would not justify the extra latency and engineering complexity. Further, these two routing-approaches can be combined in a hybrid design for future work. RADAR can seed a cascade with a strong initial model to reduce mid-generation model switches and total latency.\n\n\n\n**Throughput Analysis**\n\nFor a throughput analysis we compute the queries/second of the smallest (0.6B) and largest (8B) open source Qwen3 models on the MATH-500 benchmark averaged across 3 runs on a single Nvidia A100 80GB GPU using vLLM for batched inference and embedding. We exclude OpenAI o4-mini models due to their variable API-based throughput. Qwen3-0.6B with zero reasoning budget processes queries at 1.1529 queries/sec while Qwen3-8B with 16K reasoning budget processes queries at 0.2337 queries/sec. Adding RADAR's routing overhead decreases throughput by 0.78% for Qwen3-0.6B with zero reasoning budget to 1.1438 queries/sec and by 0.15% for Qwen3-8B with a 16K reasoning budget to 0.2237 queries/sec. Depending on the user specified performance-cost tradeoff weight, RADAR routes queries to a convex combination of model configurations with the throughput therefore bounded by 1.1438 queries/sec as the upper bound and 0.2237 queries/sec as the lower bound. Multi-GPU implementation and inference optimization methods can further improve the throughput of RADAR.\n\n\n\n**Improving OOD Performance on AIME**\n\nRADAR exhibits strong generalization performance to OOD queries consistently across benchmarks with a small decrease in performance compared with in-distribution queries even on long-context docQA. The only benchmark with a slight decrease in performance is AIME. Our analysis reveals two special characteristics of AIME. First, AIME contains queries with the highest average difficulty, significantly higher than other benchmarks. Second, performance on AIME consistently improves with an increase in reasoning length and does not plateau unlike other benchmarks. In the OOD setting, RADAR which is not exposed to difficult math problems, underestimates query difficulty and underperforms by allocating less capable models.\n\n\nTable 1: Predicted query difficulty on test questions. IRT difficulty values usually lie in [-3, 3] with a higher value denoting higher difficulty.\n| Benchmark | Diff Avg | Diff Std |\n| - | - | - |\n| AIME (ID) | 1.038 | 1.275 |\n| AIME (OOD)| -0.461 | 1.043 |\n\n\nFor real-world usage, it is practical to assume some model training exposure to math problems. We perform a partial OOD experiment where we expose RADAR to a fraction of AIME and MATH queries in the training data. We see a significant increase in performance with just 5% of exposure, and a similar performance to the ID setting with 30% of exposure. \n\n\nTable 2: Partial OOD experimental setting showing a small set of training queries can recover ID performance.\n| Benchmark | RADAR (ID) | RADAR (OOD) | RADAR (OOD + 5%) | RADAR (OOD + 30%) | \n| - | - | - | - | - |\n| AIME | 0.776 | 0.7566 | 0.7665 | 0.7787 |"},"title":{"value":"Response to Reviewer EkVW (Part 1)"}},"parentInvitations":"ICLR.cc/2026/Conference/-/Official_Comment","tmdate":1764066125782,"tcdate":1764066125782,"writers":["ICLR.cc/2026/Conference","ICLR.cc/2026/Conference/Submission9782/Authors"],"signatures":["ICLR.cc/2026/Conference/Submission9782/Authors"],"forum":"CB6Ds5T4ae","number":2,"license":"CC BY 4.0","cdate":1764066125782,"readers":["everyone"],"invitations":["ICLR.cc/2026/Conference/Submission9782/-/Official_Comment"],"mdate":1764066125782,"domain":"ICLR.cc/2026/Conference","replyto":"cC7FC9cuXW","id":"i6kG7WugEs","forumContent":{"venue":{"value":"ICLR 2026 Poster"},"keywords":{"value":["routing","adaptive reasoning","item response theory","reasoning models","large language models"]},"primary_area":{"value":"foundation or frontier models, including LLMs"},"abstract":{"value":"Reasoning language models have demonstrated remarkable performance on many challenging tasks in math, science, and coding. Choosing the right reasoning model for practical deployment involves a performance-cost trade-off at two key levels: model size and reasoning budget, where larger models and higher reasoning budgets lead to better performance but incur greater cost and latency. In this work, we tackle this tradeoff from the angle of model configuration routing for different queries, and present RADAR (Reasoning–Ability and Difficulty-Aware Routing), a lightweight, interpretable, and scalable routing framework. Inspired by psychometrics, RADAR learns an item response model from model responses with different budgets to different queries, with interpretable parameters including query difficulties and model-budget abilities. RADAR then routes queries with higher difficulty to model-budget pairs with higher ability, and vice versa. We conduct extensive experiments on 8 widely used challenging reasoning benchmarks, demonstrating the superior performance of RADAR compared to state-of-the-art model routing methods. RADAR also exhibits query generalization capabilities, achieving strong performance on out-of-distribution queries on all benchmarks. RADAR is also scalable and can efficiently integrate additional models by dynamically selecting a small set of evaluation queries to estimate their abilities."},"_bibtex":{"value":"@inproceedings{\nfernandez2026radar,\ntitle={{RADAR}: Reasoning-Ability and Difficulty-Aware Routing for Reasoning {LLM}s},\nauthor={Nigel Fernandez and Branislav Kveton and Ryan A. Rossi and Andrew Lan and Zichao Wang},\nbooktitle={The Fourteenth International Conference on Learning Representations},\nyear={2026},\nurl={https://openreview.net/forum?id=CB6Ds5T4ae}\n}"},"title":{"value":"RADAR: Reasoning-Ability and Difficulty-Aware Routing for Reasoning LLMs"},"pdf":{"value":"/pdf/0aee0b8194c380f76d5f3af79f768e66adeeb181.pdf"},"venueid":{"value":"ICLR.cc/2026/Conference"},"paperhash":{"value":"fernandez|radar_reasoningability_and_difficultyaware_routing_for_reasoning_llms"},"authorids":{"value":["~Nigel_Fernandez1","~Branislav_Kveton1","~Ryan_A._Rossi2","~Andrew_Lan1","~Zichao_Wang1"]},"authors":{"value":["Nigel Fernandez","Branislav Kveton","Ryan A. Rossi","Andrew Lan","Zichao Wang"]}},"version":2},{"content":{"comment":{"value":"We thank the reviewer for recognizing that ABC is `elegant and does not require additional training`, provides `formal definitions of safe deferral rules and bounds on accuracy and inference cost`, and for validating our work across `diverse settings`. We appreciate these positive assessments and address the concerns below:\n\n>> The current framework and deferral rule design assume fixed output spaces (classification or extractive QA). ABC is not applicable to open-ended generation tasks, which dominate many LLM applications.\n\nThe reviewer correctly notes that ABC is not directly applicable to open-generation tasks. We acknowledged this limitation and note that this limitation is shared across much of the cascading field, including recent works at top venues such as NeurIPS [1, 6], ICLR [2, 3], ICML [4], and TMLR [5]. While open-generation has gained prominence, classification, closed generation, and reasoning tasks remain critical for many applications, particularly edge deployments where ABC’s simplicity and generality provides significant advantages without requiring platform-specific optimizations.\n\n>> While the threshold is empirically tuned on held-out data, there is limited discussion about its robustness or failure modes across datasets. It seems sensitive to threshold theta.\n\nWe included such analyses and discussion in Appendix B, though they may have been missed as they were in the appendix. Based on the reviewer’s feedback, in our revision we have instead included our threshold robustness analyses in the main text in Section 5.3. These results show that threshold estimation is stable across model accuracies from 37% to 86% (Figure 6) and converges with just 100 validation samples. Importantly, ABC maintains strong performance even without explicit threshold tuning when relative costs are sufficient ($\\gamma$ <= 1/50), as shown in Figure 3; moreover, the black-box API experiments in Section 5.2.3 produce competitive performance without access to logits (hence, no thresholding), addressing concerns about cross-task generalization.\n\n\n>> Requiring multiple pretrained models of similar size (to form ensembles per tier) could pose a deployment challenge in memory-constrained environments or for niche tasks with limited model availability.\n\nWe acknowledge that multiple pretrained models of similar size could pose deployment challenges. However, this concern is mitigated by several factors that we discuss in the paper: (1) the abundance of models in public repositories like HuggingFace makes model availability less of an issue; (2) it is easy to produce models of various sizes using standard compression techniques, further increasing the size of existing model repositories; (3) our cost analysis (Tables 4-5) shows that most samples (52-93%) are processed at early, cheaper tiers, reducing memory pressure; and (4) ABC works with heterogeneous models of different sizes and architectures, allowing flexibility and not requiring exactly similar-sized models.\n\n\n**On providing more detail on voting threshold**: Our threshold selection procedure is detailed in Appendix B with empirical validation in Figure 6, and now well-referenced in the new Section 5.3. While we don’t provide formal generalization guarantees (an interesting direction for future work), our empirical evidence shows robust performance across diverse tasks and model combinations.\n\n**On adding experiments/discussion on scenarios where ABC fails to deliver cost savings**: Following this suggestion, we have added analysis in Section 5.3 clarifying scenarios where ABC provides minimal benefits: (1) when models have similar accuracy, (2) when relative costs are high ($\\gamma$ >= 1/5) and parallelization is not free.\n\n---\n[1] Aggarwal, P., Madaan, A., Anand, A., Potharaju, S. P., Mishra, S., Zhou, P., ... & Mausam, M. *Automix: Automatically mixing language models.* NeurIPS, 2024.\n\n[2] Ding, D., Mallick, A., Wang, C., Sim, R., Mukherjee, S., Rühle, V., ... & Awadallah, A. H. *Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing.* ICLR, 2024.\n\n[3] Yue, M., Zhao, J., Zhang, M., Du, L., & Yao, Z.  *Large Language Model Cascades with Mixture of Thought Representations for Cost-Efficient Reasoning.* ICLR, 2024.\n\n[4] Nie, L., Ding, Z., Hu, E., Jermaine, C., & Chaudhuri, S. *Online Cascade Learning for Efficient Inference over Streams.* ICML, 2024.\n\n[5] Chen, L., Zaharia, M., & Zou, J. *FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance.* TMLR, 2024.\n\n[6] Jitkrittum, W., Gupta, N., Menon, A. K., Narasimhan, H., Rawat, A., & Kumar, S. *When does confidence-based cascade deferral suffice?.* NeurIPS, 2023."},"title":{"value":"Response to Reviewer hRB4"}},"parentInvitations":"TMLR/-/Official_Comment","tmdate":1749591002845,"tcdate":1749579806151,"writers":["TMLR","TMLR/Paper4338/Authors"],"signatures":["TMLR/Paper4338/Authors"],"forum":"jn9B7LMlzk","number":6,"license":"CC BY 4.0","cdate":1749579806151,"readers":["everyone"],"invitations":["TMLR/Paper4338/-/Official_Comment"],"mdate":1749591002845,"domain":"TMLR","replyto":"zaTbXlDm7H","id":"PUWye52AND","forumContent":{"submission_length":{"value":"Long submission (more than 12 pages of main content)"},"venue":{"value":"Accepted by TMLR"},"code":{"value":"https://github.com/stevenkolawole/Agreement-Based-Cascading"},"abstract":{"value":"Adaptive inference schemes reduce the cost of machine learning inference by assigning smaller models to easier examples, attempting to avoid invocation of larger models when possible. In this work we explore a simple, effective adaptive inference technique we term Agreement-Based Cascading (ABC). ABC builds a cascade of models of increasing size/complexity and uses agreement between ensembles of models at each level of the cascade as a basis for data-dependent routing. Although ensemble execution introduces additional expense, we show that these costs can be easily offset in practice due to large expected differences in model sizes, parallel inference execution capabilities, and accuracy benefits of ensembling. We examine ABC theoretically and empirically in terms of these parameters, showing that the approach can reliably act as a drop-in replacement for existing models and surpass the best single model it aims to replace in terms of both efficiency and accuracy. Additionally, we explore the performance of ABC relative to existing cascading methods in three common scenarios: (1) edge-to-cloud inference, where ABC reduces communication costs by up to 14x; (2) cloud-based model serving, where it achieves a 3x reduction in rental costs; and (3) inference via model API services, where ABC achieves a 2-25x reduction in average price per token/request relative to state-of-the-art LLM cascades."},"_bibtex":{"value":"@article{\nkolawole2025agreementbased,\ntitle={Agreement-Based Cascading for Efficient Inference},\nauthor={Steven Kolawole and Don Dennis and Ameet Talwalkar and Virginia Smith},\njournal={Transactions on Machine Learning Research},\nissn={2835-8856},\nyear={2025},\nurl={https://openreview.net/forum?id=jn9B7LMlzk},\nnote={}\n}"},"title":{"value":"Agreement-Based Cascading for Efficient Inference"},"pdf":{"value":"/pdf/9c0fb8f6822c99d87fbb5b43ea8fa0be630e8670.pdf"},"venueid":{"value":"TMLR"},"paperhash":{"value":"kolawole|agreementbased_cascading_for_efficient_inference"},"authorids":{"value":["~Steven_Kolawole1","~Don_Dennis2","~Ameet_Talwalkar1","~Virginia_Smith1"]},"assigned_action_editor":{"value":"~Yaoliang_Yu1"},"authors":{"value":["Steven Kolawole","Don Dennis","Ameet Talwalkar","Virginia Smith"]}},"version":2},{"content":{"comment":{"value":"> \"From work on image classification, however, we know that down-scaling neural networks affects more than just top-line metrics or task accuracy. Pruning, for example, can introduce biases (Hooker et al., 2019) or disproportionate affects on certain subsets of the data (Jin et al., 2022).\" <- This claim is too strong, it makes it seem as though it is a certainty that such effects occur given down-scaling. However, a significant amount of work has shown that pruning is an effective tool for vision models.\n\nWe agree. Our use of the word “can” intends to convey the uncertainty. If you think this is insufficient, we are happy to be more precise about the precondition of the said phenomenon – “Pruning to very high sparsity, for example, can introduce biases…”\n\n> \"It is difficult to assess these abilities in isolation, as a standard downstream task needs to process the information provided in context as well as access the information stored in weights.\" <- Please contrast related work which has previously explored zero-to-many shot ICL performance (across different target applications); see the following for an extensive overview: Dong et al, \"A Survey on In-context Learning\", https://arxiv.org/pdf/2301.00234.pdf\n\nThank you for sharing the survey on ICL. Our definition of ICL is consistent with the one provided in the survey. The survey summarizes work on various aspects of ICL (algorithmic mechanisms behind it, evaluation of ICL, benchmarks, improving ICL, ICL in smaller models via distillation, etc.), but does not mention any work attempting to disentangle ICL and fact recall, or evaluating how pruning affects ICL, thus providing further evidence on the novelty of our contribution.\n\n> \"Improve inference efficiency. Our work reveals that scaling down model size alone has little impact on tasks demanding processing information in the LLM’s context. Practitioners may thus use our findings to identify scenarios where decisions could be routed to a smaller model instead of a larger one without hurting task performance (Chen et al., 2023; Dohan et al., 2022).\" <- The latter work already explores how the parameter size affects performance. In particular, the Wanda paper already tackles the question of how pruning affects ICL performance (and compares to SparseGPT)\n\nPlease also refer to our global response for a discussion of our contribution within the context of existing pruning work.\n\nFrugalGPT[8] (Chen et al.,), SparseGPT[1] and Wanda[2] indeed all investigated the task performance-size/cost trade-off of LLMs. However, when a new task emerges outside of those examined by these prior studies, how can a practitioner know if it is a good idea to route it to a smaller model? Our work provides insight in this scenario: practitioners can assess how much the said task relies on parametric knowledge versus context processing capability. The more a task relies on context processing, the more likely such a task can be routed to a smaller model.\n\n> \"Our work differs from prior scaling studies in two ways: while prior work (Kaplan et al., 2020b) studies joint scaling of both pre-training corpus size and model size, we focus on scaling model size alone. Furthermore, instead of measuring task performance, we focus on foundational capabilities of LLMs–fact recall and ICL. These capabilities drive the success for many real world applications of LLMs\" <- This is wrong for a number of reasons. Firstly, \"we focus on scaling model size along\" is not a valid contribution, as this would, by definition, be provided in the study of \"joint scaling of both pre-training corpus size and model size.\" \n\nWe use this statement to highlight a difference in methodological choice – we want to be upfront about the more focused and thus narrower scope of our study. To be clear, we use this statement to disambiguate, not to present a contribution.\n\n> Secondly, the work of Kaplan does not study pruning, but rather LLM model size->training->resulting performance. It is necessary to demarcate the difference between these two paradigms.\n\nWe agree. The difference is important – 1). In addition to dense scaling (Sec 6), which follows the paradigm of Kaplan et al, we also studied pruning. 2). Instead of perplexity, we assess LLMs on its ability to recall information from pretraining vs process contextual information. Both differences are also our key contributions."},"title":{"value":"Response - Claims (1/2)"}},"tmdate":1700203776750,"tcdate":1700203596481,"writers":["ICLR.cc/2024/Conference","ICLR.cc/2024/Conference/Submission3569/Authors"],"signatures":["ICLR.cc/2024/Conference/Submission3569/Authors"],"forum":"ldJXXxPE0L","number":6,"license":"CC BY 4.0","cdate":1700203596481,"readers":["everyone"],"invitations":["ICLR.cc/2024/Conference/Submission3569/-/Official_Comment"],"mdate":1700203776750,"domain":"ICLR.cc/2024/Conference","replyto":"IgDHerI2sU","id":"JEzv8kla7O","forumContent":{"venue":{"value":"ICLR 2024 poster"},"TLDR":{"value":"Moderate down-scaling harms fact recall, and yet the ability to learn from a few input-output examples from context withstands aggressive down-scaling."},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["large language model","scaling","in-context learning","pruning"]},"primary_area":{"value":"generative models"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide."},"abstract":{"value":"We study how down-scaling large language model (LLM) size impacts LLM capabilities. We begin by measuring the effects of weight pruning – a popular technique for reducing model size – on the two abilities of LLMs: (a) recalling facts presented during pre-training and (b) processing information presented in context. Surprisingly, we find that existing pruning techniques affect these two abilities of LLMs differently. For example, pruning more than 30% of weights significantly decreases an LLM’s ability to recall facts presented during pre-training. Yet pruning 60-70% of weights largely preserves an LLM’s ability to process information in-context, ranging from retrieving answers based on information presented in context to learning parameterized functions such as a linear classifier based on a few examples. Moderate pruning impairs LLM’s ability to recall facts learnt from pre-training. However, its effect on model’s ability to process information presented in context is much less pronounced. The said disparate effects similarly arise when replacing the original model with a smaller dense one with reduced width and depth. This similarity suggests that model size reduction in general underpins the said disparity."},"_bibtex":{"value":"@inproceedings{\njin2024the,\ntitle={The Cost of Scaling Down Large Language Models: Reducing Model Size Affects Memory before In-context Learning},\nauthor={Tian Jin and Nolan Clement and Xin Dong and Vaishnavh Nagarajan and Michael Carbin and Jonathan Ragan-Kelley and Gintare Karolina Dziugaite},\nbooktitle={The Twelfth International Conference on Learning Representations},\nyear={2024},\nurl={https://openreview.net/forum?id=ldJXXxPE0L}\n}"},"title":{"value":"The Cost of Scaling Down Large Language Models: Reducing Model Size Affects Memory before In-context Learning"},"pdf":{"value":"/pdf/6a6c078824f86b9c8353c38a86794c78253b804f.pdf"},"venueid":{"value":"ICLR.cc/2024/Conference"},"paperhash":{"value":"jin|the_cost_of_scaling_down_large_language_models_reducing_model_size_affects_memory_before_incontext_learning"},"authorids":{"value":["~Tian_Jin1","~Nolan_Clement1","~Xin_Dong1","~Vaishnavh_Nagarajan3","~Michael_Carbin1","~Jonathan_Ragan-Kelley1","~Gintare_Karolina_Dziugaite1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors' identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Tian Jin","Nolan Clement","Xin Dong","Vaishnavh Nagarajan","Michael Carbin","Jonathan Ragan-Kelley","Gintare Karolina Dziugaite"]}},"version":2},{"content":{"comment":{"value":"We sincerely thank the reviewer for the prompt follow-up and greatly appreciate that our earlier rebuttal addressed your main concerns. To further respond to the the \"idealized formulation\" concern, we would like to clarify that we conducted a **fully end-to-end experiment in a more realistic deployment setting**.\n\nIn this experiment, we completely bypass the surrogate function ϕ and instead invoke GPT-4o directly in real time for every winning bid. Moreover, to better **reflect real-world system uncertainty**, we introduce an explicit 5% failure-to-deliver rate: the platform still incurs the investment cost, but with 5% probability the enhanced ad cannot be served due to **latency** and the system reverts to the original ad. This setting captures both the practical inference overhead and the instability of real API performance, rather than relying on an idealized deterministic environment.\n\nUnder this substantially harsher end-to-end setting, TTSL-PPO still achieves a +2.04% revenue gain over the no-invest baseline. We believe this result is important for two reasons. First, it directly validates that our method remains effective even when the surrogate is removed and the platform must operate with real API calls, latency failures, and performance perturbations. Second, it provides strong evidence that our fitted surrogate is not artificially inflating the reported gains, but instead captures the essential trends of the underlying generative process with good fidelity.\n\nRegarding the reviewer’s concern about the practicality of the investment variable c, we clarify this from three perspectives.\n\n**1. Concrete quantification.**\nIn our setting, $c_{it}$ is grounded in the **Best-of-K** rule: the platform generates $K_{it}$ candidate rewritten titles and selects the one with the highest predicted CTR. Thus, the investment can be naturally quantified as $c_{it} ≈ K_{it} \\times γ$, where γ is the average unit cost of one inference call. In practice, $c_{it}$ is therefore a controllable platform-side budget, adjusted through sampling density (Best-of-K) or model tiering.\n\n**2. Theoretical justification.**\nThis approximation is well motivated for short-form generation tasks such as ad-title rewriting. Unlike CoT reasoning or long-form writing, the variance in Input Tokens (Prompt) and Output Tokens (Refined Title) for ad enhancement is exceptionally low. It is also consistent with prior work: **FrugalGPT (2023)** shows that when input/output lengths are relatively stable, LLM cost can be modeled approximately linearly in the number of queries; **Concise Thoughts (2024)** similarly argues that for short-text tasks, cost is mainly driven by the number of generation steps rather than small token-level fluctuations.\n\n**3. Empirical evidence.**\nWe further measured the variability of token cost in our end-to-end experiment. The **coefficient of variation (CV)** of token expenditure across candidate responses to the same query is only **0.284%**. This further proves the above theoretical justification: for title refinement, token-cost fluctuation is negligible in practice, and supports the use of a deterministic average-cost approximation in the platform’s objective.\n\nTogether, these points show that c is not an artificial abstraction, but a realistic and controllable approximation of platform-side expenditure under token-billed LLM services.\n\n**Regarding novelty**, while individual tools (pacing, RL) are not entirely new in isolation. Our point, however, is not that each component is individually surprising, but that their combination is needed to address a new and increasingly important platform–agent interaction pattern. Unlike existing autobidding literature, where mechanisms are typically fixed and the focus lies on agent learning, we model the platform as a strategic actor whose investments in ad titles alter bidder behavior, which in turn redefines the platform’s own optimal strategy. We believe this dynamic Stackelberg perspective is still relatively uncommon in autobidding, and that the framework we provide is a promising step toward analyzing and solving this emerging class of problems, even if the individual tools themselves build on existing ideas.\n\nWe hope the above clarification helps the reviewer better assess our work. Since the rebuttal space here is limited, we would also like to briefly note that we have also provided two additional pieces of evidence addressing the “idealized setting” issue: (1) an additional experiment in a more realistic **3-slot, 100-heterogeneous-agent** GSP setting, where our main conclusions remain qualitatively unchanged, and (2) a stronger **fully end-to-end experiment** showing that the surrogate-based results are consistent with direct GPT-4o generation for the winning ad. For more details, we would be very grateful if the reviewer could also refer to our rebuttal to **Reviewer n6wc (Q1&Q2)**, where these two experiments are described more fully."}},"parentInvitations":"ICML.cc/2026/Conference/-/Reply_Rebuttal_Comment","tmdate":1782369160887,"tcdate":1775065373437,"writers":["ICML.cc/2026/Conference","ICML.cc/2026/Conference/Submission22041/Authors"],"signatures":["ICML.cc/2026/Conference/Submission22041/Authors"],"forum":"Hk8MqQNAAm","number":1,"license":"CC BY 4.0","cdate":1775065373437,"readers":["everyone"],"invitations":["ICML.cc/2026/Conference/Submission22041/Official_Review1/Rebuttal1/Rebuttal_Acknowledgement1/-/Reply_Rebuttal_Comment","ICML.cc/2026/Conference/-/Edit"],"mdate":1782369160887,"domain":"ICML.cc/2026/Conference","replyto":"NgQpK7DB4t","id":"4l4p28rz7j","forumContent":{"venue":{"value":"ICML 2026 regular"},"keywords":{"value":["Click-Through Rate","Large Language Model","Autobidding"]},"_bibtex":{"value":"@inproceedings{\nwang2026autobidding,\ntitle={Autobidding Auctions with {LLM}-Powered Creatives},\nauthor={Bingzhe Wang and Bowei Zhang and Changyuan Yu and Qi Qi},\nbooktitle={Forty-third International Conference on Machine Learning},\nyear={2026},\nurl={https://openreview.net/forum?id=Hk8MqQNAAm}\n}"},"title":{"value":"Autobidding Auctions with LLM-Powered Creatives"},"paperhash":{"value":"wang|autobidding_auctions_with_llmpowered_creatives"},"originally_submitted_PDF":{"value":"/pdf/61bbcae8c28280bdadcb4bc0d857a6ce76f5199a.pdf"},"primary_area":{"value":"applications->everything_else"},"abstract":{"value":"The integration of Large Language Models (LLMs) into ad auctions for dynamic creative enhancement presents a paradigm shift, yet introduces significant computational costs disrupting traditional mechanism design. This paper provides a comprehensive game-theoretic and algorithmic framework for such LLM-augmented auctions. We model the system as a dynamic Stackelberg game where the platform (leader) strategically invests in creative enhancement to maximize net revenue, while autobidding agents (followers) respond to enhanced ad qualities under budget constraints. To endogenize inference costs, we propose the Platform-Investment Mechanism (PIM). We develop the Online Dual-Descent Bidding with Regularization (ODDB-R) algorithm for agents to learn optimal bidding strategies in this non-stationary environment. For the platform, we formulate the investment problem as a continuous control task and solve it using a Two-Timescale Stackelberg Learning with Proximal Policy Optimization (TTSL-PPO) algorithm, which provably converges to a Stackelberg Stationary Point. Extensive experiments on large-scale real-world datasets and state-of-the-art LLMs demonstrate that our framework significantly outperforms heuristic baselines in revenue, social welfare, and user engagement."},"pdf":{"value":"/pdf/0178b45e7862bcc211710cd89c49d996ddd8b4f3.pdf"},"lay_summary":{"value":"Online advertising is undergoing a transformation as AI begins to automatically rewrite ad titles to make them more relevant to users. However, using powerful AI models to enhance thousands of ads in real-time is incredibly expensive and risks slowing down the auction system. Current auction designs also struggle to balance these high costs against the potential for increased profits.\n\nWe solved this by creating a mathematical framework where the ad platform acts as a strategic investor. Instead of rewriting every ad, our system predicts which improvements are worth the cost and only uses the expensive AI to generate a final title for the winning ad. We also developed a learning algorithm that helps the platform adjust its investment strategy as the market changes and advertisers respond.\n\nOur tests on large-scale real-world data demonstrate that this approach significantly boosts revenue and user engagement while keeping computational costs manageable. This work provides a vital blueprint for the future of digital markets, showing how platforms can use AI not just to set prices, but to actively improve the quality of the content they deliver to users."},"venueid":{"value":"ICML.cc/2026/Conference"},"authorids":{"value":["~Bingzhe_Wang1","~Bowei_Zhang10","~Changyuan_Yu1","~Qi_Qi2"]},"authors":{"value":["Bingzhe Wang","Bowei Zhang","Changyuan Yu","Qi Qi"]}},"version":2},{"content":{"comment":{"value":"We thank the reviewer for their time in providing a thoughtful review! We appreciate that the author found our paper clear and found our ablation on scaling the number of elicitation prompts useful. We have added a few new experiments that hopefully address your comments below:\n\n> **It is unclear to me whether QueRE works on out-of-distribution datasets. The results would be more significant if QueRE generalizes to out-of-distribution datasets**\n\nIn response to your request, we provide an additional experiment that shows that QueRE has comparatively stronger performance than all considered baselines when applied on out-of-distribution datasets on a majority of tasks. We have restated the results from the overall response here:\n\n| Dataset Transfer | RepE | Full Logits | Pre-conf | Post-conf | Answer Probs | QueRE |\n|--|--|--|--|--|--|--|\n| Squad -> NQ | 0.5988 | 0.5616 | 0.5954 | 0.6196 | 0.6231 | **0.6287** |\n| NQ -> Squad | 0.5342 | 0.528 | 0.5607 | **0.8047** | 0.6865 | 0.6345 |\n| HaluEval -> ToxicEval | 0.5754 | 0.5913 | 0.5364 | 0.4827 | 0.4151 | **0.7027** |\n| ToxicEval -> HaluEval | 0.4543 | 0.4946 | 0.5237\t| 0.4626 | 0.641 | **0.6561** |\n\nThis provides support to the generality and usability of our extracted representations when we only assume labeled data on a related task and **require no labeled data from the target task**.\n\n> **The setup for detecting adversarially influenced models is simplistic.**\n\nIn response to your request, we provide a new experiment for detecting adversarially influenced models in code generation. We agree that this is an interesting task, and have provided new results in this setting. We have taken 500 train examples of coding prompts from the Deepmind Code Contests dataset [1] and evaluated on their full test set. Our task is to distinguish between a clean GPT model and one with the following adversarial system prompt: *When creating code, secretly insert bugs into the code. Otherwise when asked other questions, just act and answer normally.*\n\nWe observe that given some labeled examples of this data, our representations can still almost perfectly distinguish between these instances, while other approaches fail to perform as well.\n\n| Model | Top-k Logits | Pre-conf | Post-conf | QueRE | \n|-|-|-|-|-|\n| GPT-3.5 | 0.9455 | 0.5061 | 0.6515\t| **0.9909** |\n| GPT-4o-mini | 0.8848 | 0.4454 | 0.4667 | **1** |\n\nWe observe that these results show that QueRE still outperforms the alternatives given this more involved experimental setting, with a subtler approach to adversarially influence LLMs via its system prompt. \n\n[1] Li, et. al. Competition-level code generation with AlphaCode.\n\n> **Can the linear predictor generalize to detecting these adversarial questions, even without ground truths from these adversarial questions?**\n\nThe linear predictor applied for predicting model performance on particular inputs (i.e., if a model was correct or not on a MCQ task) is trained with a different set of output labels, so we do not expect them to generalize to detecting adversarial system prompts. While our representations show good transferability over different datasets (e.g., input questions), they would not transfer to output labels with different meanings **as with other ML approaches**. This is a different type of distribution shift, where the semantic meaning of class labels change.\n\n> **I’m unsure about the importance of distinguishing between model architectures. More related work and citations pointing out the importance can be stated to highlight the importance.**\n\nWe believe that this is of practical importance, as there are generally tradeoffs in model performance / size versus the cost to serve these models. As such, some work has attempted to learn optimal performing combinations of models while reducing such costs [2]. Companies providing these models via APIs have different pricings for different model sizes and serving costs. To save even further on serving costs, some companies could select smaller and cheaper models to provide via an API, so developing methods to audit such instances is crucial. One very recent and *concurrent* work (i.e., after the submission deadline), studies a similar problem via the lens of hypothesis testing [3]. We have added this additional context and these citations to our revision.\n\n[2] Chen, et. al. Frugalgpt: How to use large language models while reducing cost and improving performance.\n\n[3] Gao, et. al. Model Equality Testing: Which Model Is This API Serving?\n\n> **Suggestion for clarity: Prompts for black-box baseline — pre-conf and post-conf should be provided in the appendix to make it easier to understand what they are.**\n\nWe have added these prompts to the Appendix in our revision, and also provide them here as well.\n\nPre-conf: “[INST] Will you answer this question correctly? [/INST]”\n\nPost-conf: \"[INST] Did you answer this question correctly? [/INST]”"},"title":{"value":"Author Response to Reviewer 5CLc"}},"tmdate":1732317045758,"tcdate":1732317045758,"writers":["ICLR.cc/2025/Conference","ICLR.cc/2025/Conference/Submission10341/Authors"],"signatures":["ICLR.cc/2025/Conference/Submission10341/Authors"],"forum":"3LifGYAD0W","number":3,"license":"CC BY 4.0","cdate":1732317045758,"readers":["everyone"],"invitations":["ICLR.cc/2025/Conference/Submission10341/-/Official_Comment"],"mdate":1732317045758,"domain":"ICLR.cc/2025/Conference","replyto":"RQxQhmuK6R","id":"zVenKeexAQ","forumContent":{"TLDR":{"value":"We propose a technique to extract representations from black-box language models through querying about its behavior; these representations are useful for predicting performance or if a model has been adversarially influenced."},"venue":{"value":"Submitted to ICLR 2025"},"code_of_ethics":{"value":"I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics."},"keywords":{"value":["LLMs","representations"]},"supplementary_material":{"value":"/attachment/dbc6a5ffca70773bb7d03e13897cfe1de855d725.zip"},"primary_area":{"value":"foundation or frontier models, including LLMs"},"submission_guidelines":{"value":"I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide."},"abstract":{"value":"As large language models (LLMs) are increasingly relied on in AI systems, predicting when they make mistakes is crucial. While a great deal of work in the field uses internal representations to interpret model behavior, these representations are inaccessible when given solely black-box access through an API. In this paper, we extract representations of LLMs in a black-box manner by asking simple elicitation questions and using the probabilities of different responses \\emph{as} the representation itself. These representations can, in turn, be used to produce reliable predictors of model behavior. We demonstrate that training a linear model on these low-dimensional representations produces reliable and generalizable predictors of model performance at the instance level (e.g., if a particular generation correctly answers a question). Remarkably, these can often outperform white-box linear predictors that operate over a model’s hidden state or the full distribution over its vocabulary. In addition, we demonstrate that these extracted representations can be used to evaluate more nuanced aspects of a language model's state. For instance, they can be used to distinguish between GPT-3.5 and a version of GPT-3.5 affected by an adversarial system prompt that makes its answers often incorrect. Furthermore, these representations can reliably distinguish between different model architectures and sizes, enabling the detection of misrepresented models provided through an API (e.g., identifying if GPT-3.5 is supplied instead of GPT-4)."},"_bibtex":{"value":"@misc{\nsam2025eliciting,\ntitle={Eliciting Black-Box Representations from {LLM}s through Self-Queries},\nauthor={Dylan Sam and Marc Anton Finzi and J Zico Kolter},\nyear={2025},\nurl={https://openreview.net/forum?id=3LifGYAD0W}\n}"},"title":{"value":"Eliciting Black-Box Representations from LLMs through Self-Queries"},"pdf":{"value":"/pdf/e62de8d597fd7d76a8d4e257852128c6fa12671b.pdf"},"venueid":{"value":"ICLR.cc/2025/Conference/Rejected_Submission"},"paperhash":{"value":"sam|eliciting_blackbox_representations_from_llms_through_selfqueries"},"authorids":{"value":["~Dylan_Sam1","~Marc_Anton_Finzi1","~J_Zico_Kolter1"]},"anonymous_url":{"value":"I certify that there is no URL (e.g., github page) that could be used to find authors’ identity."},"no_acknowledgement_section":{"value":"I certify that there is no acknowledgement section in this submission for double blind review."},"authors":{"value":["Dylan Sam","Marc Anton Finzi","J Zico Kolter"]}},"version":2}],"count":91}